We investigate experimentally and theoretically the influence of non-radiative carrier losses on the performance of VECSELs under pulsed and CW pumping conditions. These losses are detrimental to the VECSEL performance not only because they reduce the pump-power to output-power conversion efficiency and lead to increased thresholds, but also because they are strong sources of heat. This heating reduces the achievable output power and eventually leads to shut-off due to thermal roll-over. We investigate the two main sources of non-radiative losses, defect recombination and Auger losses in InGaAs-based VECSELs for the 1010nm-1040nm range as well as for InGaSb-based devices for operation around 2μm. While defect related losses are found to be rather insignificant in InGaAs-based devices, they can be severe enough to prevent CW operation for the InGaSb-based structures. Auger losses are shown to be very significant for both wavelengths regimes and it is discussed how structural modifications can suppress them. For pulsed operation record output powers are demonstrated and the influence of the pulse duration and shape is studied.