We present a microscopic many-body theory of optical refrigeration of p-doped semiconductors. Conceptually, the refrigeration mechanism is the upconversion of pump photons through absorption and subsequent luminescence by electron-hole pairs. The electron-hole pair can be an unbound pair, a pair bound by the attractive Coulomb interation (exciton), or a pair in which the hole is located at an acceptor site. Assuming the electron-hole pairs to be in quasi-thermal equilibrium, our theory calculates its absorption and luminescence spectra within a diagrammatic (real-time) Green's function approach at the self-consistent T-matrix level. The strong on-site Coulomb repulsion of holes at acceptor sites is taken into account via a truncation of the acceptor Fock space, which excludes states with higher than single-hole occupation. The resulting absorption and luminescence spectra are used in a cooling threshold analysis for GaAs that also takes into account other losses into heat. We compare the present results for p-doped GaAs with previous ones obtained for undoped GaAs.