We present the design and prototyping results for an ultra-wideband rotating polarization modulator that consists of a stack of quartz plates. The plate thicknesses and orientations were optimized such that after rotation of the modulator to 6 different angles before a polarization analyzer, the full Stokes vector can be optimally determined at all wavelengths from 300 to 2500 nm. Additional optimization parameters include minimal variation of the retardance with incidence angle and temperature, and the suppression of polarized spectral fringes for a spectral resolution of 10,000. The prototype modulator's design was re-optimized after the production and measurement of each individual quartz plate. We present the performance of the as-built prototype. To eliminate aliasing with inherent temporal variations of the source, the modulator can be used together with a polarizing beam-splitter (ual-beam approach). Because of the large sinusoidal spectral variations of the polarization modulation, this modulator can also be considered a spectral modulator for channeled spectropolarimetry. Therefore, at each modulation state, spectrally resolved polarization information can also be extracted directly, although at limited spectral resolution. We use this modulator as an example of a multi-domain polarization modulator, and outline a general approach for optimally storing polarization information in all available measurement dimensions (temporal, spatial, spectral), and rendering the overall polarization measurement independent from systematic effects in any of these dimensions.