SCExAO at the Subaru telescope is a visible and near-infrared high-contrast imaging instrument employing extreme adaptive optics and coronagraphy. The instrument feeds the near-infrared light (JHK) to the integral field spectrograph CHARIS. Recently, a Wollaston prism was added to CHARIS' optical path, giving CHARIS a spectropolarimetric capability that is unique among high-contrast imaging instruments. We present a detailed Mueller matrix model describing the instrumental polarization effects of the complete optical path, thus the telescope and instrument. The 22 wavelength bins of CHARIS provide a unique opportunity to investigate in detail the wavelength dependence of the instrumental polarization effects. From measurements with the internal light source, we find that the image derotator (K-mirror) produces strong wavelength-dependent crosstalk, in the worst case converting ∼95% of the incident linear polarization to circularly polarized light that cannot be measured. Theoretical calculations show that the magnitude of the instrumental polarization of the telescope varies with wavelength between approximately 0.5% and 0.7%, and that its angle is exactly equal to the altitude angle of the telescope. We plan to more accurately determine the instrumental polarization of the telescope with observations of a polarization standard star, and fit more comprehensive physical models to all experimental data. In addition, we plan to integrate the complete Mueller matrix model into the existing CHARIS post-processing pipeline, with the aim to achieve a polarimetric accuracy of <0.1% in the degree of linear polarization. Our calibrations of CHARIS' spectropolarimetric mode will enable unique quantitative polarimetric studies of circumstellar disks and planetary and brown dwarf companions.