Tracking and characterizing both active and inactive Space Objects (SOs) is required for protecting space assets. Characterizing and classifying space debris is critical to understanding the threat they may pose to active satellites and manned missions. This work examines SO classification using brightness measurements derived from electrical-optical sensors. The classification approach discussed in this work is data-driven in that it learns from data examples how to extract features and classify SOs. The classification approach is based on a deep Convolutional Neural Network (CNN) approach where a layered hierarchical architecture is used to extract features from brightness measurements. Training samples are generated from physics-based models that account for rotational dynamics and light reflection properties of SOs. The number of parameters involved in modeling SO brightness measurements make traditional estimation approaches computationally expensive. This work shows that the CNN approach can efficiently solve classification problem for this complex physical dynamical system. The performance of these strategies for SO classification is demonstrated via simulated scenarios.