In this paper, the attitude consensus control of a rigid body multi-vehicle system with communication time delays is studied using rotation matrices. The control objectives are attained under a fixed and undirected communication topology. Heterogeneous communication delays that are inherently present in transmission systems are considered in this paper, and sufficient conditions are derived to achieve almost global asymptotic stability of the consensus subspace. Consensus protocols are proposed for control of N heterogeneous rigid bodies in the framework of the tangent bundle TSO(3) associated with Lie group SO(3). Finally, simulation examples are given to demonstrate the proposed method.