Increasingly power-hungry processors have reinforced the need for aggressive power management. Dynamic voltage scaling has become a common design consideration allowing for energy efficient CPUs by matching CPU performance with the computational demand of running processes. In this paper, we propose Interaction-Aware Dynamic Voltage Scaling (IADVS), a novel fine-grained approach to managing CPU power during interactive workloads, which account for the bulk of the processing demand on modern mobile or desktop systems. IADVS is built upon a transparent, fine-grained interaction capture system. Able to track CPU usage for each user interface event, the proposed system sets the CPU performance level to the one that best matches the predicted CPU demand. Compared to the state-of-the-art approach of user-interaction-based CPU energy management, we show that IADVS improves prediction accuracy by 37%, reduces processing delays by 17%, and reduces energy consumed of the CPU by as much as 4%. The proposed design is evaluated with both a detailed trace-based simulation as well as implementation on a real system, verifying the simulation findings.