The nonlinear dynamics of a tool commonly employed in deep hole drilling is analyzed. The tool is modeled as a two-degree of freedom system that vibrates in axial directions and twists caused by the cutting process. The mechanical model of cutting forces is a nonlinear function of cutting tool displacement including state variables with time delay. The equations of new surface formation are separated as a specific set. These equations naturally include the regeneration effect of oscillations under cutting and it is possible to analyze continuous and intermittent cutting involving either stationary or non-stationary processes. The vibratory drilling tool feed consists of a constant as well as a periodic component. The influence of tool axial and torsional dynamics on chip formation is considered. The Poincare maps of state variables for various sets of operating conditions are presented. The obtained results allow prediction of conditions for stable continuous cutting and unstable regions. The conditions of tool axial vibrations synchronized at frequency of external excitation are analyzed. The time domain simulation allows determination of the chip shape most suitable for certain workpiece material and tool geometry. It is also shown that disregarding tool torsional vibrations may significantly change the chip formation process. The suggested model can be applied at the stage of the manufacturing process design.