The nonlinear dynamics of a tool commonly employed in deep hole drilling is analyzed. The tool is modeled as a continuous beam that vibrates in transverse and axial directions caused by cutting process. The mechanical model of cutting forces is a nonlinear function of cutting tool displacement including state variables with time delay. The equations of new surface formation are separated as a specific set. These equations naturally include the regeneration effect of oscillations under cutting and it is possible to analyze continuous and intermittent cutting involving either stationary or non-stationary processes. The vibratory drilling tool feed consists of a constant as well as a periodic component. Due to the constant component of feed, the distance between the supports of beam is regarded as a slowly varying function of time. The system includes three types of non-linearity, viz., discontinuity of cutting, nonlinear machining law, and the influence of transverse displacement of beam on axial deflection of cutting edges. Variation of machined chip thickness cause variation of axial trust force and it is an additional source of parametric excitation of system oscillations.