The nonlinear dynamics of a tool commonly employed in deep hole drilling is analyzed. The tool is modeled as a two-degree of freedom system that vibrates in the axial and torsional directions as a result of the cutting process. The mechanical model of cutting forces is a nonlinear function of cutting tool displacement including state variables with time delay. The equations of new surface formation are constructed as a specific set. These equations naturally include the regeneration effect of oscillations while cutting, and it is possible to analyze continuous and intermittent cutting as stationary and nonstationary processes, respectively. The influence of the axial and torsional dynamics of the tool on chip formation is considered. The Poincaré maps of state variables for various sets of operating conditions are presented. The obtained results allow the prediction of conditions for stable continuous cutting and unstable regions. The time domain simulation allows determination of the chip shape most suitable for certain workpiece material and tool geometry. It is also shown that disregarding tool torsional vibrations may significantly change the chip formation process.