Tapping-mode atomic force microscopy has wide applications for probing the nanoscale surface and subsurface properties of a variety of materials in a variety of environments. Strongly nonlinear effects due to large variations in the force field on the probe tip over very small length scales and the intermittency of contact with the sample, however, result in strong dynamical instabilities. These can result in a sudden loss of stability of low-contact-velocity oscillations of the atomic-force-microscope tip in favor of oscillations with high contact velocity, coexistence of stable oscillatory motions, and destructive, nonrepeatable, and unreliable characterization of the nanostructure. In this paper, dynamical systems tools for piecewise-smooth systems are employed to characterize the loss of stability and associated parameter-hysteresis phenomena.