In this paper the nonlinear dynamic responses of a rigid rotor supported by ball bearings due to surface waviness of bearing races are analyzed. A mathematical formulation has been derived with consideration of the nonlinear springs and nonlinear damping at the contact points of rolling elements and races, whose stiffnesses are obtained by using Hertzian elastic contact deformation theory. The numerical integration technique Newmark-β with the Newton–Raphson method is used to solve the nonlinear differential equations, iteratively. The effect of bearing running surface waviness on the nonlinear vibrations of rotor bearing system is investigated. The results are mainly presented in time and frequency domains are shown in time-displacement, fast Fourier transformation, and Poincaré maps. The results predict discrete spectrum with specific frequency components for each order of waviness at the inner and outer races, also the excited frequency and waviness order relationships have been set up to prognosis the race defect on these bearing components. Numerical results obtained from the simulation are validated with respect to those of prior researchers.