In many cases of rotating systems, such as jet engines, two or more coaxial shafts are used for power transmission between a high/low-pressure turbine and a compressor. The major purpose of this study is to predict the nonlinear dynamic behavior of a coaxial rotor system supported by two active magnetic bearings (AMBs) and contact with two auxiliary bearings. The model of the system is formulated by ten degrees-of-freedom in two different planes. This model includes gyroscopic moments of disks and geometric coupling of the magnetic actuators. The nonlinear equations of motion are developed by the Lagrange's equations and solved using the Runge–Kutta method. The effects of speed parameter, speed ratio of shafts, and gravity parameter on the dynamic behavior of the coaxial rotor–AMB system are investigated by the dynamic trajectories, power spectra analysis, Poincaré maps, bifurcation diagrams, and the maximum Lyapunov exponent. Also, the contact forces between the inner shaft and auxiliary bearings are studied. The results indicate that the speed parameter, speed ratio of shafts, and gravity parameter have significant effects on the dynamic responses and can be used as effective control parameters for the coaxial rotor–AMB system. Also, the results of analysis reveal a variety of nonlinear dynamical behaviors such as periodic, quasi-periodic, period-4, and chaotic vibrations, as well as jump phenomena. The obtained results of this research can give some insight to engineers and researchers in designing and studying the coaxial rotor–AMB systems or some turbomachinery in the future.