An original method for calculating the maximum vibration amplitude of the periodic solution of a nonlinear system is presented. The problem of determining the worst maximum vibration is transformed into a nonlinear optimization problem. The shooting method and the Floquet theory are selected to construct the general nonlinear equality and inequality constraints. The resulting constrained maximization problem is then solved by using the MultiStart algorithm. Finally, the effectiveness and ability of the proposed approach are illustrated through two numerical examples. Numerical examples show that the proposed method can give results with higher accuracy as compared with numerical results obtained by a parameter continuation method and the ability of the present method is also demonstrated.