A tuned mass damper is a kind of vibration damping device which has been widely used in tall buildings, machinery, bridges, aerospace engineering and other fields. In practical engineering applications, due to large deformation caused by large displacement, errors in engineering constructions and the existence of limit devices, the structure and tuned mass dampers inevitably produce some nonlinear characteristics, but these nonlinearities are often ignored. The results of this study confirm that the nonlinearity of the structure and the mass damper should be considered in the process of optimal frequency design, otherwise there will be a large deviation between the design optimal frequency of the mass damper and the actual optimal frequency. In this paper, nonlinear characteristics of the tuned mass damper and the main structure are considered. The first-order differential equations are obtained by using the complex average method, and the nonlinear equations of the tuned mass damper system are derived by using the multi-scale method. On this basis, the parameters are determined. The numerical results show that the error of the approximate solution method is small in the given example. The nonlinear tuned mass damper with nonlinear design exhibits a better control performance.