The objective function based on mesh quality metric has a major impact on smoothing unstructured tetrahedral meshes. The ability of seven mesh quality metrics to distinguish four kinds of poor-quality elements and their effects on the change of element shape are analyzed in detail. Then, four better mesh quality metrics are chosen to construct objective functions. In addition, the rational determination of searching direction and the optimal step size in the optimization algorithm of solving the objective function are proposed. Finally, comparisons with the other three objective functions are made according to different number of elements, iteration limit, and the desired accuracy in the improved mesh. It is found that with the increase of the number of elements, the time consumed during optimization increases, but the changes of the worst quality element are different. The number of iterations has little effect on the mesh quality and the time cost. The increasing of the desired degree of accuracy will improve the mesh quality and cost more time. Furthermore, the approach using objective function is compared with Freitag's common approach. It is clearly shown that it performs better than the existing approach.