In computing a periodic steady-state vibration generated in a large-scale nonlinear structure by the incremental transfer stiffness coefficient method suggested in a previous paper, the stable and unstable solutions can be computed without distinction. Thus, the stability of the obtained solution must be examined. However, it is very difficult to analyze the stability of the solution of the nonlinear system with high degree of freedom. To overcome this difficulty, a method to reduce the dimension of the equation used for stability analysis without spoiling the accuracy is developed by applying the concept of modal analysis to the variational equation used for the stability analysis. Two types of modal matrices are considered in the reduction of dimension, and a method is proposed to extract the modes that dominate the stability of the solution. The validity of the incremental transfer stiffness coefficient method and the method of stability analysis using the reduction model is confirmed by numerical computational results.