In order to identify the modal parameters of time invariant three-dimensional engineering structures with damping and small nonlinearity, a novel isometric feature mapping (Isomap)-based three-dimensional operational modal analysis (OMA) method is proposed to extract nonlinear features in this paper. Using this Isomap-based OMA method, a low-dimensional embedding matrix is multiplied by a transformation matrix to obtain the original matrix. We find correspondence relationships between the low-dimensional embedding matrix and the modal coordinate response and between the transformation matrix and the modal shapes. From the low-dimensional embedding matrix, the natural frequencies can be determined using a Fourier transform and the damping ratios can be identified by the random decrement technique or natural excitation technique. The modal shapes can be estimated from the Moore–Penrose matrix inverse of the low-dimensional embedding matrix. We also discuss the effects of different parameters (i.e., number of neighbors and matrix assembly) on the results of modal parameter identification. The modal identification results from numerical simulations of the vibration response signals of a cylindrical shell under white noise excitation demonstrate that the proposed method can identify the modal shapes, natural frequencies, and ratios of three-dimensional structures in operational conditions only from the vibration response signals.