The purpose of this study were to investigate factorial structures of physique and physical fitness, and to determine the change of physique and physical fitness with age and its sex difference in the elderly. Nineteen test items were selected from 5 domains of physique, muscle function, joint function (flexibility), neuromuscular function and lung function, considering the validity, safety and convenience of tests. The subjects were 207 males and 226 females aged 65 to 89 years. Factor analysis was applied to each correlation matrixes consisting of 8 physique variables and 11 physical fitness variables. In physique domain, three extracted factors were interpreted as body fat, body linearity and body bulk. Body bulk and body linearity in both sexes and body fat in females decrease significantly with age. Body linearity was found significantly larger in males than females. Body fat was significantly greater in females. In physical fitness domain, four factors were extracted and interpreted as muscular strength, balance, agility of upper and lower limbs, and flexibility. A significant declining trend with age was found in the above-mentioned physical fitness elements both sexes. Also, significant sex differences in muscular strength, balance, and flexibility were found, and males were superior to females except for flexibility. It was inferred that the influence of aging in muscular strength and balance is greater flexibility and agility of upper and lower limbs in the elderly. Further, the decrease of muscular strength seems to facilitate the decline of balance with age.