In the present four-year longitudinal study we investigated lifestyle factors that would improve the physical fitness of children considered to have “inferior” physical fitness. The participants were 9,593 elementary school children. Personal data obtained annually were linked over four years. In the first year, we focused on the group considered to have inferior physical fitness (T-score <45). We calculated the degree of change in terms of differences in T-scores between the first and fourth years. On the basis of the change evident in the fourth year, we subdivided the “inferior” group into two groups: those showing a decrease or increase in physical fitness. We then analyzed the lifestyle changes in the two groups. In the group showing increased physical fitness, fitness had improved to the national average after three years. Improvement of exercise habits was show to enhance fitness, whereas deterioration of exercise habits reduced fitness. This difference between the groups was significant. In contrast, health habits and dietary habits showed no significant inter-group differences, and no association with physical fitness. These results suggest that improved exercise habits can help attain the national average or even increase the physical fitness of a child who has been previously evaluated to have inferior fitness.