We investigated the association of self-reported difficulty of food access, accounting for the perception of food-store accessibility, with nutrient intake among 474 inhabitants (145 men and 329 women) aged ≥40 y in a rural area of Japan. Information on self-reported difficulty of food access and dietary intake was obtained via a self-administrated questionnaire. Analysis of covariance was performed to evaluate the adjusted associations between difficulty of food access and percentages of total energy (i.e., protein, fat, and carbohydrates). Among men, the adjusted means of percent energy from fat in the “difficulty of food access” group (19.2% energy) were significantly lower (by 2.4%) and the percent energy from carbohydrates for this group (57.3% energy) was significantly higher (by 3.3%) than for the “non-difficulty” group. In conclusion, this study found nutritional balance among residents experiencing difficulty of food access results in lower fat and higher carbohydrate intake than for those with non-difficulty of food access among middle-aged and older Japanese men in a rural area.