This study investigated how physical and cognitive function and psychological factors affected the health-related quality of life (HRQoL, hereafter HQ) of stroke patients in South Korea. The study enrolled 32 right-handed subjects with chronic cerebral infarction with disability and preserved cognitive function (Mini-Mental State Examination ≥ 20). Physical disability was assessed using the modified Rankin Scale (mRS) and Korean modified Barthel Index (KMBI). Quality of life was measured using the World Health Organization Quality of Life-Abbreviated form (WHOQOL-BREF, hereafter WB) and the 36-Item Short-form Health Survey (SF-36) in face-to-face interviews. Psychological distress was investigated using the Beck Depression Inventory Scale-II. The associations of each domain of WB and SF-36 were investigated using Pearson correlation analyses. Physical disability was negatively correlated with HQ in the SF-36. The physical function and bodily pain scales of the SF-36 were negatively correlated with physical disability. The general health domain of the SF-36 was negatively correlated with psychological scores. Emotional status was associated with physical health, social relationships, and general health in HQ. In summary, the severity of physical disability was associated with the patient's general and physical health and body pain. These findings suggest the importance of psychological, cognitive, and physiological interventions for improving the quality of life of patients after cerebral infarction.