Climate change damages agriculture, causing deteriorating food security and increased malnutrition. Many studies have examined the role of distinct physical processes, but impacts have not been previously attributed to individual pollutants. Using a simple model incorporating process‐level results from detailed models, here I show that although carbon dioxide (CO₂) is the largest driver of climate change, other drivers appear to dominate agricultural yield changes. I calculate that anthropogenic emissions to date have decreased global agricultural yields by 9.5 ± 3.0%, with roughly 93% stemming from non‐CO₂ emissions, including methane (−5.2 ± 1.7%) and halocarbons (−1.4 ± 0.4%). The differing impacts stem from atmospheric composition responses: CO₂ fertilizes crops, offsetting much of the loss induced by warming; halocarbons do not fertilize; methane leads to minimal fertilization but increases surface ozone which augments warming‐induced losses. By the end of the century, strong CO₂ mitigation improves agricultural yields by ∼3 ± 5%. In contrast, strong methane and hydrofluorocarbon mitigation improve yields by ∼16 ± 5% and ∼5 ± 4%, respectively. These are the first quantitative analyses to include climate, CO₂ and ozone simultaneously, and hence, additional studies would be valuable. Nonetheless, as policy makers have leverage over pollutant emissions rather than isolated processes, the perspective presented here may be more useful for decision making than that in the prior work upon which this study builds. The results suggest that policies should target a broad portfolio of pollutant emissions in order to optimize mitigation of societal damages.