Uganda’s agriculture faces numerous challenges, including low productivity due to declining soil fertility. Yet, the majority agricultural households in the country do not use organic and inorganic fertilizers due to not well-known constraints. Using data from the Uganda Census of Agriculture 2008/9, this paper provides insights into these constraints. Results show that most of the farm-households that use inorganic fertilizers also apply organic fertilizers. With regard to factors influencing adoption of fertilizer, lack of knowledge on use of and market information on fertilizer due to limited access to fertilizer-specific extension services is found to be perhaps the most limiting factor irrespective of fertilizer type. Low access to credit and constrained access to input and output markets due to distance are also key constraints to fertilizer use. Household characteristics including education level, household size, share of adults in the household, and ownership of livestock/poultry also stand-out as influencing factors on fertilizer adoption decisions. Results suggest that targeted interventions including extensive and intensive extension training and visits, and access to affordable credit and will be pertinent in the promotion fertilizer use in the country.