This paper examines the explanatory power of a uni-factor asset pricing model (CAPM) against a multi-factor model (The Fama-French three factor model) in explaining excess portfolio returns on non-financial firms on the Ghana Stock Exchange (GSE). Data covering the period January 2002 to December 2011 were used. A six Size- Book-to-Market (BTM) ratio portfolios were formed and used for the analysis. The paper revealed that, a uni-factor model like the (CAPM) could not predict satisfactorily, the excess portfolio returns on the Ghana Stock Exchange. By using the multi-factor asset pricing model, that is, the Fama-French Three Factor Model, excess portfolio returns were better explained. It is then conclusive enough that, the multi-factor asset pricing model introduced by Fama and French (1992) was a better asset pricing model to explain excess portfolio returns on the Ghana Stock Exchange than the Capital Assets Pricing Model (CAPM) and that there exist the firm size and BTM effects on the Ghanaian Stock market.