Using data for the 1990's, this paper examines the role of sheepskin effects in the returns to education for Japan. Our estimation results indicate that sheepskin effects explain about 50% of the total returns to schooling. We further find that sheepskin effects are only important for workers in small firms with the size of these effects being similar to comparable estimates for the US. Finally, the estimated sheepskin effects are decreasing with firm tenure, in particular for small firms. These results could be explained by the particular recruitment system of large firms in Japan, which makes university diploma as a screening device unimportant for large firms.