Many alternative forms of assessment--portfolios, oral examinations, open-ended questions, essays--rely heavily on multiple raters, or judges. Multiple raters can improve reliability just as multiple test items can improve the reliability of standardized tests. Choosing and training good judges and using various statistical techniques can further improve the reliability and accuracy of instruments that depend on the use of raters.

After identifying several common sources of rating errors, this article examines how the impact of rating errors can be reduced.