Background and aims: Poor quality examination items may result in invalid test scores that potentially misrepresent what a student actually knows about a given content area. Thus, if an examination consists largely of poor quality items it is plausible that an individual with minimal content knowledge could perform reasonably well and receive a score that erroneously inflates his or her measure of ability. This study builds on this premise by introducing a novel method for evaluating item quality and demonstrating its utility. Method: We sought to understand the extent to which medical school examination items were vulnerable to good test-taking skills and guessing strategies by administering an examination to a group of medical education professional staff. The extent to which persons with no formal medical training could perform above the odds of random guessing were used to identify zones in which items may be vulnerable to guessing strategies. Results: The performance of professional staff was able to provide excellent diagnostic information regarding which items may be particularly vulnerable to guessing strategies. Conclusions: The proposed methodology was demonstrated to be successful, thus we encourage other medical educators to adopt this model for evaluating item and examination quality in a variety of contexts.