Setting a Performance Standard for the National Assessment Collaboration Examination: Application of Generalizability Theory

• In Event: 16.031 - Large-Scale Assessments: Issues in Development and Score Interpretation

Abstract

The purpose of this study was to validate the cut-score for a national medical examination. A nationally representative panel of 18 physician experts were selected and divided into two sub-panels. A stratified random sample of 50 candidates was used to set the cut-score for each station. Each rater provided, per station, their judgment on which category each examinee belonged: unacceptable, borderline and acceptable. Generalizability-Theory analysis was conducted to evaluate the effects of persons, stations, raters, and panel. The results indicated that largest effect was for person by station, person, and station. There were small effects due to rater and panel. These results indicate a high level of confidence in the generalizability of the standard that was recommended.

Authors

• Andrea Gotzmann, Medical Council of Canada

• Andre F. De Champlain, Medical Council of Canada

• Sirius Qin, Medical Council of Canada

• Fang Tian, Medical Council of Canada