Using Multivariate Generalizability Theory to Evaluate Subscore Utility for Different Subgroups of Examinees

• In Event: 29.082 - Roundtable Session 9
  In Roundtable Session: 29.082-13 - Improving the Accuracy of Human Rating and Scoring

Abstract

Research on the utility of subscores has found that subscores are seldom worth reporting. Studies indicate that these conclusions generally apply to subgroups based on gender, ethnicity, or other factors. However, these studies have depended on correlational methods which may not be sensitive to mean differences in subscore performance. Although multivariate G-Theory provides an alternative framework for evaluating score profiles, little or no research exists. This study evaluates the sensitivity of a multivariate G-Theory index to differences in score profiles for subgroups of examinees, and compares it to an accepted standard (PRMSE). Results indicate that while the two methods often point toward the same decision, there are numerous instances where they suggest different conclusions. This investigation encourages additional work.

Authors

• Zhehan Jiang, The Center for Educational Teaching & Evaluation - The University of Kansas

• Mark R. Raymond, National Board of Medical Examiners