Development of a Competency-Based Assessment System: A Practical Guide to Procedural and Validity Considerations

• In Event: 58.082 - Roundtable Session 25
  In Roundtable Session: 58.082-11 - Roundtable Session: Methods and Measures

Abstract

The mandate requiring competency-based education and assessment in graduate medical education has led to the need for novel methods of assessment to evaluate outcomes. This paper describes development of a large-scale competency-based assessment system in pediatrics residency training and emphasizes how development activities can be structured to provide evidence for the validity of inferences made based on the resulting scores. Process outcomes indicate that SMEs engaged in structured and iterative development efforts generated the necessary competency decisions and item and instrument content, and attention to validity considerations during development provided additional confidence in the outcomes. Specific examples of framing questions and actual development activities are provided and may be used by practitioners to inform development of similar types of assessments.

Authors

• Melissa J. Margolis, National Board of Medical Examiners

• Patricia J Hicks, The University of Pennsylvania

• Alan Schwartz, University of Illinois at Chicago

• Carol Carraccio

• Brian E. Clauser, National Board of Medical Examiners