Assessor Characteristics and Reliability of Third-Year Medical Student Clerkship Assessments: A Generalizability Theory Analysis

• In Event: Validity Evidence Across Professions Education Settings

Abstract

Introduction
Optimizing the reliability of third-year medical student clerkship assessments is important given their high-stakes nature. Generalizability theory can be used to investigate the impact of contextual factors on clerkship assessment reliability.

Methods
We completed generalizability analysis of assessments about third-year medical students on five clinical clerkships during the 2021-2022 academic year. Analysis was also done by assessor type and assessor-student contact frequency.

Results
Differences in reliability by clerkship were found. Residents performed more reliable assessment than attendings. Assessors with daily student contact performed more reliable assessment than assessors with less frequent contact.

Conclusion
Differences in distribution of assessor type and assessor-student contact frequency may explain between-clerkship differences in reliability of assessment and offer targets for improvement in assessment systems.

Authors

• Christopher Mattson, Ann and Robert H. Lurie Children's Hospital of Chicago

• Celia Laird O'Brien, Northwestern University

• Yoon Soo Park, University of Illinois at Chicago

• Sandra M. Sanguino, Northwestern University

• Mary E. McBride, Northwestern University

• Brigid M. Dolan, Northwestern University