The Use of an Entrustment-Based National Assessment Tool: Opportunities and Challenges

• In Event: Entrustment as Workplace-Based Assessment in Health Professions Education: Taking the Consequences of Assessment Seriously

Abstract

In 2014, the Association of American Medical Colleges (AAMC) published 13 Core Entrustable Professional activities (EPAs) for Entering Residency that a first postgraduate year resident could be expected to perform with indirect supervision on the first day of residency.1 The guide not only identifies a list of major real-life and authentic tasks as performed by clinicians regularly, but also provides descriptions about what are considered as “entrustable behaviors” as learners progress towards readiness to be entrusted with the task, and shifts the attention of assessment to entrustment decisions. Compared with a retrospective approach such as a competency or knowledge-skill-attitude (KSA)-based rubric (e.g., did the learner demonstrate the level as described?), the EPAs-based rubric uses a prospective approach (e.g., can the learner be trusted to do this task in the future?).2 Although job analyses and personnel psychology have generally moved from the task-based to competency-based framework 3, EPAs-based assessments gained much interest among clinical educators. The purpose of this presentation is to reflect on the use of one entrustment-based assessment tool for student performance to improve school scoring policies and practices.
At Central Michigan University College of Medicine (CMED), students rotate at various clerkships and sub-internships of different campuses during the clinical stage. Emergency Medicine (EM) is one of the required 4-week sub-internships for Year 4 students. Students in EM are typically assessed on each shift and observed by different assessors at different shifts, using the National Clinical Assessment Tool for Medical Students in Emergency Medicine (NCAT-EM) adopted since 2018.4 The NCAT-EM was developed at a national conference in 2016 based on the AAMC guides, and a modified Delphi process was used to seek consensus. It has been used by numerous EM clerkships in the U.S.A. currently.5 The NCAT-EM includes six domains using a four-point rating scale, along with professionalism which is measured dichotomously and a final norm-referenced global item. The NCAT-EM is used at CMED, except the global item. The rubric is used for both formative and summative purposes. The mid-clerkship formative assessment is based on the first two weeks’ performance and the summative assessment on all the four weeks.
During the academic year of 2021-2022, 89 faculty and residents assessed 97 students on 238 submission days. Descriptive and inferential statistics were calculated, including mean scores across the six domains, documentation about professionalism concerns, and rating scores by student/faculty background (e.g., gender). Paired t-test showed no significant differences between formative and summative scoring; however, the comparison of student performance at the beginning and end of the clerkship (three each) indicated evident growth. This provides an implication in changing scoring procedures regarding the timeline of the summative assessment at CMED. Overall, from the lens of school practices, we will reflect on opportunites of the use of the EPAs-based rubric, challenges in summative assessment, tensions between formative and summative assessment, rater reliability and scoring biases at different campuses, rater training and calibration.

Authors

• Xiaomei Song, Case Western Reserve University
  Presenting Author

• Derek J. Schaller, Central Michigan University
  Non-Presenting Author