Response Process Validation of Video Communication Items for a Large-Scale Medical Licensure Exam

• In Event: 69.028 - Assessment of Behavioral and Noncognitive Competencies: Where Are We and Where Are We Going?

Abstract

Objective and Theoretical Framework: The assessment of communication skills has become a priority across the medical education continuum, including GME programs.1,2 The inclusion of communication assessment points earlier in the testing process, such as on the United States Medical Licensure Exams (USMLE), could provide additional measurement of these skills. USMLE Step 1 and Step 2 CK currently use multiple-choice question (MCQs) that are vignette-based, one-best-answer format. Some are intended to measure the quality of physician communication, but there are several challenges. First, these items are currently entirely text-based and thus somewhat artificial and contrived. In addition, it is difficult to properly describe a communications challenge without being so specific that the most correct option risks being intuitive if not obvious, leading to extremely easy items. Finally, the use of text limits the type of interactions that can be assessed, excluding non-verbal communication almost entirely.

To tackle these challenges, we propose the use of video communication items in which the clinical vignette is accompanied by a brief video clip showing physician-patient interactions within clinical scenarios. In 2015, a small number (11) of video items were developed along with a text-based equivalent item for each, for 22 total items. The goal is to allow the examinee to recognize complexities within interactions, thus expanding potential testing points within this important competency. However, the videos may introduce undesirable complexity and construct-irrelevant variance into the assessment point. The collection of response process validity evidence will be crucial in the development and pilot testing of these items, as it will be necessary to understand the examinee thought process when viewing the video interactions. This evidence will be gathered in the summer and fall of 2016 via focus groups and think-aloud studies with participants who have already passed Step 1 and Step 2 CK, but are otherwise representative of the groups who would be exposed to these item types.

Methods and Data: Participants selected for the study will be randomly divided into two groups; both groups will see the same items, but in different orders. Group 1 will be shown the text-based versions of the items first, thus advertising that they will be focusing on assessment points related to communication items. They will next be shown the video, the text portion of the vignette accompanying the video, and then the item. Group 2 will proceed in reverse order. Participants in each group will be asked to think aloud about topics including, but not limited to: the communication skills they think these videos are assessing; the skills they’re seeing demonstrated in the video; how difficult the items seems to be; the video quality and clarity; and usability/ease of item navigation.

Results and Significance: Results are expected in the late fall of 2016. The results will be used to guide decisions such as: the specific measurement construct, the skills assessed, item difficulty in relation to the media versus text, and proof of concept/feasibility of this item type.

Authors

• Kimberly A. Swygert, National Board of Medical Examiners

• Miguel A Paniagua, National Board of Medical Examiners

• Ren Liu, University of Florida

• Michael Barone, Johns Hopkins University