An Investigation of a Rater-Mediated Medical Licensing Performance Examination Equating Quality With the Rasch Model

• In Event: Equating Scores on Large-Scale Medical Performance Assessments: Practical Issues and Solutions

Abstract

Background
For several years rater-mediated performance examinations have been utilized by medical licensing and credentialing test agencies (Boulet, Smee, Dillon, & Gimpel, 2009). The examinee spends several hours in the day rotating through several test stations interacting with a standardized patient (SP) while carrying out required clinical tasks (e.g., gathering patient history and physical exam information, providing documentation of findings, etc.).
Multiple SP station test forms are developed for high-stakes licensing performance tests. It is important to gather evidence demonstrating forms are interchangeable to assure adequate linkage of test items among persons tested. Review of the literature provides limited evidence regarding the quality of linking and equating methods used for a rater medicated licensing examination. The Rasch measurement framework provides a process to equate unique sets of test forms to achieve interchangeability (cf., Wolfe, 2004). The purpose of this study is to apply Rasch model procedures to investigate the quality of equating test session forms for national medical licensure performance examination.

Method
The focus of this study is the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) USA (www.nbome.org), which is scored as a conjunctive pass/fail on two separate domains of the examination (Gimpel, Boulet, & Errichetti, 2003; Langenau, Dyer, Roberts, Wilson, & Gimpel, 2010). The Humanistic domain comprises doctor/ patient communication, professionalism, and interpersonal skills. The Biomedical/Biomechanical domain encompasses history taking, physical examination, documenting the patient encounter and osteopathic manipulative treatment.
Within each test session, a maximum of 12 candidates are randomly assigned to the first SP-station and assessed throughout the exam as they rotate through 12-SP stations completing required tasks. One station contains unscored material to pretest new SPs and medical cases. Examinees are given 14 minutes to interview the SP, and an additional 9 minutes to document their findings in a SOAP note format after exiting the station.
Facets (Linacre, 2010) was used to equate scores and to place measures from test forms onto the same linear continuum of clinical skill performance. Link quality is evaluated from four fit indices: Item-within-link fit, item-between-link fit, link-within-bank fit, and form-within-bank fit analysis (Wright & Bell, 1984).

Results
A random sample of 49 test sessions containing 574 examinees were drawn from the March 2, 2015 to February 19, 2016 testing cycle. The anchor test design for these data is shown in Table 1. Test sessions are sequentially numbered in the first column of Table 1. The twelve cases in each test session form have a unique color to illustrate the linkage of case test items among different forms. For example, case 1 is linked to cases 8, 10, and 11 in test session 7. Indices of quality for the equating process will be presented in this symposium.

Educational Significance
Equating a rater-mediated exam is central to any testing program that involves human scorers. Especially when stakes are high it is prudent to assess the quality of the equating process. Within the Rasch framework this study provides indices of the quality of linking forms from a national medical licensing performance exam.
 

Author

• William L. Roberts, National Board of Osteopathic Medical Examiners