Redesigning the Score Reports for the American Board of Psychiatry and Neurology's Certification Examinations

• In Event: 75.016 - Symposium on Recent Developments in Medical Certification Score Reporting

Abstract

The American Board of Psychiatry and Neurology (ABPN) administered its first multiple choice examinations in 1965 with the assistance of the National Board of Medical Examiners (Juul and Foertsch, 2012). Percent correct scores were reported for what were referred to as the Part I Psychiatry and Part I Neurology Examinations, and in addition to a total score subtest scores were provided for the main categories of the content outlines. These examinations had to be passed to sit for the Part II (oral) Examinations.
After much deliberation, the ABPN Directors decided to phase-out the oral examinations and to redesign the multiple-choice examinations to assess some of the cognitive tasks, such as clinical decision making, formerly addressed by the oral examinations. Some of the changes that have been made in the past decade include:
• Development of a Child Neurology Certification Examination, in addition to the Neurology and Psychiatry Certification Examinations
• Enhanced use of images, including video clips of patients interacting with physicians
• Introduction of case-based item sets, which required functionality in the computer delivery system so that item answers could not be changed once they had been recorded
• Development of two-dimensional content outlines; Dimension 1 = disorders; Dimension 2 = competencies and mechanisms
• Redesign of the score reports and the score interpretation guidelines
The development of the new content outlines for the 2017 examinations led to the redesign of the score reports. The process began with a review of the score reports from the U.S. licensure examinations (USMLE and COMLEX) as well as from other members of the American Board of Medical Specialties, including the American Board of Internal Medicine’s score report format for their Maintenance of Certification (MOC) examinations (American Board of Internal Medicine, 2018).
Several of the decisions that were made will be described:
• Selection of a score scale with a mean of 300 and SD of 50 in the context of examinee groups that varied in size (psychiatry ~ 1,500 first-time examinees; neurology ~ 675 first-time examinees; child neurology ~ 125 first-time examinees)
• Graphic presentation of individual performance relative to the pass/fail standard and to the performance of first-time examinees; historically the ABPN had emphasized performance relative to the pass/fail standard and not reported group performance
• Development of subtest scores on both dimensions based on Rasch item response theory (Wright and Stone, 1979) and graphic presentation of subtest scores relative to the passing level for the examination
• Development of new “score interpretation guidelines” for the examinees
• Electronic delivery of score reports and additional information, e.g., diplomate number, MOC information
Development of the new score reports required close coordination with test development, test scoring, and IT staff and included analysis of data from previous test administrations. The new score reports were successfully deployed for the 2017 examinations and were well-received by the examinees.

Author

• Dorthea H. Juul, American Board of Psychiatry and Neurology, Inc.