Assessing and Accounting for Differential Rater Functioning in Oral Examinations

• In Event: Equity in Assessment in the Health Professions

Abstract

Introduction
Provision of an equal opportunity to demonstrate knowledge and skills on certification tests is a critical aspect of validity1. The American Board of Obstetrics and Gynecology (ABOG), which offers voluntary board certification to obstetricians and gynecologists across the country, has taken active steps to facilitate equity and fairness throughout the certification process. To this end, ABOG regularly conducts differential item functioning (DIF) analysis and a content review by Bias and Fairness Panel. While the DIF analysis helps in identifying possibly biasing examination content, rater-mediated assessments have an additional source of potential construct-irrelevant variance - the examiner. Differential rater functioning (DRF) analysis can help assess the extent to which the severity of an examiner differs over examinees from different demographic backgrounds.10 An approach to assessing and accounting for DRF in rater-mediated assessments is proposed in the context of ABOG’s oral certifying examination.
Methodology
The 2021 ABOG Specialty Certifying Examination was administered remotely to 1491 candidates over the Zoom platform across four separate administrations between November 2021 and January 2022. The oral examination administration for each candidate is spread over three sessions of an hour each. Two examiners grade the performance of each candidate on five structured cases and a case list in each session. Data is then analyzed using the Many-Facet Rasch Model11 in Facets12 to account for differences in case difficulty and examiner severity to maintain the same standard of passing the examination for all candidates.
In the pilot implementation of the proposed approach, each examiner’s severity was estimated for white and non-white candidates when the examiner graded at least 5 candidates from each of these groups. Differential rater functioning was assessed for examiners based on the statistical significance of the difference in severity for the two groups and the magnitude of its difference greater than or equal to 0.50 logits. Severity was estimated separately for white and non-white candidates for the examiners who demonstrated DRF and accounted for in estimating fair measure average for all candidates using the Many-Facet Rasch Model. The impact of accounting for DRF was assessed by comparing pass rates based on this estimation with the pass rates from the estimation without accounting for DRF.
Results
The impact of using separate severity estimates of examiners who demonstrated DRF for white and non-white candidates on examination pass rates was small based on the analysis of the first two administrations.
Discussion and Limitations
While it is imperative to minimize bias in examinations to be fair to all candidates1, it is important to note that the proposed assessment of DRF is based solely on statistical findings. This contrasts with the assessment of the use of DIF analysis in identifying bias where statistical flagging of items is followed by review by a diverse panel of content experts to identify the source of DIF and judge whether it is relevant to the construct or not13. Limitations must be carefully considered before implementing the proposed methodology.

Authors

• Rajat Chadha, American Board of Obstetrics & Gynecology
  Non-Presenting Author

• Heath Kincaid, American Board of Obstetrics and Gynecology
  Presenting Author