Psychometric Considerations for Mitigating Assessment Bias

• In Event: Dismantling Assessment Injustice: Reconceptualizing Consequential Validity, Assessment Fairness, and Measurement Considerations in the Professions

Abstract

Assessment Bias and Potential Sources of Bias in Medical Education
Assessment bias refers to systematic errors or unfairness in an assessment process that leads to differential outcomes for different groups of individuals (Millsap & Everson, 1993; Popham, 2012). These groups may be defined by various characteristics, such as gender, race, ethnicity, socioeconomic status, or other demographic factors. The bias can occur at different stages of the assessment process, including test design, item construction, administration, scoring, and interpretation. The common potential sources of assessment bias in medical education may include 1) cultural and linguistic bias, 2) stereotype threat, 3) content validity, 4) test format and structure, and 5) assessor bias. Assessment bias distorts the measurement of knowledge, skills, or abilities and may lead to inaccurate conclusions about individuals' true capabilities. The consequences of bias in medical education can be particularly severe, as it may impact the quality of healthcare professionals' training and, ultimately, patient care.

Traditional Psychometric Analysis for Identifying Assessment Bias
Identifying and mitigating assessment bias is crucial in ensuring fair and equitable evaluation of learners in any educational context, including medical education. Differential Item Functioning (DIF) and latent subgroup analysis have long been standard methods to detect assessment bias. DIF analysis involves comparing the performance of different groups on individual test items while controlling for overall ability levels. If a test item exhibits differential performance between groups with similar overall ability, it may indicate the presence of bias. Latent subgroup analysis is another approach used to identify bias by examining differences in item performance across unobservable subgroups. These methods can help uncover subtle biases that may not be apparent in group comparisons. Yet, the effectiveness of using them in medical education has been hindered by the difficulties in collecting enough samples and defining relevant subgroups within this diverse context and the dynamic nature of medical student population. Medical student populations are highly diverse, and creating meaningful subgroups for analysis can be complex and contentious.

Challenges and Uniqueness in Medical Education
Medical education involves a unique set of complexities that make the application of traditional bias detection methods arduous. Firstly, medical assessments often cover a wide range of knowledge, skills, and competencies essential for healthcare professionals. Designing items that capture these nuanced aspects while remaining unbiased is daunting. Moreover, the high-stakes nature of medical assessments demands a robust and comprehensive approach to bias detection to avoid potential adverse effects on students' careers and, ultimately, patient outcomes.

The Call for Specialized Methods in Medical Education
Given the limitations of traditional psychometric approaches in detecting bias in medical education, it is imperative to develop specialized methods tailored to this specific domain. This conceptual session aims to raise awareness about the need for research and innovation in psychometrics to create more effective tools for bias detection in medical assessments. These specialized methods should consider the unique attributes of medical education and embrace the complexities of the domain to yield accurate and actionable results.

Authors

• Chi Chang, Michigan State University

• Ting Sun, University of Utah