A Critical Re-Examination of Fairness on Massachusetts’s High-Stakes Assessment and Graduation

• In Event: Doing Critical Quantitative Research: Methodological Commitments to Equity and Justice

Abstract

Motivation

High-stakes assessments have far reaching consequences. Some states have graduation exams tied to test performance (Macdonald et al., 2019). Unfortunately, these high-stakes consequences disproportionately impact racially and socioeconomically marginalized students and communities, not because of individual deficits, but because of systemic, intersecting forms of oppression (e.g., Kozol, 2012). One manifestation may be the assessment.

Although establishing assessment fairness for marginalized individuals has been important psychometrically over the past decades (Sireci & Randall, 2021), the recommendations remain insufficient. Randall and colleagues (2024) argue for continued critical interrogation of the fairness validation process through the Justice-oriented Anti-racist Validation. Critical approaches with intersectional differential item functioning (DIF) (Russell & Kaplan, 2021) and measurement invariance (MI) (Xu & Soland, 2024) reveal that contemporary measures may be potentially biased against multiple marginalized groups or fail to allow group comparisons. Still, existing research often focuses on isolated grades or years and overlooks high-stakes implications.

We ask: (1) To what extent do high-stakes standardized assessments exhibit fairness for multiple marginalized groups, over time and across cohorts? (2) If bias exists, what are the consequences for graduation? (3) What does fairness look like for communities most impacted?

Massachusetts Context and Data

Massachusetts has used the Massachusetts Comprehensive Assessment System (MCAS) for decisions like whether students graduate. Although the 2024 ballot measure ended the MCAS graduation requirement statewide, local districts may still require it. Despite its use for high-stakes decisions, Massachusetts has not examined the assessment for intersectional sources of bias.

We use public item-level data from 2001–2023 for grades 3–8 and 10 (~10 million observations). These data include students’ race, socioeconomic status, and gender that can be found in Table 1. We also incorporate graduation policies and public input from listening sessions and school board meeting minutes, with particular focus on historically marginalized districts.

Methods and Preliminary Results

In the first phase, we assess fairness using intersectional DIF and MI methods for each grade-year combination (e.g., 2001 3rd graders). We then simulate how removing biased items would affect marginalized students’ ability to meet graduation requirements. In the second phase, we analyze community perspectives on assessment fairness from public comments.

Our preliminary findings suggest potential persist bias against multiple marginalized students. Across 133 grade-year combinations, assessment items consistently favor white, economically advantaged males. In Table 2, we document that ~28% show potential bias against Black, economically disadvantaged females. Tables 3 and 4 show that the potential bias remain high over time and across cohorts for Black, economically disadvantaged females.

Public comments from the listening session and school board meetings, though not representative or readily available, similarly perceived the MCAS as unfair. Six (of the 20) school districts directly voted to remove the MCAS as a graduation requirement. Furthermore, we document efforts to identify alternatives to the MCAS for graduation that would match students’ interests and prepare them beyond for success beyond rote learning. By AERA, we will complete measurement invariance, simulations, and community perspective analyses. However, initial evidence suggests that fairness on high-stakes assessments remains elusive.

Authors

• John Wang, University of Virginia

• Wendy Castillo, Montclair State University