Bias on the Force Concept Inventory Across the Intersection of Gender and Race

• In Event: Advancing Methods for Evaluating Construct Validity in STEM Education Research

Abstract

In this investigation, we used item response theory and differential item functioning (DIF) analysis to examine whether the items on the Force Concept Inventory (Hestenes et al., 1992) provided unbiased data across social identifiers for race and gender. Prior investigations of the FCI have found evidence of item bias across genders (Traxler et al., 2018; Henderson et al., 2019). Our investigation expands on these findings by performing an intersectional analysis that examines the potential for bias across genders and races.

Perspectives
This investigation is driven by our intersectional perspective. Intersectional theory posits that each aspect of our social identities (e.g., gender, race, and disability status) dynamically interact with each other and society’s associated power structures to create inequities (Collins, 1990; Crenshaw, 1990). While the ways that intersectionality can inform quantitative research is still being explored (Evans, 2019), disaggregation of data is an important first step.

Methods
We performed a differential item function analysis using both classical test theory and item response theory. We disaggregate our data across the intersection of five racial and two gender identities (10 groups total).

Data sources
The data came from the Force Concept Inventory. We accessed the data through the Learning About STEM Student Outcomes (LASSO) platform's (Van Dusen, 2018) research database. The LASSO platform collects large-scale, multi-institution data by administering, scoring, and analyzing research-based assessments online. The data came from 4,848 students posttests administered at the end of 152 calculus-based introductory physics courses from 16 institutions.

Results
Our results add to the growing evidence that the FCI functions differentially for students across social identifiers. It is possible, but unlikely, that random divisions would find differences across groups. The fact that we found so many meaningful DIF scores, some of which were consistent across multiple groups, provides a strong indication that items were acting differently across social identifiers. Items 10, 14, 23, and 27 appear especially problematic, favoring White men over most other groups to a greater degree than the FCI as a whole. These results support what Traxler et al. (2018) found with items 14, 23, and 27 as having a large DIF favoring men. Traxler et al. also found items 12, 21, and 22 as favoring men. While we did find these three items as having medium-to-large DIF for some groups, we do not see any clear pattern to the groups they favored.

Scientific or scholarly significance of the study or work
The force concept inventory is the most commonly used research-based assessment in physics education research. Researchers have used data from the force concept inventory to investigate the impact of racism and sexism on student outcomes. However, the validity of these claims is limited by our lack of knowledge about biases on the assessments items across social identity groups. This research is also novel within the field of discipline based education research in its combination of an intersectional perspective, differential item function analysis, and item response theory.

Authors

• Ben Van Dusen, Iowa State University
  Presenting Author

• John Buncher
  Non-Presenting Author

• Jayson Nissen, Nissen Education Research and Design
  Non-Presenting Author

• Robert M. Talbot, University of Colorado - Denver
  Non-Presenting Author