Propensity Score Methodology in the Study of Bias: The Case of Racial/Ethnic Bias in Mild Disability Labeling of Students

• In Event: 25.027 - Multimethod Approaches to Research for Social Justice and Equity in Education

Abstract

Quantitative approaches to data analysis for justice and equity can face significant challenges in examining racial/ethnic, social class, and gender bias in schools’ decisions on issues like track placement, disciplinary referral, and disability labeling. A central problem is the lack of data from randomized controlled trials (RCTs) which can isolating effects of ascriptive and other background traits that are hypothesized to be the sources of institutional bias. In the absence of such data, quantitative work often relies on large-scale non-experimental data that support multivariate regression models where potentially conflated measures are specified as competing predictors. This approach can result in significant unobserved estimation problems. This study illustrates the consequences of such problems and, more importantly, addresses an approach for advancement in their resolution. We demonstrate how recent adaptations of the “propensity score” (PS) procedure to bias research in labor economics is a robust means to utilize large-scale non-experimental data to estimate bias in schools’ decisions. This is a method that approximates an experiment to the best extent possible with non-experimental data. We focus on racial/ethnic bias in mild disability labeling in kindergarten and early elementary grades, using the federal Early Childhood Longitudinal Study (ECLS; n=21,190). In our PS procedure, the focal trait—minority status—is empirically specified as an experimental “treatment condition,” individual assignment to which is a function of plausibly conflating factors, such as socioeconomic status (SES), behavioral patterns, and academic achievement that are also pertinent to mild disability labeling (ECLS provides measures for a rich array of such conflating factors). This helps specify two groups—minority and non-minority students—with equal average propensity to be in the treatment condition (minority status) based on all other observed attributes pertinent to mild disability labeling. In simple terms, minority students are empirically aligned with non-minority peers who could have been minority students given all relevant observed attributes other than race/ethnicity. We then test mild disability labeling differences between these two groups. Under conditions of “no school bias,” the racial/ethnic difference should be null, as the groups are homogenized in all relevant observed attributes other than race/ethnicity. But we find evidence of strong racial/ethnic bias, which varies meaningfully by grade and disability type. We also highlight other potential applications of our approach.

Authors

• Argun Saatcioglu, The University of Kansas

• Thomas M. Skrtic, The University of Kansas