Reading Between the Lines: Uncovering and Understanding the Backfire of a Middle School Literacy Intervention

• In Event: When Education Policy Backfires and What We Can Learn

Abstract

The increased use of standardized test and a focus on federal, state, and local accountability policies to drive student performance has led to the proliferation of interventions intended to improve student outcomes. While some of the incentives espoused by formal policies may create perverse responses and thereby hurt student outcomes, even well-intentioned programs may have led to worsened outcomes. In this study, I examine the design, implementation, and impact of a middle-school reading intervention that produced differential impacts on short-term measures of student performance by race.
The district (identify masked) is one of the largest in the nation, is immediately adjacent to a large metropolitan area, and is racially and socioeconomically diverse with nearly 40% each Black and White students and over 30% eligible for free- or reduced-price lunch. The policy in question used student performance on the 5th-grade Iowa Test of Basic Skills (ITBS) in reading to identify whether students would benefit from a middle-school literacy course in place of what would otherwise be an elective in a world language of their choice. Using a cutoff score of the 60th national percentile on the ITBS in reading, students below the cutoff were eligible for this supplementary course, while those who scored at/above the 60th percentile where ineligible.

The policy backfired in two ways: first, though the policy was designed to be binding, the district allowed for students to elect in and out of the requirement. Because the policy in practice was made not compulsory, the actual take-up of the intervention became fuzzy. Second, the policy tended to produce small positive impacts for White students, but medium to large negative impacts for Black students. But, Black students in this district had lower test performance to begin with. Therefore, the reading intervention program that set out to improve students with lower scores actually had the reverse effect – increasing the gap between Black and White students.

The main reason for this backfire appears to have been a myopic focus on using an objective criteria that could be applied district wide, without thinking about the within-school distributional consequences. This research emphasizes that while district-wide policies that rely on test-score cutoffs may appear to be unbiased, they may operate differentially at the school level. A second reason may be the discounting by policy makers of the potential for receiving a negative signal about one's ability – i.e., being told that they score below a certain cutoff. Going forward, policymakers should: 1) carefully consider how blanket policies - even when they rely on objective criteria - can have differing impacts when they are deployed at the building level, especially where there is considerably heterogeneity in the student population across buildings. 2) Policy makers should also be reminded that messaging and marketing are important elements of deploying policies, and that those responsible for enacting policies that impact students must buy into the plan if the policy is to be marketed authentically and convincingly.

Author

• Shaun Michael Dougherty, University of Connecticut