Research-Informed Coaching Toward Continuous Literacy Improvement

• In Event: Collaborating With Schools and Using Data to Reduce Inequities in Student Outcomes

Abstract

Objectives: Our goal is to assess the impact of the Middle School Network for School Improvement (MSNSI). The MSNSI is a group of school-based improvement teams, or groups of English language arts (ELA) and history teachers in a large urban district in California, serving grades 6–8. Each team is led by an improvement coach from an intermediary organization that facilitated the network and led this analysis. We will present an assessment of network participation for the 10 participating schools. In particular, we test (a) if network participation has a positive impact on students, and (b) the degree to which participation may or may not support mitigating negative impacts associated with COVID-19 learning gaps.

Theoretical Framework: This project follows an improvement science framework (Langley et al., 2009) in which teachers iteratively test and refine strategies to improve student outcomes using data to inform each inquiry cycle.

Methods and Data: The network includes teams from 10 of 22 middle schools; participating schools were randomly assigned to participate in the network as part of a randomized controlled trial testing the impacts of school-based improvement networks. We use district data on ELA course failures, state standardized test scores, and district benchmark data to assess the impact of participation in the intervention, and impacts for Black and Latinx students in particular, compared to peers in the control group. Initially, we create a multi-year “baseline” measure for course failures and proficiency rates on the ELA portion of the state test; this baseline is a composite measure of three years of pre-pandemic data (SY17, 18, and 19). We then compared intervention and control groups on baseline and post-pandemic performance, focusing on SY22, but also examining preliminary, benchmark data for SY23.

By Spring 2024, we will have sufficient data to conduct standardized effect size estimates (Cohen’s d using pooled standard deviations), looking at both ELA and math scores, and assessing the impact of being in the intervention group.

Results and Limitations: Initial results indicate that all schools in this analysis demonstrated a notable drop in performance (passing ELA courses and demonstrating proficiency on the state ELA assessment) during SY21, results associated with the COVID-19 pandemic. However, the 10 intervention group schools returned closer to baseline rates of ELA course passing than did control schools in SY22, with Black and Latinx students passing ELA courses at higher rates than before the pandemic in intervention schools. ELA proficiency rates on the state test are also closer to baseline in intervention schools. During SY23, benchmark data collected at three points during the school year showed intervention schools beginning below control schools in levels of proficiency on reading comprehension, but scoring higher than control schools in the second and third administrations. We will perform additional analyses and significance tests for the AERA annual meeting.

Significance: These results suggest that participation in improvement networks, including coaching and teacher collaboration, holds promise for not only improving school performance, but also for addressing COVID-19 related learning gaps, particularly for Black and Latinx students.
 

Authors

• Matthew J. Welch, American Institutes for Research

• Allison Belmont, American Institutes for Research

• Sami Kitmitto, American Institutes for Research