Evaluation of the Networks for School Improvement Initiative—Impacts on Student Outcomes Final Report

• In Event: Networked Continuous Improvement at Scale: Research Findings and Reflections from the NSI Initiative

Abstract

The Gates Foundation established the Networks for School Improvement (NSI) initiative to support networks of schools in using continuous improvement to improve student outcomes for students who are Black, Latino, or experiencing poverty. Each NSI focused its grant on improving student outcomes in one or more of the following areas:
• 8th- or 9th-Grade on track: The proportion of 8th- or 9th-grade students who meet a set of academic and behavioral outcomes related to high school graduation and college enrollment
• College-Ready on track: The proportion of 11th- and 12th-grade students who are on track academically to enroll in a college with a graduation rate of at least 50 percent
• Well-Matched postsecondary enrollment: The proportion of 12th-grade students who complete the steps needed to enroll in a college with a graduation rate of at least 50 percent
In this paper, we examine the impact of the NSI initiative on students (compared to students in similar schools that did not participate) for each outcome area. Further, we utilize measures of school engagement in network activities and supports and implementation of continuous improvement to understand the relationships between these activities and outcomes.

Analysis of the NSI initiative’s impacts during schools’ first and second years of participation (which were heavily affected by the COVID-19 pandemic) found that the NSI initiative had mixed impacts on student outcomes after NSI schools’ second year of participation. The 8th-grade on-track NSI—which primarily focused on improving teachers’ math or English language arts instruction—did not impact the academic or behavioral outcomes targeted by the initiative. The 9th-grade on-track NSI had a positive impact on three of the five targeted outcomes—GPA, core course pass rate, and credit completion. These NSI focused on a mix of strategies such as identifying students in need of academic support, developing relationships with students, and providing academic advising or tutoring. The well-matched postsecondary enrollment NSI improved FAFSA completion rates but did not have a statistically significant impact on college enrollment rates. The change ideas for these NSI focused on helping students navigate the college application and financial aid process.
This final study report will examine impacts of the NSI through schools’ fourth year of participation, after the height of the COVID-19 pandemic. The report will also provide information about the aspects of the NSI approach that were related to impacts on students. This analysis will focus on schools’ engagement in the network and supports and their engagement in continuous improvement activities.
These findings fill an important gap in the evidence as this is one of the first large-scale studies to measure the impact of supporting networks of schools in using continuous improvement (Feygin et al. 2020; Garet et al. 2021). By integrating the analysis of outcomes with measure of engagement in network activities and supports and implementation of CI, we provide useful information for intermediaries, school districts, and schools considering initiatives that bring together networks of schools focused on using continuous improvement to improve student outcomes.

Authors

• Matthew Johnson, Mathematica Policy Research, Inc

• Naihobe Gonzalez, Mathematica Policy Research, Inc

• Jeffrey Max

• Tareena Musaddiq, Mathematica Policy Research, Inc

• Sophia Seifert, Mathematica Policy Research, Inc

• Michelle Bennett, Mathematica Policy Research, Inc