Evaluation of the Networks for School Improvement Initiative—Impacts on Student Outcomes

• In Event: AERA Roundtable Session Thursday 8:00 a.m. Four Seasons Ballroom 2-3
  In Roundtable Session: Networked Continuous Improvement Across Varied Contexts: Results from the First Three Years of the Networks for School Improvement Initiative (Table 1)

Abstract

The Bill & Melinda Gates Foundation established the Networks for School Improvement (NSI) initiative to support networks of schools in using continuous improvement (CI) to improve student outcomes. The foundation focused the work of the NSI on improving outcomes for students who are Black, Latino, or experiencing poverty. We present preliminary findings on the NSI initiative’s impacts during school years heavily affected by the COVID-19 pandemic and before schools’ third year of participation (when the foundation expected the NSI to achieve full impact).
After schools’ second year of participation in the NSI, the impacts on student outcomes were mixed. The 8th-grade on-track NSI—which primarily focused on improving teachers’ math or English language arts instruction—did not impact the academic or behavioral outcomes targeted by the initiative. The 9th-grade on-track NSI had a positive impact on three of the five targeted outcomes—GPA, core course pass rate, and credit completion. These NSI focused on a mix of strategies such as identifying students in need of academic support, developing relationships with students, and providing academic advising or tutoring. These findings are based on a matched comparison analysis but are consistent with the findings from a more rigorous randomized controlled design for a subset of the 8th- and 9th-grade on-track NSI. The well-matched postsecondary enrollment NSI improved FAFSA completion rates but did not have a statistically significant impact on college enrollment rates. The change ideas for these NSI focused on helping students navigate the college application and financial aid process.
The patterns of impacts over time also differed for the 8th-grade on-track and 9th-grade on track NSI. In schools’ first year of participation, the 8th-grade on-track NSI had a positive impact on GPA, math and ELA course pass rates, and attendance rates. These impacts were comparable in size to the impacts of the 9th-grade on-track NSI, but they did not persist. The year 1 impacts appear to be driven by schools that joined the NSI before the COVID-19 pandemic and whose year 1 outcomes were not affected by the pandemic. In contrast, the impacts of the 9th-grade on-track NSI on course-related outcomes appeared to increase from year 1 to year 2.

Authors

• Matthew Johnson, Mathematica Policy Research, Inc

• Jeffrey Max, Mathematica

• Tareena Musaddiq, Mathematica Policy Research, Inc

• Breyon Williams, Mathematica Policy Research, Inc

• Silvia Robles, Mathematica Policy Research, Inc

• Michelle Bennett, Mathematica Policy Research, Inc