An Empirical Examination of The Relationship Between Improvement Network Development and Network Outcomes for Students

• In Event: AERA Roundtable Session Thursday 5:25 pm Four Seasons Ballroom 2-3
  In Roundtable Session: Improvement Network Development: Exploring the Promise of Collaboration Among Schools to Support Educational Improvement (Table 11)

Abstract

Objectives:
Educators are increasingly participating in improvement networks, but conceptual and methodological challenges have impeded the field’s capacity to explore the impact of these efforts (Poortman et al., 2022). This paper examines evidence of networks’ impacts on student outcomes. Given our clear conceptualization of network development grounded in a theory of network health and development and a validated measurement system, we have an opportunity to explore an important question – can productive improvement networks produce positive student impacts?

Theoretical framework:
Our INHD Framework conceptualizes six domains of a mature and healthy improvement network (e.g., hub leadership & continuous improvement processes) that are associated with short- and long-term outcomes (Authors et al., forthcoming). Ultimately, we expect that in networks that are more developed across these domains: 1) members perceive greater benefits; 2) this sustains their participation over time; and 3) this sustained activity culminates in measurable improvements on the network’s expressed improvement aims for students. This paper examines this set of linked hypotheses.

Methods:
This mixed methods paper relates statistical analyses of longitudinal INHD survey data (see paper 1), a measure of member participatory benefits (see paper 2), and school-level standardized effect estimates. We further conduct longitudinal qualitative analysis of network archival documents (see paper 1). Ultimately, we integrate these strands of analyses to explore the relationship between key components of improvement networks and valued student outcomes.

Data sources:
In addition to the data sources described in the papers above, this analysis includes school-level impact estimates collected by an independent evaluation organization. Impact estimates for some network schools resulted from propensity score matching, in which network schools were matched to non-network schools in the same region using a set of standard characteristics. Estimates for other network schools resulted from an RCT, in which schools were randomly assigned to network participation or control status. Outcomes vary by network aim but may include GPA, attendance, FAFSA completion, and/or college enrollment.

Results:
Preliminary analyses indicate substantial variation in member participatory benefits that are further related to network health and development. Specifically, members in the most developed networks in the initiative in 2023 had significantly more positive perceptions of participation benefits than those in the least developed networks. Ongoing analyses will explore potential impacts on student outcomes.

Scientific or scholarly significance of the study or work:
Promising claims about the potential of improvement networks have appeared in the literature, and the scope of this activity has spread quickly in practice. Thus, it is crucial that we amass empirical evidence assessing the potential contributions of improvement networks to valued outcomes and factors that mediate those contributions. This is the first large scale body of evidence about the development of improvement networks in education and their possible impacts on educators and students. As such, this study promises to yield useful evidence for both researchers and practitioners about what may work for whom under what set of conditions.

Authors

• Anthony S. Bryk, Carnegie Foundation for the Advancement of Teaching

• Megan Duff, UNC Chapel Hill

• Angel Yee-lam Li, Carnegie Foundation for the Advancement of Teaching

• Stuart Luppescu, UChicago Consortium on School Research

• Chris Matthis, Partners for Network Improvement