Development of a Survey-Based, Network Health Learning System

• In Event: Organizing Networks to Accelerate School Improvement in Pursuit of Equitable Educational Opportunity

Abstract

Objectives or purposes:
The Networks for School Improvement (NSI) Initiative is a first-of-its-kind effort to support a collective of improvement networks towards addressing educational inequities. Given the complexity of improvement networks, their leaders need information about the functioning and health of their organizations to make strategic decisions. In this paper, we describe the development of a network health survey and reporting system designed to support leaders’ decision-making.

Perspective(s) or theoretical framework:
We draw from the literature on improvement networks to derive key factors that should be considered when designing network health reporting systems. For example, improvement networks vary in terms of their objectives and their design (Kinlaw et al., 2020) which necessitates a reporting system that can accommodate a variety of network structures and naming conventions. Also, NSI are nested organizations with individuals clustered within teams, within a broader network (Authors, 2019), which suggests the importance of reporting at various hierarchical levels. Finally, leaders of these networks are in new conceptual terrain (Peurach, 2016), requiring an educative component of reporting to provide a conceptual anchor to these leaders.

Methods, techniques, or modes of inquiry:
The survey and reporting design process followed established design research processes (see, e.g., Brown, 2011) moving iteratively through idea generation, prototyping, analysis of pilot data, user testing, and implementation.

Data sources, evidence, objects, or materials:
To refine our report design, we draw on data from prior administrations of the survey (which has been administered to over 30 networks and 2000 individuals), discussions with network leaders about their interpretations of the report, and the prior experiences of members of the authorship team.

Results:
Our research and design process yielded five key principles that undergird the system. First, to support readers in learning about network health, we tie all the results in our reports to a conceptual framework and associated visual. Second, to reduce cognitive load, we provide access to many measures while using visualization strategies to highlight the most important signals. Third, because NSI have differing structures and objectives, we offer bounded customization in our survey that allows users to apply the specific language of their networks to a standard set of questions. Fourth, we rely on longitudinal comparison within networks and across networks to provide a means for substantive interpretation. Fifth, because of the nested structure of NSI, we provide users with results broken down at the level of the network, at the level of schools, and across various subgroups within the network (e.g., network roles).

Scientific or scholarly significance of the study or work:
The popularity of improvement networks, and the significant investment that is being made in them by philanthropies and governments, suggests the need to support the leaders of these networks in making informed decisions. The key design elements of the visualization system shared here promise to support others seeking to inform and accelerate the work of improvement networks in other contexts, with an emphasis on visualizing variation in performance for the purpose of learning not accountability.

Authors

• David Gilbert Sherer, Carnegie Foundation for the Advancement of Teaching
  Presenting Author

• Angel Yee-lam Li, Carnegie Foundation for the Advancement of Teaching
  Non-Presenting Author

• Jennifer Lin Russell, Vanderbilt University
  Non-Presenting Author

• Mai Ahn Bui
  Non-Presenting Author

• Anthony S. Bryk, Carnegie Foundation for the Advancement of Teaching
  Non-Presenting Author

• Stuart Luppescu, University of Chicago
  Non-Presenting Author