Measuring the Impact of Improving literacy Instruction Through Networked Improvement Community of Assistant Principals

• In Event: The Successes and Challenges of Measurement in School and Out-of-School Improvement Research

Abstract

A Networked Improvement Community (NIC), if well structured, can be a powerful vehicle to advance learning and forge continuous improvement in practice (Biag & Sherer, 2021; Bryk et al., 2015; Gomez et al., 2016; Noble et al., 2021). When coupled with professional learning and application of Improvement Science practices, a NIC can be transformational: individually, collectively, and organizationally.

In fall 2023, the presenter and team formed a NIC for 20 assistant principals who were supporting implementation of a new literacy curriculum in their urban district. While literacy coaches assisted the schools’ new curriculum rollout, the research team focused the NIC work around surfacing the implementation problems they were experiencing, particularly in engaging teachers in continuous improvement. NIC consisted of 10 seminar sessions and 4 1:1 facilitator consultations over a six-month period (January-June 2024), structured around Improvement Science principles and the application of several new instructional improvement strategies.

Kirkpatrick’s evaluation framework was used to evaluate the impact—participant reaction, learning, and application and organizational change (Kirkpatrick & Kirkpatrick, 2007). Data were collected through APs’ feedback survey responses on reactions and learning; a pre-post survey of improvement leadership proficiency and organizational practices; and tracking improvement work through individual workbook documentation of each phase of their problem analysis and PDSA cycles pursued. Student-level assessment results will be collected this fall. Feedback and workbook documentation were qualitatively analyzed for themes related to the NIC and improvement work, and the survey was quantitatively analyzed for changes in proficiency-related practices. Additionally, responses from open-ended feedback surveys at the end of each session were analyzed to uncover what the APs learned and planned to do next, the insights they gleaned from each other, and how the seminar discussions shifted their assumptions about continuous improvement work.

The pre-post improvement leadership proficiency results showed that the APs improved in 9 of the 10 practices, with the greatest gains in related to collecting and using qualitative and quantitative data to examine a problem of practice, working with teachers to find solutions to student learning needs, and persisting in finding solutions through trial-and-error learning.

Results from the workbook analysis revealed that the APs were able to identify a problem of practice, unpack the factors that contributed to the problem, identify a high-leverage focus area, and try out solutions with a subset of their staff, such as strategies on learning to use student assessment data to plan learning activities, form small groups and monitor student progress. The APs often worked with a small group of teachers, focusing on individual changes and monitoring the results, whereby they learned to track the implementation and consequences of a solution over time.

Taken together, the findings underscored what was learned through the NIC experience, how what was learned was applied and the changes in school practice. The tools and measures developed for this inquiry proved to be robust and useful, both during the NIC experience to guide facilitation, support and communication among NIC members, and summatively, to evaluate what was learned and accomplished, serving the district’s improvement priorities.

Author

• Margaret Terry Orr, Fordham University