Measuring Implementation Fidelity: A Practical Example

• In Event: Linking Implementation to Impacts: Year 2 Results From the Mixed-Methods Evaluation of the Reading Recovery Investing in Innovation (i3) Scale-Up

Abstract

Typically, it has been assumed by education researchers that the application of an intervention as it is designed is a critical issue for the successful implementation of an evidence-based program. More recently, education research has begun to provide evidence of the relationship between implementation fidelity and impacts. Yet, within the field of implementation research, there is little agreement as to what fidelity is and how it should be measured. Additionally, there is a dearth of research that provides insights into the processes by which researchers have constructed and tested their fidelity measures.
The implementation of Reading Recovery under this i3 scale-up provides a rich context for beginning to explore the aforesaid issues in implementation research. This paper contributes to the conversation around connecting implementation to impacts by explicating the mixed-method approach that CPRE researchers have taken to understanding and measuring implementation fidelity. Specifically, this paper describes the process for designing, measuring, and refining indicators of implementation fidelity in order to provide a practical example for implementation researchers and program evaluators.
Leveraging the definition of fidelity posited by Abt, the technical assistance contractor for the i3 program, CPRE’s year-one study of the implementation of Reading Recovery centered on a fidelity framework that assessed structural indicators of implementation fidelity. As part of the effort to produce objective quantification of implementation fidelity, CPRE drew upon the Standards and Guidelines of Reading Recovery in North America (RRCNA, 2009), an implementation manual that codifies the essential components of the model. Surveys including items that operationalized the Standards and Guidelines were administered using Qualtrics survey software to each member of the Reading Recovery Network (e.g. Reading Recovery Teachers, Teacher Leaders, and Site Coordinators). Subsequently, survey responses were collected, coded for meeting (or not) each standard or guideline, and then used to compute a fidelity score for each site.
CPRE found that essential implementation and instructional standards were met with relatively high fidelity. Yet, year one analysis showed that there was no relationship between an aggregate measure of implementation fidelity and school-level impact estimates. However, qualitative work conducted during year one suggests that some aspects of implementation may be more important than others when trying to understand variation in impacts. For instance, many teachers downplayed the role of the site coordinator in relation to student outcomes, but emphasized the importance of the training experience. With this knowledge, CPRE has narrowed the fidelity framework to solely include those critical implementation factors that are introduced in paper two, described more fully here, and used in paper four to explain the variation in impact estimates.

Authors

• Michael Armijo, University of Pennsylvania

• Abigail Gray, University of Pennsylvania

• Philip M. Sirinides, University of Pennsylvania

• Jessica Gillespie, University of Pennsylvania