Measuring Implementation Fidelity: Decoding a Multifaceted Early Childhood Initiative

• In Event: Roundtable Session 25
  In Roundtable Session: Using Fidelity Data to Improve Initiatives for Early Childhood Teachers

Abstract

Comprehensive early learning initiatives will have limited value without an understanding of what it takes to replicate them. As practitioners and stakeholders seek to replicate and scale effective interventions, evidence about implementation is needed to inform implementation strategies and support continuous improvement (Fixsen et al., 2005). However, few programs have data about which program components are critical and which can be adapted without jeopardizing outcomes. This paper discusses how the FMTI’s programmatic components were operationalized and how implementation fidelity was systematically measured.
SRI is conducting a randomized controlled trial to determine FMTI’s impact on PreK-3 teachers and their students. The study also examines the extent to which the program was implemented as intended. To measure implementation fidelity, researchers first collaborated with program developers to develop a logic model that specifies how key programmatic components are expected to produce changes in teacher practice, school climate, and student performance. Researchers and developers then operationally defined the components using quantitative benchmarks that measure the “active ingredients” of each component, and set thresholds for inadequate, low, medium, and high fidelity. Activities considered more important for improving outcomes were assigned more points. Additionally, fidelity measures were created at different levels (e.g., teacher, school, population of intervention schools) to enable an examination of the relationship between fidelity and outcomes at each level.
Fidelity data come from a variety of sources, including training sign-in sheets and administrative data. Analyses of data showed considerable variance in implementation fidelity across components. For example, while 95% of schools had medium or high fidelity for the teacher fellows program, only 75% did so for the principal fellows program. Further, there was variation across levels. While 100% of teachers had medium or high fidelity to the master’s degree program, only 55% of schools did so for this component.
Having fidelity data provides many benefits. It informs program developers so they can make mid-course corrections. It helps explain results from the experimental study, illuminating if some components matter more than others, or why some schools have stronger outcomes than others. It enables more accurate interpretations of outcomes (e.g., a lack of outcomes may be due to ineffective implementation). Measuring fidelity can also inform replication and scale-up while avoiding program drift that can lead to poor outcomes.
Creating valid, reliable, and practical measures of fidelity is not a simple task. As developers change program components, partly in response to evaluation feedback, the fidelity instrument must change. Other challenges include determining how to measure the fidelity of elements that occur at different time intervals (e.g., master’s degree students are expected to complete a certain number of courses annually, but they are expected to lead a professional community at any point during their program), and determining how to handle attrition so that a program is not repeatedly penalized for dropouts.
This paper will present the fidelity instrument developed and data collected, examine the affordances and challenges of measuring implementation fidelity, and discuss the benefits of including measures of fidelity in program evaluations.

Authors

• Marjorie E. Wechsler, SRI International

• Shari Golan, SRI International