Measuring Implementation of Education Interventions: A Five-Step Model of Fidelity Assessment

• In Event: Fidelity of Implementation: Identifying Common Components and Sources of Variation in Implementation and Intervention Effects

Abstract

1. Objectives or purposes

Educational environments are complex, and interventions designed to improve the context are often costly. Due to this complexity, interventions aren’t always implemented as planned and, unfortunately, the null result is a frequent finding in education research. However, this doesn’t mean that we can’t learn from what happened (Bloom, 2005). Instead, by leveraging assessment to capture intervention implementation, we can learn more from educational experiments, even those that don’t impact the outcome. The intervention implemented in an experiment frequently differs from the intervention that was designed, making it unclear whether unfavorable results are due to an ineffective intervention model or the failure to implement the model fully. It is vital to accurately assess fidelity of implementation and, where possible, to incorporate fidelity data in the analysis of outcomes.

2. Perspective(s) or theoretical framework

In this paper, we present a five-step model of fidelity assessment that highlights the assessment of core components of educational interventions (Hulleman et al., 2013; Nelson et al., 2012). Importantly, we emphasize not only capturing the extent to which the intervention was implemented as designed (i.e., fidelity), but also the extent to which the implemented intervention differs from the counterfactual or comparison group (i.e., differentiation). Measuring such differentiation, also known as achieved relative strength (Hulleman & Cordray, 2009), is key in understanding the relative impacts of interventions, and how developers and practitioners should proceed in modifying and/or implementing the intervention in the future.

3. Methods, techniques, or modes of inquiry

All data presented in this paper were collected as part of randomized experiments, and include both qualitative and quantitative measures.

4. Data sources, evidence, objects, or materials

The assessment model draws on examples from randomized experiments in education settings, including those targeting high school science instruction, elementary mathematics instructional practices, preschool science curriculum, and parent-child interactions.

5. Results and/or substantiated conclusions or warrants for arguments/point of view

N / A

6. Scientific or scholarly significance of the study or work

This presentation serves as a how-to-guide for fidelity assessment of educational interventions. As such, it is designed for program developers, researchers, evaluators, and practitioners who are interested in developing, testing, and implementing interventions. This paper provides a systematic framework to help evaluators and developers enhance what they learn from evaluation of educational interventions.

Author

• Chris S. Hulleman, University of Virginia