A Framework for Incorporating Intervention Fidelity in Educational Evaluation Studies

• In Event: New Norms for Educational Research

Abstract

Many researchers consider the randomized controlled trial (RCT) to be the gold standard for rigorously evaluating interventions (IES, 2014). While the RCT is a crucial tool for understanding the average causal impact of educational interventions, it does little to illuminate the processes through which an intervention impacts outcomes—leading some to label randomized experiments as a “black box” (Cook, 2003; Imai et al., 2011). To open and explore this black box, there is growing interest in collecting data on intervention fidelity (i.e., to what extent was the intended intervention actually delivered) to inform our understanding of how, and under what conditions, interventions work (Bloom, 2005).
Assessing intervention fidelity will help researchers understand and interpret impact estimates, as well as assess the generalizability of their results. Intervention fidelity assessment and analysis also serves to test plausible mechanisms for interventions effects, which can help protect against spurious findings. However, if fidelity measures are added to evaluation studies ad hoc and without careful planning, they may fail to capture requisite information about the implementation of the intervention and the nature of the treatment contrast. Additionally, without carefully planned assessment of fidelity, researchers may not be able to evaluate and further develop the conceptual models of the intervention.
Although funding agencies and education researchers generally accept the importance of both assessing intervention fidelity and linking fidelity assessments to outcomes of interest (IES, 2014), little information exists about the appropriate procedures for accomplishing these goals (Nelson et al., 2012; O’Donnell, 2008; Weiss et al., 2014). This presentation presents a framework to help researchers conceptualize, measure, and analyze the fidelity of specific interventions. First, we describe the framework conceptually, explaining each step. We then use the framework to give a more detailed example of an intervention evaluation.
Our framework decomposes fidelity assessment into five steps: 1) define intervention logic models (conceptual, operational), 2) identify fidelity measures, 3) conduct psychometric analyses of fidelity measures, 4) conduct within- and between-group fidelity analyses, and 5) link fidelity to outcomes. As presented in Table 2, this assessment framework provides a useful and systematic way to assess intervention fidelity in educational settings. It also helps to clarify the conceptual model of an intervention, and allows researchers to organize and identify indicators needed to measure fidelity. The framework also emphasizes the investigation of reliability and validity information to maximize the usefulness of measures. The common need to combine indices further emphasizes the need for good measurement practice and psychometric techniques. Finally, if the first four steps have been effectively executed, the fifth step allows researchers to test whether or not fidelity affects outcomes. As a result, the five-step framework combines theoretical, methodological, and analytical approaches to bolster the validity arguments made about an intervention's effectiveness. In the presentation, we will leverage data from the implementation of a previously published (Hulleman et al., 2016) online motivation intervention for undergraduate general education students to demonstrate the usefulness of the framework.

Table 2. The Five-Step Framework of Fidelity Assessment.


 

Authors

• Chris S. Hulleman, University of Virginia

• William Madison Murrah, University of Virginia

• Jeff John Kosovich, University of Virginia