Identifying the Critical Components of Values Affirmation: Fidelity of Implementation in a Large-Scale Randomized Controlled Trial

• In Event: Scaling Up a Social-Psychological Intervention to Combat Stereotype Threat and Narrow Achievement Gaps

Abstract

Objectives

Prior research suggests values-affirmation can serve as a simple yet powerful tool for reducing achievement gaps (Cohen et al., 2006; 2009; Sherman et al., 2013; Miyake et al. 2010). Small samples, highly controlled implementations, and close monitoring of teachers delivering the intervention have ensured high levels of fidelity to the laboratory ideal. Our district-wide randomized trial of values-affirmation revealed important, but relatively modest, impacts. In this paper, we investigate fidelity of implementation as an explanation for the modest effects, and we identify the most critical components of implementation for affecting student outcomes.

Perspectives

Evaluation of fidelity establishes internal validity, highlights opportunities for adaptation, and identifies critical components of an intervention (O’Donnell, 2008). Fidelity is of particular importance with regard to social-psychological classroom interventions, as affirmation theory points to many subtle ways in which values affirmation can be undermined, including: poor timing (Steele, 1988; Critcher et al., 2010); differences in school context (Yeager & Walton, 2011); awareness of the participants to the intervention’s potential benefits (Sherman, et al., 2009); and the nature of student engagement (Shnabel, et al., 2013).

Methods

Fidelity analyses require explication of a “gold-standard” for implementation, or an idealized model of the intervention against which a real-world replication may be compared (Hulleman & Cordray, 2009). We describe and compare Cohen et al.’s (2006; 2009) ideal design and implementation protocols to our own with attention to timing, dosage, and participant responsiveness, as these are important moderators of the impacts of values affirmation (Critcher et al. 2010; Sherman et al. 2013). We create fidelity indices to estimate adherence to the ideal, and estimate the moderating effect of these factors on the observed impacts.

Data Sources

We draw on qualitative data gathered from the schools, including end-of-year teacher surveys and classroom notes about each of the four intervention administrations, as well as achievement data from the school district, and data on student participation and engagement from a content analysis of student responses.

Results

Overall, our implementations across the school year were close to the ideal. However, implementation departed from the gold standard in one important way: researchers had less control and teachers implementing the intervention had more. We find, therefore, greater variability in the implementation processes across classrooms. Using an index of participant fidelity, we find high levels of fidelity to the cognitive process of values affirmation among treated students. However, participant fidelity and participation decline with each passing administration. This pattern was mirrored by the declining rates of teacher fidelity to implementation processes.

Significance

Our findings suggest the critical importance of facilitation strategies that may help teachers implement the intervention in the future with greater fidelity. Previous studies have suggested that increasing students’ exposure to values affirmation beyond the initial treatment has little incremental effect on students’ academic outcomes. Our analysis suggests that declining fidelity beyond the initial treatment, caused by both decreased engagement among teachers and students in the affirmation process, may in part explain this lack of a dosage effect found in our study and prior work.

Authors

• Dominique Bradley, University of Wisconsin - Madison

• Evan Crawford, University of Wisconsin

• Sara Elizabeth Dahill-Brown, Wake Forest University