Issues With Fidelity of Implementation of Educational Interventions in Out-of-School Time

• In Event: The Unique Challenges of Conducting Evaluations of Out-of-School-Time Programs: Lessons Learned From the Field

Abstract

When designing evaluations of school-day programs, researchers can rely on the assumption that students will receive a level of dosage aligned to average school-day attendance rates (>90%; NCES, 2008). However, given the ‘drop in’ mentality of many OST programs, there is much greater variation in attendance, thus increasing the need to consider dosage as a mediating variable. In addition to within-year variations, cross-year longitudinal studies designed to assess the impact of staff training on student outcomes must also consider substantial differences in staff turnover rates in OST (upwards of 40% annually) relative to school-day settings (10%; NCES, 2015). The researchers will outline these challenges as well as provide some methodological considerations that evaluators can use when designing such studies to minimize their impact on a study’s power. They will describe how these issues are being addressed in the context of randomized control trial of a STEM-focused middle school OST program funded via a US Department of Education i3 grant.

Authors

• Kathryn Brohawn, ExpandED Schools, Inc.

• Lizzie Murchison, ExpandED Schools