Adapting Lessons Learned: An RCT–Turned–Quasi-Experimental Design That Developed Buy-In and Adapted to Local Context

• In Event: Square Peg, Round Hole: Negotiating the Appropriateness of "Gold Standard" Research Design for Out-of-School Time

Abstract

This presentation describes an RCT that was designed with lessons learned from Topic Area 1 in mind, and that was ultimately adapted to a quasi-experimental design to fit the needs and limitations of the particular OST context. This study was designed for a high school level work-based learning program (funded by the US Department of Education’s Education Innovation and Research program) led by community-based organizations throughout NYC. The originally designed RCT sought to understand where student outcomes improved as a direct result of participation in the program compared to students in a control group. Strategic decisions were made to mitigate limitations that emerged during the study described in Topic Area 1. One core aspect of this was engaging community input throughout the research implementation process, with a recommendation to engage practitioners even earlier in the design process. Another example included randomizing at the student level instead of the site level, which shifted the comparison away from partner nonprofit organizations. While this mitigated challenges at the site level, this randomization design presented new challenges in recruitment and admissions, including challenges generating sufficient and equitable demand for the program, maintaining student engagement throughout the application process, and ensuring clarity in eligibility requirements. Ultimately, due to these challenges in randomization, the design was shifted to quasi-experimental, which further allowed students who simply didn’t complete the full application process to be included in the study. This mitigated equity concerns and offered additional opportunities for adaptations in research measures as the program evolved. Early-stage results of the quasi-experimental study are presented with considerations for next-phase research adaptations. Presenters will then discuss implications and recommendations for broader OST contexts.

Authors

• Candace Brazier-Thurman, ExpandED Schools

• James J. Kemple, New York University