Lessons Learned: Design Challenges and Equity Considerations of a Randomized Controlled Trial in Afterschool

• In Event: Square Peg, Round Hole: Negotiating the Appropriateness of "Gold Standard" Research Design for Out-of-School Time

Abstract

This presentation offers a deep-dive into lessons learned from a completed RCT conducted on a middle school STEM afterschool program. Key aspects of the program model included collaborative co-teaching between a school-day educator and afterschool instructor; curricular bridging between the school day and afterschool settings; and design-based instruction. The RCT (funded and required by the US Department of Education’s Investing in Innovation program) sought to understand whether this program model increased student achievement, engagement and interest in STEM compared to a control group. Results were mixed, with significant effects around student exposure to design-based instruction but no significant effects found in student outcomes, including interest and engagement in STEM. However, throughout implementation of the study, multiple challenges arose posed by core tenets of out-of-school time settings that substantially constrained the ability of the model and the study to accurately reflect OST common practice. Some examples include a requirement for students to remain in the program over multiple years and a lack of flexibility for educators to adapt lesson plans. Further, the study was challenged by difficulties obtaining data and maintaining relationships with control sites, as well as a lack of knowledge of the types of programs run at control sites, many of which were led by partner nonprofits who contribute to a thriving OST STEM ecosystem throughout NYC. Presenters will explore pain points and benefits of the RCT in this context and extrapolate recommendations for future RCTs in an OST setting, notably a middle school program.

Authors

• Emma Banay, Expanded Schools

• Fran Agnone, ExpandED Schools

• Cheri Fancsali, Research Alliance for New York City Schools