Design2Learn: Year 3 of a Randomized Controlled Trial of an After-School Science Program for Middle-Grade Students

• In Event: Building a Stronger After-School: How Researchers and Program Staff Can Translate Data Into Impact

Abstract

Objectives and theoretical framework:

Design2Learn, funded through the US DOE’s Investing in Innovation fund, is an afterschool program for middle school students—a period in which performance in science often declines sharply. The program is aimed at increasing students’ interest, engagement, and academic performance in science. Students participate in weekly science instruction after school, facilitated by a three-person educator team (one certified teacher; two informal educators) designed to connect science learning goals to real-world activities. Students enroll in Design2Learn in sixth grade, and continue the program through eighth grade. The program combines three strategies: 1) collaborative teaching, which brings together certified teachers and staff employed by nonprofit organizations that provide school-based afterschool programs to jointly deliver instruction; 2) curricular bridging, which coordinates in-school and afterschool instruction to create seamless student learning experiences; and 3) design-based learning, which provides students with hands-on, real-world activities. This paper will describe the D2L model for teacher collaboration and how results from the first year of an implementation and impact study of the model were used in practice.


Methods:

This study is a randomized controlled trial with 32 afterschool programs located in NYC public schools randomized to either participate in D2L (treatment) or business as usual STEM programming (control). The confirmatory impact research question addressed by this study is: What is the impact of the opportunity to participate in Design2Learn for at least two years on 8th grade students’ interest and engagement in science? The implementation study also explores program fidelity of implementation and implementation facilitators and barriers.


Data Sources:

From students, we collected pre/post interest and engagement surveys, attendance (in the afterschool and in school), and science achievement (standardized tests and course grades) data. From educators, we collected end of year surveys to measure fidelity of implementation, changes in practice, and successes and barriers to implementation. We also conducted observations of treatment and control program activities and interviewed program staff to describe what D2L looks like ‘on the ground.


Results:

Early results from our impact analyses indicated that D2L did not significantly impact students’ interest and engagement in science. However, there are several issues to take into consideration when interpreting these findings. First, results are preliminary as they reflect only one year of implementation. The D2L model hypothesizes that the full impact of the intervention will not be seen until the end of the third year of participation. It is reasonable to expect that the model is stronger and that the sites have stronger implementation in the second and third years of operation. Further, the program developers made a number of refinements to the model based on their experience and lessons learned in the first year that may have improved educators’ ability to implement D2L in subsequent years. In fact, data from the Year 2 educator surveys suggest these changes were valuable.

In this presentation, the external evaluator and program developer will focus on how the findings were used to refine the model and implementation.

Authors

• Cheri Fancsali, Research Alliance for New York City Schools

• Ben Schwab, Research Alliance for New York City Schools

• Emma Banay, Expanded Schools