3-110 - RCTs of Educational Programs in Pre-, Elementary, and Middle Schools: Explaining Variation in Treatment Effects across Sites

Session Type: Paper Symposium

Integrative Statement

Randomized controlled trials are increasingly used to evaluate promising educational programs, but the results are often not as expected, with many RCTs producing largely inconsistent effects. In the past year, two reports were prepared on variation in program effects in education, commissioned by the William T. Grant Foundation (Weiss, Bloom, & Brock, 2013) and the Institute of Education Sciences (Schochet, Puma, & Deke, 2014). These reports offer recommendations responding to increasing concerns about the focus on average program effects in the face of research indicating (1) programs vary in their effectiveness, (2) a program that works for one group may not work for another, and (3) a program that is effective under certain conditions may not be effective under other conditions. The goal of this symposium is to identify factors that explain why program impacts vary across sites.

The first paper will examine setting characteristics to explain variation in treatment impacts found across 10 preschool settings for two professional development interventions designed to improve the quality of early childhood teachers’ interactions with children. The second paper will present the effects of a social-emotional and literacy program on student outcomes from 3rd through 5th grade and the role of implementation and student/teacher mobility in variation in intervention effects across schools over time. The third paper will examine variation in program effects of a discussion-based vocabulary program for students in 4th through 7th grade, including characteristics of classrooms (e.g., behavior management, teacher stress, teacher-student relationships) and schools (e.g., teacher turnover, school disorder).

Sub Unit

• Education, Schooling

Chair

• Maria D LaRusso, Harvard University

Discussant

• J. Lawrence Aber, New York University

Individual Presentations

• Variation in Impacts of a College Course and MyTeachingPartner Coaching Model on Preschool Children’s Development - Presenting Author: Bridget K. Hamre, University of Virginia; Robert C Pianta, University of Virginia; Jamie DeCoster, University of Virginia; Jason Downer, University of Virginia

• Three-Year Impacts of the 4Rs Program on student outcomes: Variation by implementation and teacher/student patterns of mobility - Presenting Author: Stephanie M Jones, Harvard Graduate School of Education; Joshua L Brown, Fordham University; J. Lawrence Aber, New York University

• Variation in Impacts of a Discussion-Based Vocabulary Program for Early Adolescents: Examining school and classroom conditions - Presenting Author: Maria D LaRusso, Harvard University; Stephanie M Jones, Harvard Graduate School of Education; Luke Miratrix, Harvard University Department of Statistics; Catherine Snow, Harvard School of Ed; Suzanne Donovan, Strategic Education Research Partnership