BackHome
Refresh: OffView Personal Schedule
Paper Summary

Integrated Findings From a Mixed-Method, Longitudinal, Multisite Evaluation of Supplemental Educational Services

Sat, April 14, 12:25 to 1:55pm, Vancouver Convention Centre, Floor: First Level, East Ballroom C

Abstract

This paper draws on data from an ongoing, mixed-method multi-site study of the implementation and impact of supplemental educational services. The central purpose of this study is to understand whether and how providing students with academically focused out-of-school tutoring in reading and mathematics contributes to improvements in their academic performance, specifically in reading and mathematics. We are conducting integrated quantitative and qualitative investigation in five urban school districts: Milwaukee, Wisconsin; Minneapolis, Minnesota; Chicago, Illinois; and Austin and Dallas, Texas. We are conducting in-depth qualitative inquiry designed to define key elements of SES program models and the policy and practice variables that mediate implementation, as well as quantitative inquiry investigating selection into SES (i.e., who registers and participates) and SES program impacts, using propensity score matching and fixed-effect methods with nonequivalent (internal) comparison groups.

The quantitative component of the study examines whether eligible students participating in the program demonstrate gains relative to eligible students not participating in SES, and whether and how impacts differ based on students’ grade level, intensity (hours) of service, and selectivity into different kinds of programs and SES providers. Quantitative findings from the first two years of the study indicate a strong relationship between the intensity of SES—the number of hours students attend—and its effectiveness. SES provider hourly rates directly influence the number of hours of SES students can attend. In addition, we find that students attending SES with on-line providers and for-profit providers gain less in reading and math as compared to other off-line and not-for-profit providers, respectively. In general, the effectiveness of SES has been limited, producing primarily small effects for primarily elementary-aged students who get a sufficient number of hours of tutoring.

In addition to statistically assessing SES impacts, this study examines questions about SES implementation that are well answered by qualitative methods. Qualitative findings from the first two years of the study indicate a mixed quality of tutoring instruction, often mirroring more traditional instructional techniques (e.g. reliance on worksheets and drills). Instruction is rarely specialized towards serving students with special needs (e.g. ELL and students with disabilities). Districts and providers face many common challenges in implementing SES, including enrollment and registration, as well as navigating policy at the federal, state and local levels. Qualitative data includes interviews with district and state administrators, interviews with directors of tutoring programs, interviews with tutoring staff, parent focus groups, systematic observations of tutoring practice, provider records, and district and state records.

The data generated by these questions contributes to a nuanced interpretation of effects as well as a deeper understanding of conditions at the classroom, provider, district, and state levels that may contribute to these effects. This study leverages the unique, mixed-method design to bring important insight to the landscape of SES across various sites and contexts. The level of nuance and depth in this analysis helps illuminate the complex interactions of policy, contexts, practice and outcomes.

Authors