A Randomized Controlled Trial Examining the Impact of a District's Virtual, High-Impact Tutoring Initiative

• In Event: Emerging Evidence on Implementing High-Impact Tutoring at Scale

Abstract

Objective
We report results from a randomized controlled trial conducted with a large school district in Texas. The district is implementing virtual, synchronous tutoring to 3rd-8th grade students who did not “meet expectations” in either reading or math on the state test to satisfy the components of House Bill-4545. Students were randomly assigned to receive tutoring in the first wave (Spring 2022) or a future wave (a future school year). Our primary research question is: Does receiving virtual, high-impact tutoring in reading and math during Spring 2022 lead to corresponding improvements in student academic outcomes?

Methods & Data
Our partner school district is a large suburban school district in TX that serves 35,000 pre-K-12th grade students. The district partnered with a tutoring provider that provides live, online, comprehensive interventions including small group tutoring, assessments, and analytics.

This study seeks to explore the effects of synchronous, small-group virtual tutoring in 3rd through 8th grade math and reading on student academic performance and school attendance. Groups of assigned students met with a tutor virtually during the school day as part of a regularly scheduled intervention block.

5,349 students were identified by the school district as qualifying for math and/or reading tutoring based on their prior state standardized test scores. Students were then randomly selected to receive tutoring from within school, grade, subject blocks to meet enrollment quotas set by the district. As designed, students were assigned to receive tutoring in one subject area for 40 minutes, 5 day-a-week from the middle of January to early May.

Analysis & Preliminary Results
We have preliminary results on student participation in the program. Table 1 details characteristics of students who received tutoring. Of the 1,115 students who were scheduled for at least one session, 1,051 of them were on the initial randomized roster, suggesting that the randomization was successful. However, we find that participation in tutoring was low. In total, 890 students attended at least one tutoring session and only 15 students attended 30 hours of tutoring. Only 40.26% of the 43,812 scheduled sessions were attended.

Upon receipt of the achievement data in August 2022, we will estimate the impact of being assigned to receive tutoring during the Spring 2022 semester on students’ math and reading test scores (STAAR and NWEA MAP). We estimate both intent-to-treat and treatment-on-the-treated effects using student-level demographics and prior performance as covariates, school and grade level fixed effects, and cluster our standard errors at the school by grade level. We will present program effect estimates at the conference.

Significance
We report the results of our research partnership studying a district wide tutoring program in Texas. Our varied methods of analyses allow us a panoramic view into the implementation and outcomes of the tutoring program. Our preliminary results show a diverse group of students in need of support received tutoring. However, participation was much lower than district leaders intended. Future analyses will allow us to examine the causal effect of the tutoring program on student outcomes.

Author

• Carly D. Robinson, Brown University
  Presenting Author