Enhancing Student Engagement in Virtual Tutoring Sessions: A Randomized Control Trial of a Tutoring Intervention

• In Event: Equalizing Learning Opportunities Through High-Impact Tutoring: Results From Four Randomized Controlled Trials

Abstract

Objectives
We report results from a randomized controlled trial of an intervention to increase student engagement during virtual chat-based tutoring sessions. We conducted this study with a virtual tutoring provider serving a large school district. Prior correlational analyses by the provider suggest that students who earn higher average tutor-awarded participation points, given to students for demonstrating effort in sessions, experience greater growth on end-of-year (EOY) standardized tests. This experiment explores whether training tutors on how to provide participation-oriented feedback has a causal impact on EOY student outcomes. Our primary research questions are: Does receiving additional training on how to administer more specific feedback for participation points lead to an increase in student:
(1) engagement, measured by participation points?, and
(2) academic achievement, measured by Spring NWEA MAP scores?

Methods & Data
We preregistered our primary hypotheses and analysis plan on Open Science Framework.
Data come from a U.S. school district serving over 30k students from diverse backgrounds (See Table 1 for demographics). Participants include 447 tutors and 1,138 students in grades 3-10 (See Table 2 for grade breakdown) who received tutoring in Math and/or Reading in the 2022-23 school year.
We first randomized tutors into treatment and control groups. Treatment tutors received supplemental training on how to provide more specific feedback for receiving participation points. Control tutors continued to use existing procedures for administering participation points. Second, we matched students within school x grade strata using Fall MAP scores. One student per pair was then randomly assigned to treatment tutors and the other to control tutors.
After excluding students who never received services, or who were missing outcome data, our final sample consists of 307 and 342 students in the treatment and control group, respectively. There is no statistical evidence for differential attrition across treatment groups (p = .298).

Analysis & Preliminary Results
We use an OLS linear regression to estimate the impact of student treatment assignment on average participation points and Spring (EOY) Math and Reading scores. We control for student-level covariates including Fall MAP scores, add fixed effects of (school x grade) strata, and cluster residuals by strata. As summarized in Table 3, students assigned to the treatment group had significantly increased average participation points (Figure 1) and Spring Reading scores (Figure 2).
Because students sometimes worked with tutors not in their assigned condition, we also conduct an instrumental variable regression analysis where we instrument treatment assignment on the percent of sessions students completed with a treatment tutor. As shown in Table 4, we find that, among compliers, having completed a greater percentage of sessions with a treatment tutor leads to significant increases in average participation points and EOY Reading scores.

Significance
Virtual tutoring has become a widely-adopted means for providing students access to high-impact tutoring at scale. However, questions remain about the quality of students’ engagement and the general magnitude of its benefits. This intervention offers a concrete strategy for tutors to enhance student engagement in virtual tutoring programs, ultimately improving program efficacy and benefitting student outcomes.

Authors

• Carly D. Robinson, Stanford University

• Susanna Loeb, Stanford University

• Emma Strouse, Stanford University