Individual Submission Summary

Direct link:

Boosting Middle School Math Learning With Online Homework

Wed, April 7, 4:20 to 5:50pm EDT (4:20 to 5:50pm EDT), Virtual


Only about one-third of U.S. eighth-grade students have reached the proficient level on the last three administrations of the National Assessment of Educational Progress (NCES, 2020). [Program], a free web-based platform, provides opportunities for students to learn more from math homework independent practice. The platform is used by 11,000+ teachers (fivefold the number of teacher users prior to COVID-19 school closures), with students solving 30 million problems in the 2019-2020 school year.

Unlike intensive interventions that require dramatic shifts in instruction, [Program] requires teachers to change relatively little. The tool can embed any curriculum and other instructional materials teachers use. [Program] offers a unique library of problems for teachers to create assignments and provides students with immediate feedback as well as supplemental tutoring support. Performance reports flag common wrong answers, helping teachers pinpoint shared areas of struggle and facilitate guided class discussions.

A prior study involving 43 schools found students who used [Program] for homework for a school year significantly outperformed students in the control group on a standardized math assessment (g = 0.22). The finding supported the launch of this replication study to determine whether the evidence will generalize in a different setting with a more diverse sample. The hypothesis is that students who used [Program] to support homework would show a greater increase in math achievement than a business-as-usual group.

Study Population
The sample in the replication study is comprised of 63 diverse schools of 5000+ 7th-grade students in North Carolina (14% Hispanic, 20% Black, 51% economically disadvantaged).

The study uses a clustered randomized trial design. Schools were randomly assigned into treatment (access to [Program], n=31) and control (business-as-usual homework practices, n=32) conditions. Treatment teachers received a face-to-face training followed by classroom visits at least two times per week to support their usage of [Program] during two consecutive school years. The study uses hierarchical modeling to determine whether [Program] was more effective for improving math achievement. Fidelity of implementation was monitored via [Program] backend system log. Data was collected via instructional logs, classroom observations, interviews, and surveys to understand the implementation and help establish contrast between conditions.

During the 2019-20 school year, implementation went well with expected variations before the unprecedented schooling disruption due to COVID-19. Fifty-three present of the 61 treatment teachers stopped using [Program] due to various limitations during distance learning. Twenty-six teachers continued using [Program] consistently. Almost all continued teachers reported dramatic drops in assignment completion rate during distance learning. Student outcome data was collected using an online assessment (in place of the cancelled state test) from 2,386 students of 41 schools. We will present findings from the implementation analyses that show (a) usage patterns and variation, before and after COVID-19; (b) the ways teachers used [Program] to support adaptation of instruction, and how the usage has shifted because of distance learning; and (c) how instructional practices contrast between treatment and control classrooms. Preliminary analyses will compare the learning outcome between conditions and explore associations of implementation with math learning outcomes.