Research Design and Main Effects From the Maine ASSISTments Efficacy Study

• In Event: 54.028 - Efficacy of an Online Mathematics Homework Intervention: Rigorous Research Transforming Knowledge to Action

Abstract

Objective
This presentation will both provide an overview of the ASSISTments Efficacy Study (as context for the session) and also present the main effect findings.

Framework
The ASSISTments platform was developed through set of federal grants to build on progress in Intelligent Tutoring Systems (e.g. Ritter et al, 2007) while addressing practical needs of mathematics teachers in the classroom. The platform provides students with feedback and hints as they do mathematics tasks and provides teachers with easy-to-comprehend reports about students’ work. The theoretical framework builds both on past research on formative assessment (e.g. Shute, 2008), emphasizing the value of timely feedback to students and enabling teachers to adapt instruction based on student work. ASSISTments also build on past research on designing instruction for cognitive skill development, such as spaced practice (Koedinger, Booth & Klahr, 2013; Rohrer, 2009). We offered ASSISTments as a way to improve the value of homework, building on the fact that homework is an established practice which could be made both more efficient and productive.

Methods
This study, funded by the Institute of Educational Sciences, was a cluster randomized trial to evaluate the impact of an intervention comprising ASSISTments and associated teacher professional development on student learning of mathematics during a full year of 7th grade instruction. A sample of 46 Maine schools was recruited, and 43 were in the final data set. Schools blocked into pairs, and one school in each pair was randomly assigned to ASSISTments and the other to a business-as-usual control. In the ASSISTments conditions, teachers were provided with training on how use of ASSISTments, aligned with the research above. After a warm-up year for the teachers, ASSISTments was used with a new cohort of 7th grade students for a school year. Impacts were analyzed using a hierarchical linear model (HLM).

Data Sources
The outcome measure was the TerraNova Common Core standardized assessment, given to students at the end of their 7th grade year. We used the students’ 6th grade mathematics and reading scores on statewide assessment (NECAP) as covariate. The project collected additional measures to be described in later presentations.

Results
There were no significant differences between groups in prior mathematics achievement. In an HLM model that controls for prior mathematics and reading scores as well as students’ IEP and Free-and-Reduced Price lunch status, there was a significant difference in mathematics achievement (see Table 1); students whose schools were in the ASSISTments condition learned more. The effect size of the intervention was d=0.18.

Significance
Given strong school interest in learning technology and in improving mathematics, and the rarity of finding effects in rigorous experiments, these results are significant. The magnitude of the effect can be described in terms of an improvement index: students at the 50th percentile without the intervention would improve to the 58th percentile if they received the ASSISTments treatment. More generally, the intervention was relatively easy for schools and teachers to adopt, as it fit within established homework policies and practices.

Authors

• Jeremy Roschelle, Digital Promise

• Mingyu Feng, WestEd

• Robert F. Murphy

• Craig Mason, University of Maine