Quantitative Analysis of Moderating and Mediating Factors in the Maine ASSISTments Efficacy Study

• In Event: 54.028 - Efficacy of an Online Mathematics Homework Intervention: Rigorous Research Transforming Knowledge to Action

Abstract

Objective
This presentation will explore whether the treatment effects found in the ASSISTments Efficacy Study varied for different subgroups of students.

Perspective
While interventions may be found to be effective, understanding whether these effects are broadly based for all students, or whether there are differential treatment effects in which some groups may benefit more (or less) than others is vital. Understanding possible differential treatment effects can help to ensure that programs target those students who are most likely to benefit from specific interventions.

Methods
Data from the ASSISTments Efficacy Study were examined in order to test cross-level interactions between treatment condition and student sex, special education status (i.e., IEP), free or reduced lunch status (i.e., an indicator of lower socio-economic status), and prior mathematics performance on standardized statewide 6th grade assessments. We also examined whether teachers reported different homework review practices in the treatment and control conditions, and in on-going analysis, will explore this as a potential mediating variable.

Data Sources
The outcome consisted of 7th grade end-of-year TerraNova mathematics scores. Predictor variables were student demographic characteristics and prior performance on statewide standardized assessments contained within the state longitudinal data system. For change in teaching practice, we collected data from teachers via surveys, instructional logs, and by observation (see the next paper).

Results
Analyses consisted of two-level hierarchical linear models, expanding on the overall outcome results presented by Roschelle and Feng (see paper one). These analyses found no statistically significant interaction between treatment and special education status (𝛾=6.846, t(2759)=0.875, p=0.382), free and reduced lunch status (𝛾=1.113, t(2759)=0.370, p=0.712), or a sex (𝛾=5.841, t(2758)=1.793, p=0.073). Analyses did find a statistically significant interaction between treatment condition and prior mathematics performance, in this case dichotomized into prior performance above or below the median (𝛾=8.717, t(2770)=2.432, p=0.015). As illustrated in Figure 1, adjusted scores for treatment condition students whose 6th grade mathematics performance was above the median were 4.501 points higher than similar control students with 6th grade scores above the median. However, among those students who scored below the median in 6th grade, those receiving ASSISTments scored 13.218 points higher than their peers in the control condition. The effect size (d=.28) is larger for students whith lower prior mathematics achievement. Preliminary analysis of the mediating effect of teaching practice shows that teachers did change their practice in the treatment group; by 2017 results on these findings will be available.

Significance
Results of these moderator models suggests that overall, the treatment effect was fairly consistent for both males and females, students receiving special education services and those not receiving special education services, and those eligible and not eligible for free or reduced lunch. The findings for prior mathematics performance suggest that ASSISTments may be particularly beneficial for student who are previously lower-performing, helping them to close the gap with their peers.

Authors

• Craig Mason, University of Maine

• Robert F. Murphy