Evaluation of ST Math Treatment Effects for Special Populations and by Length of Implementation

• In Event: Facets of Large-Scale Evaluation Illustrated With ST Math: Examining Outcomes, Mediators, and Moderators

Abstract

This paper presents the effect of use of STMath on math CST scores. In response to the need for innovative solutions for math education in elementary school, MIND designed STMath as an individually-paced computer learning environment to teach mathematical reasoning through spatial temporal representations. The game-like exercises within are formulated to engage and motivate students to solve mathematics problems; successive games present problems of increasing difficulty, eventually leading to challenging, multi-step problem-solving and covering a year's worth of math curriculum.

Program developers report that STMath is currently used by about 473,000 K-8 students across 1,355 schools in 24 states. Our study's 52 Southern California schools (85% Hispanic, 63% ELL, 90% Free/Reduced Lunch) were randomly assigned to receive STMath in either second and third or fourth and fifth grades. A pretest included in the analysis was given to students starting in second grade, therefore this paper reports results for third through fifth graders only.

When considering all students within the study (N=13,803), STMath results in a marginally significant (using conservative standard errors) increase in math performance as measured by the CSTs (d=.07). This effect size, although small, is consistent with those found in other math interventions measured by broad standardized tests (Hill, Bloom, Black, & Lipsey, 2008). Because of the design of STMath, it was hypothesized that three factors might influence effects: ELL status, initial proficiency, and length of treatment. They are discussed below.

By drawing on innate spatial-temporal ability (Shaw & Peterson, 2000), STMath is designed to provide access to math instruction for those students who struggle in traditional language-heavy learning environments. ELL students, who consistently perform below fluent English speakers in standardized math tests (see Hemphill & Vanneman, 2011) were hypothesized to show stronger treatment gains when compared to their English proficient peers. Effect sizes for ELL students were non-significantly lower than non-ELL peers (d=.06 vs. d=.08), rejecting this hypothesis.

STMath's individualized curriculum is designed to allow students to progress to new concepts only once they have mastered the foundational materials. We hypothesized that, because they can take time to review basic concepts at their own pace, lower performing treatment students may make greater gains relative to their proficient peers. This hypothesis was also rejected—one year gains across the five CST proficiency categories were consistent.

New instructional technology, like STMath, may initially boost student achievement due to engagement and motivational benefits from the novelty of the intervention (e.g., Song & Keller, 2001; Tung & Deng, 2007). Conversely, students who have one year of experience in the program may be more adept at working with the medium and translating what learn to assessments outside of the software. We tested the effect of two years of STMath on a subsample of students (N=2,677) and found that after the second year, treatment students had significantly higher gains than their control-group peers with no STMath (d= .16). Both first and second year gains were roughly equal, supporting neither the novelty or experience theories.

Authors

• Teomara Rutherford, North Carolina State University

• George Farkas, University of California - Irvine