An External Evaluation of Student Progress During the Planned Variation Phase

• In Event: What Works for Beginning Readers With Individualized Education Programs? Recent Findings From a Large, Multisite Development Project

Abstract

Objectives
This paper discusses the methods and results of the external evaluation of HEROES at the end of Year 2.

Theoretical Frame

The modeling approach to evaluate student progress is closely linked to the theoretical frame that guides the design of the development work. We assume that early literacy progress may be evaluated and described in terms of progress on measures of highly predictive component skills; including measures of phonemic awareness, reading text, letter identification, word identification and print concepts. We also assume that student outcomes will vary by teacher.

Methods
We conducted a quasi-experimental study in six school districts, in three states. Altogether 57 schools and 89 teachers participated in the study. 37 schools and 66 teachers were in the treatment group and 20 schools and 23 teachers were in the control group. Each posttest outcome was modeled separately to examine whether the treatment effect was statistically significant on each while controlling for pretests, demographic characteristics, as well as taking into account the nesting of students within schools.

Data Sources
62% of the students were male, 54 percent White, 24 percent Black and 20 percent Latino. 82% percent of the students were eligible for free or reduced price lunch. 87% of the students were in grades 1-3. 74% of the students in the sample received the treatment. Nearly 44% of the students had some learning disability and 35% of the students had some developmental delay.

There were seven literacy outcome measures: letter identification (LI), word test (OWT),
Print concepts (CAP), writing vocabulary (WV), hearing and recording sounds in words (DIC), text reading level (TRL), and a word recognition test (Slosson). The main independent variable was the treatment that compared teachers who implemented the literacy intervention to those who did not. The model controlled for potential state, gender, race, SES, age, type of disability, English language learner status, and grade effects. Finally, pretest measures were also included in the model. We utilized two-level models to capture the potential nesting of students within teachers.

Results
In four of the seven outcomes the treatment was positive, statistically significant at the 0.05 level and considerable in magnitude. Specifically, the treatment effect was positive and statistically significant for CAP and TRL and slightly larger than one-half of a standard deviation (SD). For WV and Slosson the treatment effect was also positive and statistically significant but smaller in magnitude and nearly 0.30 SD’s. In addition, the treatment effect was positive for OWT, statistically significant at the 0.10 level and about one-fourth of a SD.

Significance
According to the What Works Clearinghouse, effect sizes that are 0.25 SD’s or higher are meaningful; thus these results suggest that overall the intervention had a significant and meaningful impact on student outcomes since the treatment effect in five of the outcomes was 0.25 SD’s or larger.

Authors

• Spyros Konstantopoulos, Michigan State University

• Jerome V. D'Agostino, The Ohio State University