Two Years In: Evaluation of the DigiLit Intervention

• In Event: Print to Digital in the Language Arts: Small-Scale Studies of Digital Innovation

Abstract

Purpose. The main purpose of this study was to evaluate the impact of DigiLit on reading achievement and literacy among K-5 students in an inner-ring school district at two time points: (a) the end of the pilot phase, or after a year of initial program design and implementation, and (b) at six months follow-up. In addition, the study also evaluated treatment effects on STEM-related content vocabulary, motivation to read, and reading activity after the initial pilot phase of the intervention. To provide a more meaningful, contextualized explanation of the results, interviews conducted among the staff at the treatment school were analyzed and integrated into the quantitative findings.

Design and Participants. The evaluation is based on quasi experimental design, comparing outcomes between one K-5 school selected for piloting DigiLit (School A) and another K-5 school within the same district which did not participate in the intervention (School B) possessing similar student characteristics.The sample from School A consisted of 285 students (46.3% male; 41.1% female). In terms of race, the sample was 65.3% Black/African American, 8.1% Hispanic/Latino, 7.4% White, 3.9% Multi-ethnic, 1.1% Asian, and .7% Other, with 13.7% missing; 48.8% was eligible for free lunch and 6.3% for reduced-price lunch, whereas 31.2% was not eligible and 13.7% was missing data. The sample from School B consisted of 277 (39.7% male; 44.8% female). The racial/ethnic composition was 66.1% Black/African American, 11.6% White, 3.6% Hispanic/Latino, 2.5% Multi-ethnic, .7% Asian, with 15.5% missing. The SES was 59.2% free lunch and 2.9% reduced-price lunch, with 22.4% not eligible and 15.5% missing.

Measures. The central outcome of interest, literacy, was measured across three occasions (beginning, middle, and end of year) using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS). State-mandated achievement test scores (Ohio Achievement Assessments) in Reading were collected on 3rd, 4th, and 5th grade students. In School A, the My Motivation to Read Profile (MMRP) was administered to grades K through 2, while the Motivation to Read Profile-R (MRPR) was administered to grades 3 through 5. The MMRP and MRPR were given pre- and post-test. At 6 months follow-up, the DIBELS and OAA Reading test scores are examined at both schools. Lastly, the Reading Activity Inventory (RAI) was administered at School A only (pre- and post-test) from 3rd to 5th grade. Qualitative data obtained from teacher interviews (n = 19) is based on a semi-structured protocol assessing teachers’ experiences in adopting and implementing the intervention.

Results. By the time of the presentation, the evaluation will be completed. At this time, analyses are being performed for the end of the pilot phase. The main analytic methods will employ ANCOVA with clustering corrections, longitudinal growth curve analyses, and consensual qualitative research. Effect sizes will be computed based on Hedge’s improvement index. Sub-group analyses based on grade level, gender, and SES will also be performed.

Significance: Collectively, the results offer implications for the innovative program diffusion process, including professional development and intervention studies.

Authors

• Justin C. Perry, Cleveland State University

• Adam Voight, Cleveland State University

• Chelsea Beabout