Impact Evaluation of the National Writing Project's College-Ready Writing Project in High-Poverty Rural Districts

• In Event: Successful Teaching Practices: The What Works Clearinghouse Practice Guide on Secondary Writing

Abstract

Objectives
This paper presents a randomized controlled trial (RCT) that found positive impacts of the National Writing Project’s College-Ready Writers Program (CRWP) on student writing outcomes. This study had three research questions:
Was CRWP implemented with fidelity?
What impacts did CRWP have on teachers’ practice?
What impacts did CRWP have on student argument writing?

Framework
Drawing on Cohen, Raudenbush, & Ball’s (2003) conception of an instructional triangle composed of teachers, students, and content, the study posits that professional development (PD) is more likely to impact instruction when integrated with instructional materials (targeting both teachers and content).

Design and Methods
The evaluation used a cluster RCT design, randomly assigning 44 rural, high-need school districts to the treatment or a business-as-usual control condition (Table 1). Participating districts were recruited in pairs and randomized within pair to balance contextual issues.

The teacher sample consists of all 7–10th grade ELA teachers in the 44 districts. The student sample included the students who began in the 7th–9th grades in the 2013–14 school year and were still enrolled in 2014–15.

Student outcomes were measured using an on-demand writing assessment, scored on the Analytic Writing Continuum for Source-Based Argument Writing, which prior studies have found to be valid and reliable (Bang, 2013). (Table 2 presents descriptives.) Teacher outcomes were measured via a teacher survey and an instructional log. Implementation fidelity was measured via data from Writing Project sites on teacher participation in CRWP. Finally, the study relied on the teacher survey to understand the treatment-control contrast. All outcome measures were administered at baseline and at the conclusion of the study.

The predicted writing ability for student i, in district j, in blocking pair k as a function of attending a district assigned to treatment is given as:

Yijk=β0 + β1(Treatmentj) + (Covariatesi)β2 + εijk + ηjk + μk

Random effects εijk, ηjk, and μk allow for error at the student, district, and block level, respectively. Vector Covariatesi improve the precision of the estimate. β1 provides an estimate of the effect of district assignment to treatment on student writing ability within that district (the Intent-to-Treat effect). To estimate the impact of CRWP on teacher outcomes, we compare survey and log indicators of teaching practice using similar methodology as used to analyze student outcomes.

Selected Findings
CRWP was implemented largely as intended (Table 3).
There were large, statistically significant differences between the treatment and control group in the PD they experienced (Table 4).
Treatment teachers spent slightly more instructional time on writing, though they were substantially more likely than control teachers to focus that time on argument writing (Tables 5 and 6).
CRWP had positive, statistically significant, and robust impacts on student argument writing (Table 7), with effect sizes of approximately .20.

Conclusions
This study adds rigorous experimental evidence to a body of literature that indicates that PD that focuses on more than one aspect of instructional capacity (i.e., PD and instructional materials) is more likely to lead to meaningful student learning.

Authors

• Katrina R. Woodworth, SRI International

• H. Alix Gallagher, SRI International

• Nicole Leigh Arshan, SRI International