Impact Evaluation of the National Writing Project's College-Ready Writers Program in High-Poverty Rural Districts

• In Event: 18.071 - Roundtable Session 5
  In Roundtable Session: 18.071-4 - Building and Leveraging Teacher-Leaders' Expertise to Improve Students' Writing: National Writing Project's College-Ready Writers Program

Abstract

This paper presents a randomized controlled trial (RCT) that found positive impacts of the NWP’s College-Ready Writers Program (CRWP) on student writing outcomes.

This study had three main research questions:
1. Was CRWP implemented with fidelity?
2. What impacts did CRWP have on teachers’ instructional practice?
3. What impacts did CRWP have on student argument writing?

Sample
The sample consists of all 7-10th grade ELA teachers in 44 rural districts. The student study population included the students who began in the 7th–9th grades in the 2013–14 school year and were still enrolled in the 2014–15 school year.

Research Design
The evaluation used a cluster RCT design, randomly assigning 44 districts to the treatment or a business-as-usual control condition (Table 1). Participating districts were recruited and randomized in pairs to balance contextual issues.

Framework
Drawing on Cohen, Raudenbush, & Ball’s (2003) conception of an instructional triangle composed of teachers, students, and content, the study posits that professional development (PD) is more likely to impact instruction when integrated with instructional materials (targeting both teachers and content).

Data Collection and Analysis
Student outcomes were measured using a study-administered on-demand writing assessment, which prior studies have found to be valid and reliable (Table 2) (Bang, 2013). Teacher outcomes were measured via a survey and instructional log. Implementation fidelity was measured via data from Writing Project sites on teacher participation in CRWP. Finally, the study used the survey and site visits to understand the treatment-control contrast. All outcome measures were administered at baseline and again at the conclusion of the study; measures of implementation fidelity were collected throughout.

The predicted writing ability for student i, in district j, in blocking pair k as a function of attending a district assigned to treatment is given as:
Yijk=β0 + β1(Treatmentj) + (Covariatesi)β2 + εijk + ηjk + μk

Random effects εijk, ηjk, and μk allow for error at the student, district, and block level, respectively. Vector Covariatesi improve the precision of the estimate. β1 provides an estimate of the effect of district assignment to treatment on student writing ability within that district (the Intent-to-Treat effect). To estimate the impact of CRWP on teacher outcomes, we compare survey and log indicators of teaching practice using similar methodology to that used to analyze student outcomes.

Selected Findings
• CRWP was implemented largely as intended (Table 3).
• There were large, statistically significant differences between the treatment and control group in both the quantity and content of the professional development they received (Table 4).
• Treatment teachers spent slightly more instructional time on writing, and were substantially more likely than control teachers to focus that time on argument writing (Tables 5 and 6).
• CRWP had positive, statistically significant, and robust impacts on student argument writing (Table 7), with effect sizes of approximately .20.

Conclusions
This study adds experimental evidence to literature that indicates that programs designed with more than one mechanism to improve instruction—in this case teacher-led PD and instructional materials—are more likely to lead to meaningful student learning.

Authors

• Katrina R. Woodworth, SRI International

• H. Alix Gallagher, SRI International

• Nicole Leigh Arshan, SRI International