Implementation at Scale: Assessing Discipline-Specific Professional Learning and a Multistate Model

• In Event: Reading Apprenticeship: Lessons From Impact Studies Over Time and Across Contexts

Abstract

Purpose
This paper presents findings from a i3-funded formative and summative evaluation of RA. The findings of the process study produced a scale-up model that has informed subsequent iterations of RA.
Theoretical Framework
The RA instructional framework was implemented in discipline specific groups. The intensive face-to-face PD included ten days (65 hours) of PD sessions over 12 months with on-site support from teacher leaders. It was designed to include the key characteristics shown to improve teaching and learning: content focused; structured and sustained follow-up; engaged collective participation; coherence by building on and valuing teacher’s existing knowledge and beliefs, and inquiry based (Desimone, et. al. 2009).
Methods
The RCT included 42 high schools in two states during 2011-2014. The study investigated impacts on ELA, biology, or U.S. history teachers and students enrolled in their classes (about 16,000 students).
Data Sources
Student standardized state tests as well as literacy outcomes were measured by scenario-based literacy assessments developed by ETS. Students’ attitude and behavior measures were gathered using a survey administered at the end of the school year. The student constructs included a) collaboration among students; b) use of comprehension strategies; c) content and literacy activities integration; d) engagement in metacognitive conversations; e) reader identity; f) reading of a variety of texts, and academic engagement.
Teacher practice and attitude measures were developed based on items from monthly teacher surveys. The teacher constructs included a) variety of text types used in class; b) fostering of student independence; c) teachers’ instruction practices regarding metacognitive inquiry and comprehension strategies; d) teachers’ fostering of student collaboration and engagement; and e) teacher self-efficacy.
Results
Students integrated content and literacy activities (effect size =0.12) and metacognitive conversations (ES=0.11). Teachers showed impacts in: 1) fostering student independence; 2) students practicing metacognitive inquiry; 3) students practicing comprehension strategies; 4) student collaboration; and 5) teacher confidence in literacy instruction, with estimated ES ranging from 0.46 to 0.65.
The formative evaluation of program scale up found that RA uptake and scale-up outcomes decreased between the first and second year of implementation. This led to the development of a multi-stage scale-up model that emphasizes school- and district-level ownership for RA. To support the development of local ownership, WestEd focused on developing LEA capacity to support project coordination, take on responsibility for PD delivery and instructional support, increase participation in RA over time and engage in cycles of continuous improvement.
Significance
RA was effective in increasing student ability to integrate content with literacy activities and to engage in metacognitive conversations about their own learning. We also found an increase in teacher confidence and ability to foster student independence, higher order thinking, metacognition and collaboration. The study underscored the importance of LEA buy-in and ownership if scale up is to be successful and impacts sustained over time. This study led to an increased emphasis in the evolving program model to include shifting ownership for literacy reform from the external provider to local leadership.

Authors

• Michaela Gulemetova, American Institutes for Research
  Presenting Author

• Cheri Fancsali, Research Alliance for New York City Schools
  Non-Presenting Author