Adapting to the Times: Assessing Cross-Disciplinary Professional Learning and Multistate Scalability

• In Event: Reading Apprenticeship: Lessons From Impact Studies Over Time and Across Contexts

Abstract

Purpose
We will illustrate how RA PD has responded to the changing needs of school districts for supporting teacher PD across disciplines and the needs for different support mechanisms for scaling PD programs. We will present the results of two impact studies and one scalability study.

Theoretical Framework
The first PD implementation consisted of 5 in-person days (3-day Summer Institute; 2-day Winter Institute) during Year 1, followed by online Professional Learning Communities (PLCs) and in-person school team meetings throughout Year 2. The planned implementation for the second PD was similar with the addition of two Teacher Leader regional meetings per year. However, the COVID-19 pandemic changed PD significantly. Cohort 1 teachers completed most PD activities as intended before lockdowns began. Cohort 2 teachers received the PD entirely online. The Institutes were converted into synchronous and asynchronous modules while the PLCs and Teacher Leaders were eliminated.
Methods
The impact evaluations included an RCT designed to assess program impacts on teacher practices and student achievement across multiple states. The first RCT included ELA, science and social studies teachers and their students from 40-middle schools during 2016-2018. The second RCT included science teachers and their students from 50 middle and high schools during 2019-2021. The scalability study employed a mixed-method approach during 2021-2023.

Data Sources
For the RCTs, teacher measures included surveys and student measures included administrative data, state assessment scores, reading assessments, and student surveys. The scalability study included observations of face-to-face and online PD events, focus groups with teacher leaders, and interviews with instructional leaders, and administrators.

Results
Teachers used fewer traditional practices (i.e., lecturing) and the estimated impact was further strengthened over time with effect sizes (ES) ranging from 0.415 to 0.471. Teachers modeled collaboration practices (ES=0.302) and used differentiated instruction (ES=0.320). Students used reading strategies (ES=0.405) and metacognitive inquiry strategies (ES=0.403) and spent more class time reading (ES=0.160). Impact analysis for the second RCT study is ongoing and results will be available at the time of AERA 2023 and included in our final presentation.

The COVID-19 pandemic hindered PD implementation. Teachers struggled due to decreased student engagement and collaboration during remote-learning. Challenges continued during in-person learning due to staffing shortages and other demands on teachers’ time. Some teachers were able to use certain RA strategies, but they didn’t feel prepared to fully implement the program. PD activities were not as well attended as in previous studies. Few teachers finished all portions of the online modules, and many teachers simply did not have time for it due to other demands.

Significance
RA’s PD prepared thousands more effective teachers whose students increased their metacognitive awareness and use of reading strategies. However, questions remain whether shortened PD in cross-discipline groups is of enough intensity and duration to shift teachers’ practices and improve student outcomes, and how local contexts affect the success of different scalability models.

Authors

• Robyn Elkins, American Institutes for Research
  Presenting Author

• Michaela Gulemetova, American Institutes for Research
  Non-Presenting Author

• Mikhail Pyatigorsky, IMPAQ International, LLC
  Non-Presenting Author

• Raquel C. Sanchez, American Institutes for Research
  Non-Presenting Author