Understanding the Impact of Personalization for Academic and Social Emotional Learning

• In Event: The Successes and Challenges of Measurement in School and Out-of-School Improvement Research

Abstract

Improvement science approaches to implementation in education have increased over the last fifteen years, showing great promise in building buy-in and capacity among stakeholders
(Peurach et al., 2022). They offer educators the ability to adapt to local context and iterate over
time, resulting in a greater match to district and school needs and culture. From a research
perspective, however, they present a challenge. With schools tailoring reforms to their unique
needs over time, what are the most effective methods for understanding the effectiveness of a
Reform?

Here, we examine the implementation and outcomes of Personalization for Academic and Social Emotional Learning (PASL), a secondary school reform aimed at strengthening relationships across stakeholders, particularly between adults and students. Between 2014 and 2022, as part of the process of implementation and scale, over 48 Broward County Public middle and high schools agreed to adapt PASL to their school sites and participate in a networked improvement community (NIC) of quarterly meetings and continuous improvement activities. By the end of the 2021-22 school year, over 40,000 students had participated in PASL. PASL as a reform is comprised of five components: 1) routine rapid check-ins (RCIs); 2) goal setting activities; 3) intentional use of data; 4) educator teams; and 5) an explicit culture of personalization within the school (Author et al., 2020).

While all participating schools implemented all components of PASL, the size and nature of implementation varied across sites, with some schools focusing more on RCIs and use of data, for example, while others emphasized educator teams and school culture. Schools also varied widely in the size of their PASL program. Researchers and evaluators used multiple research methods to understand variation in implementation and outcomes of PASL. Researchers interviewed district and school administrators, teachers, and students annually to understand the approaches and quality of implementation. During the fourth year of the reform (2016-17), an independent evaluation team used a quasi-experimental design to study six first-year adopters compared to six schools that had not yet implemented PASL, with surveys of teachers and students and linking students to their administrative records.

In the qualitative findings, researchers identified a high degree of commitment to PASL by stakeholders, accounting for its success in scaling. Using quasi-experimental designs, evaluators found weak statistically significant findings in SEL outcomes of PASL such as in student engagement, while no effects on self-efficacy and self-regulation (Rosen et al., 2022).
Academically, they found some improvement on unexcused absences and credits earned, but no
effects on grades or test scores. Comparisons of findings from the qualitative study with findings from the quantitative evaluation found that the administrators identified many of the same positive effects of the quantitative study, but saw them as making a stronger impact on schools and students. With strong qualitative findings yet weak and inconsistent quantitative findings, how do we make sense of the impact of PASL? Further, given the variation in implementation, a key feature of improvement work, what are the implications for high quality comparative quantitative outcome studies?

Authors

• Stacey A. Rutledge, Florida State University

• Marisa A. Cannata, Vanderbilt University

• Jeffrey A. Rosen, RTI International

• Yihua Hong, RTI International

• Seungbae Kim, Florida State University