Measuring Social-Emotional Skills at Scale: Evidence From California's CORE Districts

• In Event: 53.060 - Rethinking Accountability: Early Research on California's CORE Waiver Districts

Abstract

The importance of non-tested skills for students’ academic and life success (Duckworth et al., 2010; Almlund et al., 2011) has generated calls to incorporate so-called “non-cognitive” or “social-emotional” skills into evaluations of educational performance and even school accountability systems. Yet we have little evidence on the ability of extant measures of social-emotional skills, most of which are based on self-reports, to provide valid and reliable evidence on differences in student skills across schools when administered at scale.

To shed light on this issue, we draw on survey responses linked to district academic and behavioral administrative data for more than 250,000 students in grades 3-12 within California’s CORE Districts.

We use these data to examine a number of issues relevant to proposals to use of self-reports of social-emotional skills for the purpose of school accountability. First we assess the reliability of student self-reports by examining the inter-item correlation of the survey scales. Next we evaluate the validity of the socio-emotional skills as predictors of school-level academic and behavioral outcomes. Third we assess comparability across school settings by examining how estimates across the sample and within schools differ. Fourth we investigate the ability of rankings based upon the social-emotional measures to differentiated schools by examining the confidence intervals around these rankings. Finally, we begin to explore how these correlations between socio-emotional skills and academic and behavioral outcome vary by subgroups and how this affects an accountability system using socio-emotional skills.

Our analysis of data from the CORE Districts’ field test confirms that student self-reports of key social-emotional skills at the middle and high school level demonstrate acceptable levels of internal reliability and, when aggregated to the school level, are strongly correlated with available indicators of academic performance and student behavior. We do not find evidence that student self-reports are systematically biased by differences in reference groups in ways that undermine the comparability of responses across schools. Also, while we find that average levels of social-emotional skills can effectively differentiate schools in the top performance deciles from those in the bottom deciles, there is considerable overlap in the confidence intervals of schools in the middle of the performance distribution. However, we also show that the degree of overlap is only slightly larger to that observed for average test scores. Finally, we confirm that, similar to test scores, social-emotional skills’ relationship with academic and behavioral outcomes vary for subgroups of students and accountability systems should addresses these when evaluating a school’s performance.

This paper offers a substantive addition to questions surrounding the use of social-emotional skills as an additional benchmark of school performance. Our preliminary results provide a broadly encouraging view of the potential for self-reports of social-emotional skills at scale as an input into systems used to evaluate school performance. However, our results cannot address questions about how the properties of self-report measures would change if stakes are attached.

Authors

• Martin West, Harvard Graduate School of Education

• Ethan Scherer, Harvard University

• Aaron W. Dow, Harvard University