Predictors of Rater Bias in the Assessment of Social Emotional Competence

• In Event: 42.079 - Social and Emotional Learning Assessment: Bias, (dis)Agreement, and the Great Direct Assessment Versus Behavior Rating Scale Debate

Abstract

The Devereux Student Strengths Assessment Mini (DESSA-Mini; LeBuffe, Shapiro, & Naglieri, 2011) was designed to overcome obstacles to monitoring the growth of Social Emotional Competence in the routine implementation of Social Emotional Learning (SEL) programs (Maras, 2015). This strength-based assessment system includes four interchangeable 8-item DESSA-Mini forms and a longer 72-item assessment (DESSA).
Behavior rating scales like the DESSA strive to measure the frequency of behaviors that indicate age and context-appropriate emotional recognition and management, social awareness and interactions, and responsible decision-making. Behavior rating scales can be used repeatedly, across settings, and with multiple informants to capture an understanding of the child’s behavior over time and relative to a standardized reference group. Yet, behavioral rating scales have been criticized for their potential to incorporate rater bias into assessment scores (Elliot, Frey, & Davies, 2015).
Rater bias is “the presence of substantial and systematic error in ratings of performance or behavior caused by rater attitudes, beliefs, or experiences” (Mason, Gunersel, & Ney, 2014). Studies of other constructs have considered sources of divergence between teacher ratings and a “true” criterion (Podsakoff, MacKenzie, & Podsakoff, 2012). Since Social Emotional Competence does not yet have a consensus indicator of “true” levels, this paper seeks to understand the variance in DESSA-Mini scores that is accounted for by rater characteristics.

Method:
Data were collected within a diverse California school district (59% White/European American, 12% Asian/Asian American, 12% Hispanic/Latino(a), 7% multi-race, 6% Black/African American, and 4% other). Elementary school classroom teachers (n = 72) assessed 1,676 students using the DESSA-Mini. The students were assessed to have similar levels of SEC (M = 50.88; SD = 11.74) as the national reference group (M = 50.00; SD = 10.00). Hierarchical linear modeling with maximum likelihood estimation (Rabe-Hesketh & Skrondal, 2012) was used to estimate the relationship between teachers’ ratings of student SEC (DESSA-Mini scores; level one) and teachers’ self-reported characteristics and perceptions (level two) as they started implementing TOOLBOX, a social emotional learning program.

Results:
Only 16% of variance in DESSA-mini ratings was attributable to raters. Teachers’ perception of general barriers to student learning (b = -1.70, p = .02), report of their “typical” student’s SEC (b = 2.79, p = .004), anticipation of SEL program implementation challenges (b = -1.90, p = .02), and intention to fully implement the SEL program (b = .94, p = .03), predicted DESSA-Mini ratings. Together, these predictors explained nearly 6% of the variance in DESSA-Mini ratings.

Conclusion:
This study clarifies the extent to which DESSA-Mini scores reflect characteristics of the rater, in addition to characteristics of the individual student being rated. Findings of this study help us to unpack student assessment scores into their component parts, which may increase our responsible use of behavior rating scales like the DESSA-Mini for the rating of student social emotional competence.

Authors

• Paul Augustine LeBuffe, Aperture Education

• Valerie Shapiro, UC Berkeley

• B. K. Elizabeth Kim, University of Southern California

• Sarah Accomazzo, University of California - Berkeley

• Joe N. Roscoe, University of California - Berkeley