Exploring the Convergence Between Self- and Out-of-School-Time Staff Report of Social Emotional Competence Among Student Leaders

• In Event: 42.079 - Social and Emotional Learning Assessment: Bias, (dis)Agreement, and the Great Direct Assessment Versus Behavior Rating Scale Debate

Abstract

Background:
In the assessment of youth, it is useful to include both adult and youth perspectives (Eiser & Morse, 2001). Youth and adults, however, tend to have different perspectives, resulting in low correlations (r = .20) between assessments across these informant types (Achenbach et al., 1987). These estimates primarily come from deficit-based behavior rating scales, and some have suggested that strengths-based behavior ratings scales may have higher correlations between youth and adult raters (January, Lambert, Epstein, Walrath & Gebreselassie, 2015; Synhorst, Buckley, Reid, Epstein, & Ryser, 2005).
The Devereux Student Strengths Assessment Mini (DESSA-Mini; LeBuffe, Naglieri, & Shapiro, 2011) is a brief strengths-based behavior rating scale assessing social emotional competence (SEC), found to be psychometrically sound for adult raters. The DESSA-Mini as a self-report instrument among children in late elementary school has not yet been explored. This pilot study invited students to provide ratings of their own SEC, with students responding to the DESSA-Mini questions in first person format. Student ratings were compared to DESSA-Mini ratings provided by Out-of-School-Time (OST) staff. This study examined: 1) internal reliabilities within both rater types, 2) differences in mean SEC, as reported by students and staff, and 3) the correct classification rate (%) between rater types.

Method:
The sample includes 89 4th and 5th grade students participating in an after school leadership program run by Playworks (53% male; median age=10; 53% Hispanic/Latino(a), 16% White/European-American, 14% Asian/Asian-American, 7% Black/African-American, 6% Other, 3% Multi-Ethnic, 2% Pacific Islander/Native Hawaiian). DESSA-Mini items for each rater were summed into a Raw Score Total and standardized within rater to parallel instructions in the user-manual. A binary “Need for Instruction” variable was created (1 = lowest 16%, 0 = all others) to determine if student self-assessment and staff assessment would lead to the same classification. A paired sample t test, a consistent classification percentage, and Cohen’s kappa were calculated to explore the relationship between student report and staff report of student Social Emotional Competence.

Results:
Both students and staff ratings of Social Emotional Competence (SEC) had high internal reliability (𝛼 = .83 and 𝛼 = .93, respectively). There was not a significant difference between the average student self-report (M = 25.25, SD = 5.28) and staff (M = 25.08, SD = 4.08) ratings of students’ SEC (t = .34, p = .74). Students and staff consistently classified students who had a “Need for Instruction” 79% of the time (𝜅 = .22).

Discussion:
Though student and staff ratings of Social Emotional Competence were internally reliable within rater, the kappa statistic indicates only “fair” (𝜅 = 0.21–0.40) agreement between raters. This is broadly consistent with the literature that suggests lower cross-informant inter-rater reliabilities than one may hope, although perhaps not lower than one might expect. This level of agreement may be noteworthy for elementary-aged youth but this study does not suggest that inter-rater reliability patterns are very different for strengths-based assessment.

Authors

• Jennifer Robitaille, Aperture Education

• Valerie Shapiro, UC Berkeley

• Sarah Accomazzo, University of California - Berkeley

• Sophie Shang, University of California - Berkeley