Comparing the Direct Assessment of Social-Emotional Comprehension to the Indirect Assessment of Social-Emotional Competence

• In Event: 42.079 - Social and Emotional Learning Assessment: Bias, (dis)Agreement, and the Great Direct Assessment Versus Behavior Rating Scale Debate

Abstract

Direct assessment (DA) and behavior rating scales (BRS) are both useful assessment methods (Elliott, Freu, & Davies, 2015), but studies indicate only low to medium correlations (Merrell, 1993). This may be because DAs are traditionally refined for clinical case evaluations, focusing on a small number of highly specific behaviors, while BRSs may seek to efficiently assess a broader band of behavior within an entire population. One study in the SEL domain found a correlation (r = .42) between DA (SELweb) and BRS (Devereux Student Strengths Assessment; DESSA) scores among students in K-3rd grade (McKown, et al., 2015). This paper examines the relationship between another DA (Virtual Environment for Social Information Processing; VESIP™) and the DESSA among 4th-5th grade students.

Method:
Data were collected by Playworks, a provider of play-based enrichment activities, and obtained through a data sharing agreement. VESIP™, based on a social information processing theory (Dodge, 1996), is a DA of social-emotional comprehension, or children’s social information processing skills (McKown, 2015). VESIP™ is a computerized simulation in which children adopt an avatar, interact with other avatars, respond to customized challenging social situations, and engage in social decision-making (Russo-Ponsaran, et al., 2012). Children virtually-experience 10 scenarios, earning points for prosocial responses. After the child’s initial behavioral response, children identify problems, attribute intentions, generate goals, construct final behavioral responses, and evaluate self-efficacy. Final scores are averages across the scenarios.
Children (n = 62) participating in leadership programming at 5 schools were assessed with VESIP™. Teachers (56%), program staff (27%), or parents (17%) assessed two thirds (n = 41) of these 4th (44%) and 5th (56%) grade students using the DESSA. The DESSA is a 72-item, strength-based, BRS for assessing social-emotional competence (LeBuffe, Shapiro, & Naglieri, 2009).The students (modal age = 10) were 66% female, 39% Hispanic/Latino(a), 27% Asian/Asian American, 10% White/European American, 10% Mixed Race, and 14% Unknown/Other (e.g., American Indian/Alaskan Native, Black, Pacific Islander). Pearson correlations and multiple regression models were estimated to examine the relationship between VESIP™ scores and DESSA scores.

Results:
As determined by the DESSA, 2.4% of students were assessed as having a “need” for SEL instruction (DESSA score ≤ 40), 68% to be typical, and 29% to have social emotional strengths (DESSA score ≥ 60); a sample skewed toward competence. DESSA scores significantly (p < .05) correlated with the child’s initial behavioral responses (r = .41), intention attributions (r = .33), goals (r = .31), final behavioral responses (r = .37), and congruence between the child’s goals and response constructions (r = .51) on VESIP™. Additionally, DESSA categories predicted VESIP™ scores. For example, students in the DESSA strength range provided more prosocial intent attribution on VESIP™, relative to students in the DESSA typical range (β = .33; p < .05).

Conclusions:
This paper examined method effects between VESIP™ and DESSA and the extent to which social emotional comprehension relates to social emotional competence. The correlations found are higher than are typically reported in the literature and may lend credibility to both instruments for meeting the diverse assessment needs of schools.

Authors

• Valerie Shapiro, UC Berkeley

• Sarah Accomazzo, University of California - Berkeley

• Nicole Russo-Ponsoran, Rush NeuroBehavioral Center

• B. K. Elizabeth Kim, University of Southern California