Using Behavioral Tasks to Overcome Frame of Reference Biases When Measuring Socioemotional Competencies

• In Event: 29.079 - Investigating Performance Tasks as Measures of Social and Emotional Learning Skills

Abstract

The substantial evidence for the real-world benefits of increasing socioemotional competencies—such as grit and self-control—understandably underlies recent pushes to measure such competencies in schools (Duckworth, 2016; Duckworth & Yeager, 2015; Zernike, 2016), both for school-level assessment and for evaluation of programs intended to promote these competencies. However, schools’ own evaluations have shown null or even paradoxically negative relationships between socioemotional competencies and key outcomes (West et al., 2016).
Traditional self-report approaches to measuring these competencies also incorporate school-level distortions that render them highly misleading. This “frame of reference bias”, changing of personal standards in ways that correlate with differences in a school’s rigor—interferes with processes of survey responding (Krosnick, 1999; Tourangeau, Rips, & Rasinski, 2000), leading higher-performing schools to appear as lower performing ones, and vice-versa. With N = 1,283 high school seniors across 15 schools whose cultures explicitly promote high standards of grit and self-control (albeit with differential effectiveness)—we found that, on average, the schools with the lowest self-report scores of grit and self-control in fact tended to attend and persist in college at the highest rate.
However, comparing students strictly within schools—where students share common standards and thus frames of reference for responding—self-report still predicted persistence in the expected direction. The constructs themselves remain valid. Yet, the between-school differences were inverted by differing response patterns, and were completely invalid.
In the same study, however, we demonstrate an alternative approach to assessment that avoids the frame of reference bias problem entirely: the use of validated behavioral assessments. Behavioral task assessment is not subject to frame of reference bias because it does not use self-report; thus, when the same sample as above was assessed using a behavioral task measuring grit and self-control, higher scores on the task predicted increased college persistence—regardless of whether the comparisons were within-school or between-school.
A subsequent study attempts to correct reference frame biases in self-report by using explicit reference frames. In a 2 x 2 multi-school experimental design, students (N = 7,419) responding to a grit and self-control survey were instructed to respond by comparing themselves against either the highest or lowest exemplars of either their school or the world.
These small changes in instructions did shift mean scores in the expected direction—again demonstrating how easily varying reference frames can alter responding. However, effect sizes from the manipulation were not large enough that they could have been used to overcome those frame of reference biases found naturally in the first study. Additionally, there were no differences between the “school” or “world” conditions of either valence—ruling out the possibility that invoking a global reference frame could render schools comparable.
Behavioral tasks avoid frame of reference biases entirely, while self-report surveys are quite susceptible to them, and not easily corrected. The creation and use of behavioral tasks can thus be an excellent investment towards the goal of valid socioemotional assessment—and may benefit assessment with many other traditionally self-reported constructs as well.

Authors

• Joseph O'Brien, The University of Texas at Austin

• David Scott Yeager, Stanford University