Understanding Why Between-School Analyses of Student-Reported Noncognitive Factors Are Flawed

• In Event: Poster Session 9
  In Poster Session: New Developments in Psychometrics, Measurement, and Assessment: Poster Session 1

Abstract

Shifts in individuals’ response tendencies on surveys due to differing local contexts—i.e., reference bias—may help explain null or negative average change in assessments of school programs designed to promote non-cognitive constructs such as grit and self-control. We examine this possibility by comparing a composite of two non-cognitive survey instruments against a reference-bias-immune behavioral task of diligence. For each instrument, within-school and between-school-average comparisons were used to predict post-1-year college persistence. Results were consistent with the reference bias hypothesis: self-report instruments positively predicted persistence only within-schools, while negatively predicting persistence when comparing between-school averages. The behavioral instrument, meanwhile, was consistent in positively predicting later college persistence.

Authors

• Joseph O'Brien, The University of Texas at Austin

• David Scott Yeager, Stanford University

• Brian Galla

• Angela L. Duckworth, University of Pennsylvania

• Sidney K. D'Mello, University of Notre Dame