Self-Report Bias and Functional Abilities

• In Event: Measuring What Matters: Improving Data Quality and Decision-Making under Uncertainty

Abstract

Measuring individuals’ functional abilities is critical to being able to assess their work capacity. We administered a survey in the Understanding America Study, a nationally representative Internet panel of American adults who agree to participate in regular online surveys, in 2024 that included three major components: (1) respondents’ self-assessments of 52 abilities aligned with the O*NET Abilities database of occupational requirements; (2) cognitive performance tests; and (3) vignettes designed to evaluate how accurately individuals perceive their own cognitive abilities. In the first component, respondents rated their abilities on a 0–7 scale. In the second component, respondents completed brief tests of written comprehension, oral comprehension, inductive reasoning, number facility, and memory. In the third component, they estimated how many questions they answered correctly and evaluated hypothetical individuals’ ability levels based on randomized test performance. Analyses of the performance and vignette data revealed two systematic patterns of estimation error: respondents tended to believe they performed better on the tests than they actually did, and they applied stricter standards when rating the ability of a hypothetical individual compared with rating their own ability. We develop a method to adjust self-reported abilities to account for these biases using information from the vignette evaluations and respondents’ actual test performance and then construct individual work capacity measures for each respondent aggregated across occupations in the O*NET database. Preliminary results indicate that accounting for individuals’ estimation errors substantially lowers estimated work capacity but does not qualitatively affect patterns across demographic characteristics such as age. We are currently designing a follow-up survey implementing vignettes based on physical ability measures from wearable Fitbit trackers such as average weekly step count, active minutes per week, resting heart rate, and VO2 max score. This new survey will be fielded by summer 2026 and we plan to report preliminary findings incorporating new results for physical abilities in the session.

Author

• Kathleen Jennifer Mullen, University of Oregon
  Presenting Author