Measuring What Matters: Improving Data Quality and Decision-Making under Uncertainty

Session Submission Type: Panel

Abstract

Accurate measurement is central to effective policy design and analysis. However, the data used to inform decisions are often noisy, biased, or sensitive to how information is presented. This panel brings together three papers that develop and apply innovative methods to improve the measurement of disability and advance our understanding of decision-making under uncertainty. Spanning Bayesian estimation, experimental methods, and novel survey-based measurement techniques, the papers demonstrate how improving data quality, correcting bias, and accounting for framing can meaningfully change empirical conclusions and inform policy.

The first paper, by Ben-Shalom and colleagues, addresses a persistent challenge in disability policy: obtaining reliable local estimates from noisy survey data, in particular for rural counties and smaller demographic groups. Applying Bayesian small-area estimation to published estimates from the American Community Survey (ACS), the paper produces more stable and precise county-level estimates of disability prevalence and employment rates by demographic group. By borrowing strength across counties and groups, the approach reveals geographic and demographic patterns that are obscured in standard ACS data. This approach provides a scalable, reproducible framework for generating more precise policy-relevant statistics. These refined estimates are particularly valuable for state and local policymakers allocating resources across heterogeneous counties and demographic groups.

The second paper, by Zhong and colleagues, examines how the presentation of information influences economic decision-making. Using a nationally representative survey experiment, the study tests how alternative descriptions of lifetime income products affect individuals’ choices between converting savings into a steady income stream or taking a lump sum. The findings show that relatively small differences in wording can significantly shift choices, even after accounting for financial literacy, risk preferences, and socioeconomic characteristics. This work highlights how framing shapes observed preferences and underscores the importance of clear and effective communication in policy design.

The third paper, by Mullen and colleagues, develops a novel approach to measuring functional abilities and work capacity by addressing systematic biases in self-reported data. Combining self-assessments aligned with O*NET occupational requirement measures, cognitive performance tests, and vignette evaluations, the study documents consistent patterns of overestimation in self-assessments and stricter standards applied when evaluating others. It then introduces an adjustment method that integrates objective performance and vignette responses to correct these biases and construct revised measures of work capacity. The results suggest substantially lower estimated work capacity than unadjusted self-reports, with important implications for disability measurement and related policy analyses.

Taken together, these papers show that improving measurement and correcting bias, both in data and in human judgment, can materially change empirical conclusions and the policy decisions that flow from them.

Policy Area

• Methods and Tools of Analysis

Chair

• Na Yin, Baruch College (City University of New York)

Discussants

• Philip Armour, RAND School of Public Policy

• Jody Schimmel Hyde, University of Michigan Ann Arbor

Organizer

• Yonatan Ben-Shalom, Mathematica

Individual Presentations

•  From Noisy Data to Reliable County Disability Estimates: Improving Precision with Bayesian Methods - Presenting Author: Yonatan Ben-Shalom, Mathematica

• Wording Matters: Framing Effects on Lifetime Income Recommendations - Presenting Author: Mingli Zhong, Urban Institute; Non-Presenting Co-Author: Michael Neal, Urban Institute

• Self-Report Bias and Functional Abilities - Presenting Author: Kathleen Jennifer Mullen, University of Oregon

• Evaluating the Quality of a National Voter Registration File Using Administrative Records - Presenting Author: Andres Felipe Mira, U.S. Census Bureau