50 States, 50 Statutes: Measuring What States Actually Enacted

Session Submission Type: Roundtable

Abstract

State policy variation is a workhorse source of evidence in criminal justice research and policy analysis. Researchers use differences across states in policing, sentencing, victim protection, firearm policy, and other legal regimes to estimate policy effects and inform public debate. But that work depends on a measurement layer that is often weaker than the field assumes. Widely used policy datasets can contain errors in adoption timing, omit relevant enactments, collapse distinct policy types into a single indicator, or provide too little documentation for researchers to verify what states actually enacted. Those weaknesses matter for both research and policy. If the underlying legal measures are wrong or incomplete, the resulting evidence can mislead scholars, agencies, advocates, and funders. One widely used firearm waiting period database omits ten states that in fact enacted such laws and records effective dates that differ from the session-law record by as much as six decades—the kinds of discrepancies that can materially change estimated policy effects or make them hard to interpret.

This roundtable focuses on the policy and research infrastructure needed to measure state criminal justice laws more accurately. The motivation is practical. In ongoing work, we use large language models applied to state session laws, combined with human validation, to reconstruct policy histories directly from primary legal sources. The waiting period findings above come from that pipeline, benchmarked against leading existing sources. At the same time, this work has made clear that better tools do not eliminate the need for legal judgment, transparent coding rules, and careful validation.

The session brings together complementary perspectives on this problem: a researcher involved in building a major state policy database, a practitioner who has had to use this kind of information in real policy settings, and a methods-oriented scholar whose work on policy evaluation methods and LLM-assisted research sits at the intersection of the two. Rather than presenting a single paper, the goal is to open a broader discussion about how criminal justice policy data are built, where they most often go wrong, and what better infrastructure could look like.

Questions for discussion include what it takes to build a policy database that researchers and policymakers should trust; what LLM-based extraction from session laws can improve, and where its limits remain; how to ensure that expanded policy measurement strengthens the evidence base rather than creating new opportunities for selective reporting; and where better legal measurement makes criminal justice evaluations more credible, or leaves important limits in place.

This session speaks directly to the conference theme of “Fifty States, Fifty Systems”: variation across states is often the basis for policy learning, and the quality of that learning depends on how well that variation is measured. The session is aimed at clarifying what researchers, funders, database builders, and policy users should expect from this next generation of legal measurement—and what they should still be skeptical about.

Policy Area

• Crime, Justice, and Drugs

Moderator

• Elliott Ash, ETH Zurich

Organizer

• Zubin Jelveh, University of Maryland, College Park

Speakers

• Rosanna Smart, RAND Corporation

• Abdul Nasser Rad, University of Oxford/Campaign Zero

• Scott Cunningham, Baylor University