Mismeasuring Impact: How Randomized Controlled Trials Threaten the Nonprofit Sector

• In Event: Mismeasuring Nonprofit impact: What Goes Wrong and How to Do It Better

Abstract

This project examines the role of randomized controlled trials (RCTs) in the evidence-based policy movement, especially for the public-private system of social service provision that characterizes the United States. Although most social services—workforce development, behavioral health care, family support, and more—are supported by public funds, they are delivered by nonprofit organizations in communities across the nation. Evaluation and evidence standards coming from the public sector thus are of critical interest to nonprofits. This presentation examines how RCTs came to be seen as the most legitimate form of evidence that nonprofit programs “work,” what really happens when RCTs are carried out inside nonprofits, and why the RCT method is poorly matched to the goals of the nonprofit sector. The authors go beyond existing critiques of RCTs to address unintended consequences for nonprofits’ goals of equity, sustainability, responsiveness, and innovation. Drawing on interviews with professional evaluators, foundation program officers, and nonprofit managers, Mismeasuring Impact identifies five problems with using RCTs in nonprofits:  1) “False Certainty”: RCTs aren’t a foolproof method of evaluation; 2) “Programs Need Organizations”: RCTs assess programs, but programs are embedded in organizations; 3)“Communities Need Organizations”: RCTs threaten the community-level benefits provided by nonprofits; 4) “Rich Get Richer”: RCTs advantage well-resourced nonprofits whose way of working is more easily adapted to RCT demands.  And 5) “Agility”: As high-stakes assessments, RCTs may hinder nonprofit responsiveness and innovation.

Author

• Nicole Marwell, University of Chicago
  Presenting Author