The Critical Role of Implementation Data in Replicating High-Impact Programs

• In Event: The Critical Role of Data in Translating Evidence to Practice

Abstract

Roca, Inc. is a nonprofit organization that serves the highest risk youth in the Greater Boston area—young men aged 17-24 years who have had involvement with the justice system, may be involved with gangs or drugs, have dropped out or are at-risk of dropping out of high school, and have no history of stable employment. As part of their participation in the Commonwealth of Massachusetts’ Juvenile Justice Pay for Success project, Roca rapidly expanded their service capacity, adding an additional service location and approximately 200 slots to serve their high-risk, high-need population. Such rapid capacity expansion and replication can threaten implementation fidelity, risking a program’s effectiveness in serving participants and achieving outcomes, and potentially undermining its reputation. Data on staff and participant performance, organized along metrics that align to the program’s model, are critical for monitoring program fidelity and evaluating the influence of program adaptations during implementation on measured outcomes. In this presentation, we describe the components of Roca’s model and the data sources tied to these components and present findings from an evaluation of Roca’s implementation fidelity during replication. We will discuss the implications of findings for Roca’s future replications and describe the tools necessary for successful replication.

Authors

• Holly Swan, Abt Associates, Inc.

• Sarah Jalbert, Abt Associates, Inc.