Meeting the Challenges of Rigorously Evaluating Child Welfare Programs

Session Submission Type: Roundtable

Abstract

Evaluating child welfare programs and services requires cooperation and coordination with agency administrators, program managers, and front-line workers. The evaluation can require staff to conduct additional activities, organize their work differently from their normal procedures, and involve connections with other agencies not involved with the particular program or service. Most of these challenges can be overcome when agencies feel committed to understanding whether a program or service “works” and when researchers design and resource their evaluation to meet the needs of the agency. This roundtable brings together researchers who have experience working with child welfare agencies and service providers to conduct rigorous program evaluations, and agency staff who have experienced evaluations from the “inside.” The researchers will discuss key challenges they have observed when considering evaluations involving child welfare agencies such as (1) agencies don’t typically plan for the resources required for evaluations; (2) staff burden is significant; (3) data sharing requires substantial resources and collaboration within and across agencies; (4) timelines for sharing results back with agencies are long; and (5) resistance to randomized control trials (RCTs) and rigorous quasi-experimental designs (QEDs) is common. Next, agency staff will share their reactions and thoughts about the challenges identified by the researchers as well as their own experiences of contending with competing demands, altering established procedures, creating tracking systems, and sharing data across agencies and with evaluators. They will also describe the factors that influenced their agency’s willingness and ability to undergo a rigorous program evaluation (e.g., input into study design, resources to support the randomization and data sharing process, commitment to evidence-driven decision making). Panelists will provide specific examples of evaluations that reflect a diverse set of programs and contexts, including an evaluation of a program for transition age youth in New Jersey; one on New Jersey’s Keeping Families Together program, which provides housing vouchers and services to families in the child welfare system; an evaluation of an intensive, in-home clinical treatment program in Connecticut for families with children (birth to 5 years old inclusive) who are at risk for abuse and/or neglect, poor developmental outcomes and removal from their home due to parental substance use; an Connecticut program serving families using the Multi Dimensional Family Treatment and Recovery Model for adult caregivers that have substance use concerns and are pregnant or have a child 0-5; and an evaluation of HUD’s Family Unification Program (FUP), which provides housing choice vouchers to families in the child welfare system. Finally, panelists will engage session participants in a facilitated, interactive discussion about solutions that can support rigorous evaluation while minimizing the imposition on the agency, and considerations for determining whether a particular program and agency are well suited for implementation of a RCT or QED.

Policy Area

• Family and Child Policy

Moderator

• Bridgette Lery, Urban Institute

Organizer

• Michael Pergamit, Urban Institute

Speakers

• Pamela Lilleston, State of New Jersey

• Mark Courtney, University of Chicago

• Shannon Michelle Quinn, Washington State Department of Children, Youth & Families