Implementing an Innovative Approach to Cost and Implementation Research in the Context of Program Evaluation

• In Event: Program Evaluation in a Large Urban District: A Nuts-and-Bolts Perspective

Abstract

Objectives of purposes: In this randomized control trial, costs and implementation are addressed systematically in coordination, such that the task of conducting research on both components is simplified. The purpose of this paper is to detail the research team’s process for conducting concurrent cost and implementation studies in the context of the RCT study. It will address:
• Our team’s process for collaboratively and simultaneously developing the research designs for these two related components, creating shared instruments, and data collection and analysis
• The communication and coordination required to implement this collaborative approach
• The benefits, challenges and trade-offs our team has identified through our concurrent approach to cost and implementation reserach

Perspective(s) or theoretical framework: Field trials are conducted to provide information about a program’s effect on an outcome of interest. The primary focus of the research within an evaluation is estimating impacts. However, within the evaluation, it is also critical to examine the processes and resources utilized during implementation in order to understand the contrast between the treatment and control groups. Traditionally, implementation studies are conducted concurrently with evaluations, and cost studies are conducted retrospectively (Levin, McEwan, Belfield, Bowden, & Shand, 2017; Century & Cassata, 2016; Hulleman & Cordray, 2009). Separating these components duplicates effort and is ultimately less precise than a coordinated approach. Our work in designing a collaborative, concurrent approach is therefore a contribution to both the cost and implementation literature.

Methods, techniques, or modes of inquiry: These components of the evaluation rely upon program management information system data, school district data, classroom observations, interviews, teacher surveys, and teacher time logs. From these data, the costs will be estimated following the ingredients method. It is important to note that because data collection on resource use is done simultaneously with data collection on implementation, in this study we are able to estimate costs for each classroom in each school.

Data sources, evidence, objects, or materials:Data collection tools and processes for integrating cost and implementation research were designed for this study, and will be detailed in this paper. Concrete examples will be provided of integration in interviews, surveys, observations, and time logs.

Warrants for arguments/point of view :Guidance is provided based upon this exemplary work, which followed standards set forth by the What Works Clearninghouse, as well as those specifically addressing cost research (Levin & Belfield, 2015).

Scientific significance of the work: Because most cost analyses are generally done retrospectively and because resource information is extremely challenging to recover, most cost analyses on education have relatively small sample sizes and are subject to the usual objections that come with it. Therefore, this study provides a novel large dataset on costs on education that is seldom seen in the literature. This paper will serve as a reference for any program evaluation in education or other social science areas that is tasked with rigorously examining costs and implementation contemporaneously.

Authors

• A. Brooks Bowden, North Carolina State University

• Viviana Rodriguez, Columbia University

• Ryan Fink, University of Pennsylvania