Evaluating Locally Responsive Funding Interventions Through a Cost Analysis Framework

• In Event: AERA Roundtable Session Thursday 5:25 pm Four Seasons Ballroom 2-3
  In Roundtable Session: Costing the Rubicon: Addressing Methodological and Conceptual Complexities in Cost Analysis (Table 10)

Abstract

Objective
Locally responsive policies, wherein federal or state policies allow local agencies to determine the interventions they wish to implement to meet the program’s goals, are an increasingly popular policy. These programs allow interventions to be responsive to the local needs, something uniform implementations at a higher bureaucratic level cannot typically accomplish. For cost studies, the variance in how local agencies choose to implement the policies can make for complicated evaluations, since unlike traditional interventions, locally responsive interventions are not clearly defined, since the local agency is at liberty to implement with only mild stipulations.
In our paper, we will illustrate the substantial variation in cost and the challenges of conducting cost analysis on school-level locally responsive interventions using 6 case studies across two locally responsive social emotional learning interventions in Illinois. Both policies are locally responsive towards the goal of improving social emotional learning. We show that this local responsiveness leads to substantial fluctuations in assessed societal cost across program sites.
We extend our findings by arguing that while the programs are similarly structured and both show substantial differences in estimated costs across sites, our ability to meaningfully interpret what the substantial variance means for the program is completely different across our two interventions. While space does not permit a detailed elaboration, we wish to use our paper to highlight how very similarly structured programs may be differentially appropriate for cost study design, and ideally facilitate a conversation with the audience on our approach and interpretation of our findings and implication for other cost analysis on locally responsive interventions.
Data and Methods
We have collected data from all six sites, which includes program records, surveys, interviews conducted by the research team, and public data sets on student and school characteristics. These data allow us to develop resource cost models for each case study site using an ingredients method approach (IES, 2023).
Results
We are now completing data collection and will finalize analyses this fall. In initial analyses of the three sites that each of the two programs is implemented at, we observe substantial variation in how schools chose to implement the two policies, and thus we observe varying demands on space, materials, and time from school staff. This observed variance within both programs gives us strong confidence that our final analysis will be able to illustrate the difficulty of evaluating locally responsive programs and demonstrate how similarly structured programs may still vary in their appropriateness for cost analysis due to what these variation in costs tell us about the program.
Significance
Beyond providing novel insights into the costs of social emotional learning interventions, this study will demonstrate the nuances of utilizing and interpreting cost analysis, which is an increasingly prevalent component of education research. By demonstrating how cost analysis has differing utility across two similar locally responsive programs, we aim to help other researchers to consider the appropriateness of cost studies in their own work and how this is dependent on careful case-by-case consideration of the policy itself.

Authors

• Christopher Brooks, American Institutes for Research

• Stephanie Levin, American Institutes for Research

• Karthik Ramesh, University of Washington

• Katherine Laird, American Institutes for Research

• AnhMinh Nguyen, American Institutes for Research