Tightening what matters: A research-practice partnership focused on understanding and addressing variability in critical drivers of impact

• In Event: Highlighted Session: Towards effective scaling and systems-change: Lessons from implementation research on improving foundational learning in India, Kenya, Uganda, and Senegal

Proposal

This paper presents an exemplar implementation research study and collaboration aimed at deepening organizational learning. It is animated by two convictions: 1) To be optimally effective, implementation research must be strategic; innumerable factors can be explored and change efforts attempted and studied, and not all are equally valuable for yielding improvements. 2) Secondly, the benefits of implementation research and research-practice partnerships should go beyond the lessons-learned from study findings, building capacity within organizations to think and act strategically about theories of action (ToA), measurement, and improvement, to learn fast and deepen impact at scale.

As Bryk et al. (2015) note, most interventions’ have an underspecified ToA. Thompson and William’s (2007) Tight but Loose Framework offers a way of strategically specifying aspects of one’s ToA that must be “tight,” namely, those most critical to impact. As emerging research has emphasized (D’Agostino et al., 2024; Angrist & Meager, 2023), variable dosage and fidelity powerfully explain variability in the effectiveness of interventions, corresponding with large differences in average effect-sizes when taken into account. As such, key dosage measures of critical program components tend to be the most important aspects of a ToA to be tightly delivered.

This paper will focus on the findings of an integrated implementation research study including: 1) an organizational case study, 2) a mixed methods study of variation in a key dosage indicator, and 3) an improvement process aimed at both strengthening BT’s learning approach and deepening their specification and tightening aspects of their ToA. It focuses on the number of camps delivered across schools and communities, a primary dosage indicator of its core TaRL-based intervention.

A qualitative case study first explores how BT staff conceptualized key performance indicators and used M&E systems, feedback loops, and support systems to provide quality control and strengthen performance prior to the capacity strengthening process. It identified strong systems but an inadequately strategic attention to key leading indicators and opportunities for a more strategic orientation to adaptive management driven by a clearer articulation of critical tight aspects of the ToA.

Secondly, BT leadership and ND faculty collaborated on a process of deepening the specification of BT’s ToA, with a focus on core elements that should be tight to maximize impact. The BT team collaboratively identified key dosage and fidelity indicators related to the delivery of TaRL school and community camps as key tight drivers. The concepts of leading and lagging indicators were introduced and suggestions explored for how to strengthen timely, adaptive management practices capable of “tightening” take-up and dosage for the target number of camps delivered.

Finally, we report on findings from a mixed-methods study exploring factors explaining variability in the number of camps delivered in the two-year intervention period. The design uses monitoring data, surveys of teachers, head teachers, volunteer camp facilitators, and program implementation support personnel, and a qualitative sample of schools with varying levels of camp dosage, a form of positive/negative deviance study.

Authors

• TJ D'Agostino, University of Notre Dame

• Whitney Warren, Building Tomorrow

• Maurice Sikenyi, University of Notre Dame

• Fernanda Soares, University of Notre Dame