Examining fidelity of implementation at the systems-level: Assessing learning and uptake from capacity-building events among diverse stakeholders

• In Event: The relationship between fidelity of implementation and learning: what the data tells us

Proposal

USAID believes that strengthening local capacity will lead to locally-owned and locally-sustained development outcomes (USAID, 2018). At the project-level, the resulting theory of change posits that trainings and workshops will increase knowledge and skills among local stakeholders, who in turn will integrate them effectively into their practice, which will consequently lead to positive development outcomes. Oftentimes to measure this, projects focus on the link between a series of inputs (did training occur) as well as the outcome variables (student achievement), without also assessing the fidelity of implementation of the intermediary components (did participants learn, and did they integrate the approaches into their practice). Better measurement of these components would help implementers assess the value-added of their capacity-building activities to know what improvements could be made.

To address this within the particular context of the USAID/LAC Reads Capacity Program (LRCP), we apply the Kirkpatrick’s Learning Evaluation Model to evaluate the fidelity of implementation of pre-reading and pre-writing workshops in four project countries: Honduras, Guatemala, Dominican Republic and Nicaragua (Kirkpatrick 1959; 2009). This method includes a pre-survey and post-survey to measure participants’ learning of the specific technical content covered in the workshops as well as follow-up interviews several months afterwards to examine stakeholders’ self-reported experiences implementing the skills or approaches in their work. Where available, documents or other concrete examples of how the approaches have been utilized are collected and examined. Based on the research questions, qualitative data are analyzed using thematic analysis. A systematic review of Stakeholder Interview Guides identifies patterns and themes at both the country and regional level.

The open-ended constructs of this qualitative methodology enables the LRCP to examine changes in participants’ knowledge and understand the various degrees of uptake by participants. It is our belief that by regularly assessing these components, projects are able to; (1) improve the quality of their trainings; (2) better understand the professional development needs of their participants and; (3) more accurately target participants for additional support and follow-up. Ultimately, with this additional information, implementers can further explore the linkages between different training models and the degree of uptake by their participants.

Findings from the pre-survey and post-survey in Honduras suggest that participants learned concrete activities that they expect to integrate into their teacher training modules, curriculum documents and coaching materials. This presentation will present and discuss; (1) the strengths and limitations of the application of this methodology to measure the fidelity of implementation of capacity-building activities; (2) common themes and trends across all four countries and unique findings at the country-level and; (3) lessons learned for future adaptation and implementation.

Author

• Juan Luis Córdova, USAID/LAC Reads Capacity Program Regional Director Juárez and Associates