Quality of USAID-funded evaluations: Findings from a review of 92 evaluations

• In Event: Evaluations of USAID-Funded Education Projects: Quality Review and Synthesis of Findings (2-part panel, first part)

Proposal

A team led by Management Systems International led the assessment of the quality of almost 100 USAID-funded evaluations in the education sector from 2013 to 2016. The Office of Education set two main objectives for this study:
1. Develop a tool for appraising the quality of evaluation reports that is responsive to USAID’s cross-sector guidance on evaluations as well as being sector-specific to education evaluations;
2. Ensure that the information resulting from the application of this tool to a multitude of evaluation reports can be used to identify areas of strength and weakness in the evaluations funded by USAID in the education sector.
The Office of Education set the following inclusion criteria for evaluations to be reviewed in this study: (i) USAID-funded evaluations of education interventions; (ii) Performance and impact evaluations (additionally, the Office of Education requested the inclusion of a small number of research studies that did not evaluate a specific intervention); (iii) Relevant to the Education Strategy; (iv) Published between 2013 and 2016; (v) Single, latest published report (in case of reports for multiple phases of an evaluation); and (vi) Evaluation reports from multiple countries (in case of a multi-country education intervention).
For the review process, the Office of Education requested that organizations nominate staff to serve as reviewers on this study. This served three purposes: gathering broad feedback on the tool, disseminating the BE2 framework based on seven principles of quality, and providing an opportunity for community members to read and discuss each other’s evaluations. Thirty-six reviewers from 21 organizations generously contributed their time and expertise for this study. The reviewer team was complemented by MSI home and field staff and consultants. The study team developed an online platform for each evaluation to be reviewed by two reviewers. Each pair of reviewers also met virtually to reconcile any differences in scoring and produce consensus responses. In addition to piloting the evaluation quality tool, members of the education community also agreed to participate in a full-day reviewers’ meeting.
This presentation will discuss the following results:
• Overall feedback on the evaluation quality tool was very constructive, aimed at improving the tool, with comments centered around the tools breadth, length, applicability, items and scoring. Importantly, reviewers also raised concerns, mainly that the quality standards need to be responsive to the constraints that implementers and evaluators face, such as requirements in the procured statements of work (SOWs).
• The consensus ratings of the co-reviewers of each evaluation suggest areas of strength in current practice and areas that might be lacking. Most evaluations satisfied basic reporting requirements. This aligns with previous assessments of the compliance of evaluation reports across E3 sectors, which showed that evaluations in the education sector tended to outperform evaluations for the other E3 sectors in this regard. Overall, evaluations reviewed showed greater strength in cogency, conceptual framing, robustness of methodology, and openness and transparency, and greater weakness in validity, reliability, and cultural appropriateness. These results align somewhat with findings from a recent performance audit which reviewed final evaluation reports for several foreign assistance agencies , which may speak to the difficulties of evaluating programs in often challenging environments. Notably, the results suggest that country income level, crisis and conflict status, and Education Strategy Goal are poor predictors of whether evaluations adequately addressed principles of quality, whereas evaluation type (impact evaluations, qualitative performance evaluations, quantitative performance evaluations) was often a strong predictor.

Author

• Thomaz Alvares de Azevedo, MSI