Impact evaluation of Espacios para Crecer (epc), an after school program, in Nicaragua

• In Event: Literacy programs in Latin America: analyzing early grade reading practices both in and out of school

Proposal

Background: The average number of years of schooling in Nicaragua’s southern Caribbean region (RACCS) is only 2.9 years. Nearly half of all school-age boys and girls are not in school and approximately 25 percent are illiterate. In 2012, only 12.5 percent of RACCS first graders scored at or above an international reading standard. To address these issues, U.S. Agency for International Development (USAID) funded the Community Action for Reading and Security (CARS). CARS consists of several activities intended to strengthen educational outcomes and community security , including Espacios para Crecer (EpC, or Spaces to Grow in English). We present findings from a randomized-controlled trial of EpC, implemented in 166 communities in the RACCS. The EpC model hypothesized that children’s participation in high quality academic and recreational activities, and in other activities that promote personal growth, would improve children’s academic and non-academic outcomes.

Objective and Research Questions: The study sought to answer three questions: (1) What services were delivered at the child, school, teacher, and community levels?; (2) What impact did the intervention have on reading and on other education, socio-emotional, and security-related outcomes?; and (3) What were the costs of EpC, and were the effects of EpC large enough to justify the costs?

Setting: The evaluation was conducted in 219 educational communities (147 medium and 52 large) across five RACCS municipalities in Nicaragua. An educational community is defined as a school and its immediate catchment area.

Sample: Children were eligible to participate in EpC if they were 5 to 16 years old and either enrolled in school and identified by program staff as being at risk, or were not enrolled in school during the intake process. Our analytic sample consisted of 2,820 eligible children, randomly sampled from each educational community.

Intervention: EpC is a half-day program that was implemented over an 18-month period in the RACCS. It was offered to children who were deemed at risk, including those who were having difficulties at school, dropped out of school, or never attended school. Program administrators delivered learning materials that met the linguistic needs of children in the RACCS. EpC facilitators were trained on: (1) novel teaching techniques and recreational activities designed to facilitate learning and create a positive social environment; (2) social development and academic leveling activities; and (3) methods to respond to children’s personal needs, including the use of formative reading assessments.

Research Design: To estimate the intervention’s impacts, we randomly assigned children or educational communities to treatment or control groups, depending on the size of the educational community. In larger communities (with more eligible children), we randomly assigned children to the treatment or control group. In communities with so few children that it was not feasible to form two separate groups , communities themselves were the unit of random assignment, with all children assigned together to either the treatment or control group. Having all eligible children in the same research group minimized contamination between the groups.
Data Collection and Analysis: Data collection followed the roll out of the EpC intervention, which occurred in phases: Cohort 1 started in 2014 and Cohort 2 in 2015. Intake data were collected before activities began in each community, and included key socioeconomic information on all participants. Intake data represented the baseline data for the project and were also used to conduct random assignment. Additionally, we conducted a base-year household survey for the Cohort 2 study sample. Base year data from Cohort 1 were not collected due to the timing of intake and project rollout. At endline, outcome data were collected for each cohort, in 2016 for Cohort 1 and 2017 for Cohort 2. We administered a household survey that included caregiver and child questionnaires, as well as a child reading assessment in the community’s language of instruction. We also administered questionnaires to school administrators, teachers, and EpC facilitators that asked about teaching practices and school and classroom characteristics that could affect children’s learning.

To estimate the impacts of EpC we conducted multivariate regression analyses. Dummies were used to control for child demographic characteristics and to account for design features (e.g., stratification and clustering of children in educational communities).

Findings: EpC was implemented largely as intended. EpC were opened in targeted communities, materials were delivered to those EpC, and enough educators were trained to operate EpC. However, there were delays in the distribution of materials, difficulties engaging out-of-school children, and fewer than expected follow-up visits for EpC facilitators. In addition, there was lower than expected participation of children assigned to the treatment group, together with non-trivial participation from children assigned to the control group. EpC had impacts on some of the key children’s outcomes of interest (i.e., progression through school, invented-word decoding, reading fluency, and reading comprehension) but impacts were found in medium communities only. Moreover, learning impacts were driven by impacts on girls and on children who were out of school at intake. The evaluation did not detect impacts on children’s social-emotional outcomes. After controlling for base-year differences in learning skills, impacts on literacy skills were attenuated and became non-significant or marginally significant. This suggests that such impacts may have been generated rapidly and did not increase over time.

Conclusions: The evaluation EpC adds to the field of comparative and international education by rigorously evaluating whether a program that offers social development and academic leveling can improve child outcomes for high-risk children on the Atlantic Coast of Nicaragua. Although the study did not detect impacts of EpC on all hypothesized outcomes for all populations, there was evidence of improvement in some early-grade reading skills in largely rural, medium communities. We will discuss how to interpret our findings, including characteristics of the local context and evaluation design that may underlie the lack of impacts in large communities, differential impacts by child’s gender and school-enrollment status, and how findings change when controlling for base-year outcome differences in a portion of the sample.

Authors

• Catalina Torrente, Mathematica Policy Research

• Emilie Bagby, Mathematica Policy Research

• Ivonne Padilla, Mathematica Policy Research

• Nancy Murray, Mathematica Policy Research

• Steve Glazerman, Mathematica Policy Research