Reading Profiles of Growth as an indicator for program evaluation

• In Event: Youth and immigration: Evaluating the effectiveness of intervention programs

Abstract

In this paper we describe the initiative to include growth in the monitoring and evaluation plan of the USAID Lifelong Learning project. The reason to include growth as an outcome indicator, as opposed to status or percentage of students who achieve national performance standard, is to demonstrate effectiveness of schools where the program intervenes with a better outcome measure. Betebenner and Linn (2009) have discussed the downside of achievement or status as an indicator of effectiveness since is blind to the possibility that students in highly effective schools do not achieve performance standards because of their characteristics related to poverty. Growth on the other hand is a better indicator of school efficiency in highly disadvantage contexts.

Thus, in addition to including growth in the monitoring and evaluation plan, the initiative includes an innovative methodology to summarize and report growth to different stakeholders related to USAID Lifelong Learning Project, under the label of progress profiles.

A progress profile answers three questions: are students learning? What does it mean to grow in the elementary grades in Guatemala? And is student academic growth in correspondence with national curriculum? A progress profile, in this study, is a summary that provides concrete descriptions of growth based on information from different growth models: 1) the gains model (DePascale, 2006), 2) the Student Growth Percentile Model (Betebenner, 2011), 3) the categorical model (Ho & Castellano, 2013), and 4) the item growth model. The four models chosen provide descriptions of growth under different perspectives and correlate among each other. Calculations of the models were done using the same assessment at the beginning and end of a school year in two languages (Mayan and Spanish).
The item growth description is based in two procedures: 1) estimated average theta at the beginning and end of the school year and its correspondent representative item, and 2) separate calibration of items, at the beginning and end of school year, to verify a decrease in difficulty.

The following is an example of an individual progress profile for a student:

María is a second grade student who turned 10 in 2013. She is Mayan and she is in Miss [206] classroom of school [09-13-0375-43]. At the beginning of the year she could only answer 9 items correctly in the national reading assessment for first grade in Spanish. However, by the end of the year she answered 13 correctly, which represents a gain of 4 points, and a 0.924 progress in the ability scale. An example of the type of items she can answer now is finding the main character in a three-sentence story. This item is harder than she could do at the beginning of the year where she could only read single-sentence items. By the end of second grade, she achieved first grade reading standard. In relation to her peers with the similar ability at the beginning of second grade, she has made substantial progress; she performed better than 71% of her peers. She made no progress in reading in K’iche’ (Mayan Language).

The previous progress profile is a typical one for the region where the study was carried on in Guatemala. In general, students learn but their progress is minor. In fact, many students (about 30%) decline instead of growing. Most students are a year behind in achieving grade level standards. And very few demonstrate progressing in the Mayan languages of the country. Progress profiles are then classified into positive and negative profiles according to established criteria.

In the context of our country, there are at least four practical and policy implications of the study: 1) we are building a system that allows collecting achievement information at least twice for each student during the life of the project, including creating new assessments for grades were no assessments are available; 2) interventions directed to teachers are focalized based on status and growth results taking into account, regional languages; 3) combining models that correlate highly with each other explains growth under different perspectives to stakeholders and; 4) there is an emphasis to report meaningful results of learning to education practitioners.

Authors

• Fernando Ernesto Rubio, Juarez and Associates, Inc

• Leslie Rosales de Veliz, USAID/Lifelong Learning