Program evaluation of cross-country comparison – ASER, a common reading tool

• In Event: Lisez-vous Français? Early Grade Reading in Francophone Africa: A multi-country comparison of assessment tools, data, and program design

Proposal

IMPAQ is involved in the program evaluations of Food for Education (FFE) programs in Ivory Coast, Burkina Faso, Senegal, and Mali. All 4 FFE programs include education components aimed at improving literacy among students from primary school. We will present comparatively the evaluation designs for each country (e.g., randomized control trial in Burkina Faso and quasi-experimental designs the other three countries); the evaluation timelines; sample sizes of program beneficiaries; and instruments used for data collection (e.g. surveys and reading assessments).

To measure the program effectiveness in improving literacy skills, we used ASER assessment in French. The ASER tests are designed as criterion-referenced assessments to categorize children on an ordinal scale indexing mastery in each of the basic skills in reading (letter identifications, word decoding, etc.). The ASER reading test can classify children at the ‘nothing’, ‘letter’, ‘word’, ‘sentence, or ‘story’ levels based on defined performance criteria or cut-off scores that allow testers to classify children as masters or non-masters of any given level. We will discuss the choice of instrument in each country context, we will provide an overview of the tool, and we will explain how it was adapted and calibrated with the help of local reading, curriculum, and assessment experts from Ministry of Education and with the support of program implementers.

IMPAQ team will present baseline findings for each country. We will share distributions of reading skills by grade levels representing the proportion of students in each reading skill. We will also present baseline reading levels by student gender. We will briefly discuss what the baseline results tell us about literacy in the region (e.g. common low levels of reading at all grades, girls demonstrating lower reading skills than boys). Applying the same reading measure ASER offers an opportunity to compare findings across countries. We will highlight the need to carefully account for different educational, social, economic and even political context in understanding the baseline findings.

We will discuss challenges in comparing baseline reading outcomes across different contexts and different programs theories which will set the stage for the next presentations.
We will also discuss the necessity to have clear understanding how each program’s theory of change is intended to lead to improved reading. The next presentations and discussions will also help shape up the additional data (e.g. interviews, focus groups, observations) that can be collected in each country to shed light on the mechanisms of change.

Authors

• Michaela Gulemetova, IMPAQ International

• Laurence Dessein, IMPAQ International

• Elnaz Safarha, IMPAQ International