Benchmarking for Proficiency in Reading and Mathematics: Improving Methods for Data Analysis and Stakeholder Engagement

Group Submission Type: Panel Session

Proposal

Sustainable Development Goal (SDG) for education includes an indicator (4.1.1) on the proportion of children achieving at least ‘minimum proficiency’ level in reading. With increased availability of early grade literacy data, defining and reporting against expectations of ‘minimum proficiency’ is a critical step towards establishing benchmarks that allow stakeholders to track system-level performance and progress towards improved reading skills. Demonstrating the reliability of the data used in benchmarking and providing measures of the validity of the relationship that underpins the benchmarking process are critical to ensuring broad acceptance of established benchmarks. As practitioners, making the case for the reliability of the data also increases the likelihood that the proposed benchmarks will be officially adopted and used by stakeholders.
This session will unpack what it means to establish meaningful benchmarks – through the intended purpose of a benchmark and resulting implications for policy and practice, differences (or not) across languages and orthographies, populations, and methodologies. The session will be chaired by a moderator, who will provide a brief background of what benchmarks are and provide examples of benchmarks for oral reading fluency established using Early Grade Reading Assessment (EGRA) data. This will be followed by four presentations. The first presentation will focus on the process of setting benchmarks across 40 languages, specifically examining differences across higher- and lower-performing populations, language, and the purposes of benchmarks. The intended purpose of a benchmark directly dictates how each country may choose to define ‘minimum proficiency’ and in turn, the level at which a benchmark can be operationalized. The presentation also reviews effective methods for engaging policy makers and other stakeholders in the benchmarking process. The second presentation will present an analysis of 75 datasets across nine languages and seven countries of the effectiveness of fluency benchmarking methods. Three questions were examined in this analysis, including the differences across methods, factors affecting level and precision of an estimate, and the utility of the resulting benchmarks. The third paper will build on the second paper, and examine the implications of these methodological considerations on programming and policy at the country-level. Drawing on the analysis across 75 datasets, this presentation will highlight implications for stakeholders in adapting to country-specific benchmarks, instrument development and drawing conclusions across student populations of varying characteristics. The fourth paper will examine the reliability of benchmarks using a method reviewed in the first paper and compared in the second and third papers. Comparing the consistency of classification of students (as either meeting a benchmark or not) across populations and EGRA test forms, the presentation underscores the need for an iterative benchmarking process that yields reliable results on which meaningful inferences can be drawn.
A discussant will guide discussion about how to make meaning of issues raised by the various perspectives on the benchmarking process, as well as potential opportunities and lessons learned.

Sub Unit

• General Pool

Chair

• Mitch Kirby, US Agency for International Development (USAID)

Individual Presentations

• Review of benchmarking processes in 12 Countries and almost 40 languages - Joseph Destefano, RTI International

• Analyzing the effectiveness of fluency benchmarking methods - Chris Cummiskey, RTI International; Matthew Jukes, RTI International

• Data-driven oral reading fluency benchmarks: Implications for measurement, programming, and policy at the country-level - Christine Beggs, Room to Read; Peter Cooper, Room to Read

• How reliable are benchmarks? Examining the consistency of classification of students reading fluency across test forms and populations - Hetal Thukral, School-to-School International

Discussant

• Luis Crouch, RTI International