Applying best practices for EGRA benchmarking: Lessons learned from standard setting in India

• In Event: Rethinking early grade reading benchmarks: Innovations in setting oral reading fluency standards

Proposal

Purpose: The purpose of this study was to apply lessons learned and best practices from the recent EGRA Benchmarks and Standards Research Report (RTI International, 2018) to a five-language benchmarking activity for early grade reading in India.


Framework: The main purpose of oral reading fluency benchmarking is to simplify EGRA results into a single indicator against which countries can measure their children’s reading progress. The most common data-driven approach to benchmark setting is to take the median ORF from all students reaching the comprehension threshold (e.g., 80% comprehension) and to work with language and education specialists set a standard near that mark. However, this estimate is sensitive to the sample characteristics (e.g., if data are collected from high-achieving students, benchmark estimates will be higher). Additionally, this approach does not account for the low precision of the estimates that are obtained when only a small sample of students reach the threshold. Lastly, the benchmark is reliant on relationship between two subtasks, which is not always consistently reliable across languages. Therefore, there was a need to develop new methods/approaches for setting EGRA benchmarks.


Methods A benchmarking workshop was held in New Delhi during the first week of May 2018. The goal of the workshop was to use project baseline data (from approximately 14,467 students) to set reading benchmarks for each of five languages. There were three main innovations for the approach used in this workshop: 1) data included both EGRA and ASER subtasks, as opposed to just a single assessment; 2) there was no a priori assumption about which variable would be used for benchmarking, so participants were provided with analyses and relationships across all administered subtasks (EGRA and ASER); 3) in addition to the traditional mean/median method, all cross-subtask relationships were estimated using a logistic regression method (estimating the point on the first subtask at which a student has a .5 probability of scoring proficiently on the second subtask)—which allows for simpler calculation of estimate precision.

Results: Upon review of the data, all groups concluded that the most appropriate benchmark for these purposes would be a measure of oral reading fluency that constitutes a proficient level of reading comprehension. Participants also felt strongly that it was best to use all available information in order to inform their decision about where the oral reading fluency benchmark should be set (i.e. using both the mean and logistic regression methods). Ultimately, providing estimates from two assessments, using a variety of methods, in a workshop with participants from 20 organizations led to commonly agreed upon and supported benchmarks for each of five languages.

Significance: As the first large-scale benchmarking activity across languages in India, this activity has provided insight into best practices for EGRA benchmarking and will serve as the starting point for a larger conversation regarding the appropriate methods for estimating and using early grade reading benchmarks in India (beyond the scope of these projects).

Author

• Jonathan Stern, RTI International