Searching for common ground: A comparison of minimal oral reading fluency benchmarks established using local expert knowledge versus data-driven processes

• In Event: Rethinking early grade reading benchmarks: Innovations in setting oral reading fluency standards

Proposal

ORF benchmarks have been established using sophisticated, data-driven methods developed specifically for use with EGRA data sets (see Jukes et al., 2017). Each of the four methods (mean, median, linear regression, logistic regression methods) has specific advantages and limitations (see discussion, Jukes et al., 2017). However, they share two significant disadvantages. The first is that decision makers need to have a certain level of statistical knowledge to engage in the decision-making process. Because of that, the process is usually either restricted to those with the required statistical knowledge, or enlarged to include a broad range of stakeholders, many of whom do not have the pre-requisite knowledge to engage fully and authentically in the decision-making process. Both scenarios can result in benchmarks that are neither widely-owned nor perceived as particularly valid or meaningful. This has implications for their use over the short and medium term, and for the sustainability of reading reform initiatives in general.

The second is that none of the data-driven methods align with recognized international best practices for establishing benchmarks or cut scores (i.e., the Angoff method, the Bookman method, the Nedelsky method, the Edel method…). All of these methods place decision making firmly in hands of local subject-matter experts (experienced, master teachers, curriculum and content specialists…), who are asked to analyse the assigned tasks and propose minimal performance benchmarks based on the experts’ knowledge of the content being assessed and their familiarity with pupil performance. An iterative, mediated process whereby subject experts’ share their differing perspectives leads gradually to consensus. The local-expert driven process, with its reliance on local expert knowledge, can produce benchmarks that are perceived by government-decision makers as more credible than those established using statistically complex and often-times opaque data-driven processes. This is particularly true if the latter processes are led by outside experts, using methods unfamiliar to national evaluation experts.

This presentation compares minimal ORF benchmarks established in Rwanda in 2011 using the data-driven median method with those established in 2018 using a modified Angoff method and the resultant impact on the percentage of pupils minimally meeting grade-level expectations between 2011 and 2018. It then looks beyond the benchmarks themselves to examine the advantages and challenges associated with each method, including the level of Ministry involvement in and their perceived ownership of decision-making process and the benchmarks themselves. The latter is critical for sustainability, the theme of this year’s conference. The findings have the potential to inform benchmarking processes in other countries.

Authors

• Norma Evans, Evans and Associates

• Alphonse Sebaganwa, Rwanda Education Board

• Sharon Haba, Chemonics International