An Angoff Method for setting oral reading fluency and comprehension benchmarks in Lebanon

• In Event: Rethinking early grade reading benchmarks: Innovations in setting oral reading fluency standards

Proposal

Objectives or purpose: The purpose of this presentation is to describe a performance-based standard setting methodology for establishing benchmarks on an oral reading fluency and comprehension assessment.

Research methods or modes of inquiry: A Yes/No variation of Angoff method (Impara & Plake, 1997; Plake, Ferdous, & Buckendahl, 2005) was used for setting performance standards for grade 2 Arabic reading fluency and comprehension test in Lebanon in 2016. The students were classified into four performance level categories: beginner, emerging, proficient, and advanced. The Ministry of Education and Higher Education (MEHE) recruited a total of 17 panelists for the standard setting study; 12 of them were grade 2 Arabic language teachers and 5 were reading experts from the MEHE.

A three-step process was implemented for setting the performance standards: prior to, during, and after the standard setting workshop. Prior to standard setting workshop, the MEHE in collaboration with a group of Arabic language, curriculum, and pedagogy experts developed policy definitions and detailed performance level descriptors (PLDs) for Grade 2 reading fluency and comprehension. The panel ensured that the PLDs were aligned with grade level expectations as defined in their Arabic language curricula.

During the standard setting workshop, panelists were introduced to the policy definitions and detailed PLDs, and an overview of the Angoff procedure that would be used in setting the performance level cutpoints. After that, the panelists were engaged in discussion of the knowledge, skills, achievements, and abilities of students who were at the cutpoints for Emerging, Proficient, and Advanced levels of reading fluency and comprehension performance (these students were described as just emerging, just proficient, and just advanced). They were then handed out Grade 2 item response form to record their individual and independent judgments about just emerging, just proficient, and just advanced students’ performance in reading the passage and answering comprehension questions. The panelists were given two chances (also called “rounds”) to provide their independent judgements, and feedback data was provided to them between the rounds. Following the round 2 judgments, panelists anonymously filled out an evaluation form that asked about their confidence and comfort levels in establishing the cut scores.

After the standard setting workshop, the recommended cutpoints (from Round 2) with ±3 standard error of estimate (SEE) adjustments and their corresponding impact data (e.g., percentage of Grade 2 student meeting at least proficient performance level category) were presented to the MEHE. The MEHE chose the cutpoints that they thought were most appropriate, meaningful, and achievable for Lebanese children.

Results: The MEHE set the cutpoint for emerging, proficient, and advanced categories at 29, 44, and 55 correct words per minute (CWPM) respectively on the oral reading fluency test. They also set reading comprehension cutpoints at 3 out of 6 questions for emerging, 4 out of 6 for proficient, and 5 out of 6 for advanced category. It was revealed from the evaluation of the standard setting process that 94% of the panelists were confident and comfortable with the resulting benchmarks.

Scholarly significance, originality and/or creativity of the study or work: The Yes/No method has been used with multiple assessment programs (e.g., South Dakota, Virginia, Massachusetts, PARCC) to set cutpoints for student performance. The application of this method is unique for determining cutpoints for reading fluency and comprehension consistently and concurrently.

Author

• Abdullah Ferdous, MSI