Validation of ROAR Assessments: Made Possible through Research Practice Partnership

• In Event: Leveraging Research-Practice Partnerships to Develop Open-Access and Scalable Reading Assessments: Rapid Online Assessment of Reading

Abstract

A Problem of Practice Leading to a Research Partnership
The Rapid Online Assessment of Reading (ROAR), is a mission-driven research-practice partnership project. ROAR provides free rapid, automated, group-administered, browser-based foundational reading skills assessments to schools (blinded for review). The de-identified data is used for research on dyslexia, reading development, and assessment. In 2023, California declared that schools would need to screen all K-2 students for reading difficulties using a state-approved screener (California Education Code 53008). Partner districts asked ROAR to try to become an approved screener. In 2024, ROAR was approved as a screener for Grades 1-2, in English, but not for Kindergarten or in Spanish because sufficient validation data was not yet available. Leaders in (blinded for review) District offered to support research towards ROAR’s future re-application because they wanted to keep the rapid, automated, group-administered screener they were already piloting.

Mutual Goals and Co-design
The ROAR team worked together with district leadership to devise a research plan that would result in sufficient statistics and meet the needs of the school. By using ROAR in the fall and spring for the purpose of this study, the district was able to replace their existing screener that required tedious 1:1 testing and had score reports that were difficult to use.

Data and Methods
In partnership with all of the district’s K-2 teachers, the district and research team divided the schools by language group. In Fall and Spring 2024-2025, K–2 students in four schools that primarily consist of Spanish immersion and bilingual education programs took Spanish ROAR, and students in the other five schools took English ROAR, including assessments of Single-Word Recognition (ROAR-Word), Sentence Reading Efficiency (ROAR-Sentence), Phonological Awareness (ROAR-Phoneme), and Letter Knowledge (ROAR-Letter). All teachers would have access to assessments in both languages for use beyond the purpose of the study. In Spring 2024-2025, over the course of 6 weeks, a team of research coordinators assessed 927 English-speaking K–2 students on the Woodcock-Johnson, TWRE, and DIBELS ORF, and they assessed 491 Spanish-speaking K–2 students on Woodcock-Muñoz and IDEL assessments, one-on-one. This required enormous cooperation, communication, and coordination across the teachers, school leaders, and research coordinators.

Results
Extending previous validation studies (blinded for review), The data was analyzed with correlational and sensitivity analyses, comparing fall and spring ROAR assessments to spring validation assessments. The results showed that ROAR has good concurrent and predictive validity in K-2 in English and Spanish (See Figure 1; rROAR-Word = 0.64–0.87; AUCROAR-Word = 0.71–0.95). Findings held across demographic groups, including race, ethnicity, gender, English learner status, free and reduced price lunch status, and special education status.

Practical and Scholarly Significance
This research showed that a group-administered, browser-based, automated, tablet/laptop-based assessment can be used to screen K–2 students for reading difficulties at the beginning of the school year, with high validity across demographic and language groups. This breakthrough can reduce the burden of testing for teachers, giving them more time to focus on instruction.

Authors

• Carrie Townley Flores, Stanford University

• Mia Fuentes-Jimenez, Stanford University

• Kelly Rose Wentzlof, Stanford University

• Jason D Yeatman, Stanford University