Developing a Formative Academic Reading Comprehension Assessment for English Language Learners: A First-Year Pilot Study

• In Event: Developing and Using Formative Assessments for English Language Learners

Abstract

This study is part of a multi-year, federally-funded project to develop and validate a formative reading comprehension assessment for ELLs in middle school. The improvement of ELL students’ reading proficiency is of pressing importance, as it is a foundational skill for both academic attainment and vocational progress. Despite this urgent need, teachers often find the challenge of helping ELL students develop academic literacy a daunting task due to the students’ diverse English proficiency levels and the steeply increasing complexity of texts encountered in higher grades. In middle school, the complexity of texts in terms of both academic language and cognitive demands compounds many students’ reading difficulties. Prior research stresses that ELL students must be explicitly taught academic language proficiency and comprehension skills to improve “reading to learn” (August & Shanahan. 2006; Francis, Rivera, Lesaux, Kieffer, and Rivera, 2006). Despite this, older ELL students often are not provided with high-quality academic reading comprehension instruction, but rather with instruction focusing on low-level skills (Garcia, 2003, Valdes, 2001).

To address the need for improved ways of supporting ELL academic reading comprehension development in the middle grades, this research project takes a formative assessment approach. Based on a theoretical reading competency model (O’Reilly & Sheehan, 2009), the assessment construct and content developed in this project emphasize higher-order reading comprehension skills for middle-school ELL students. A set of academic reading comprehension assessments was developed, specifically targeting intermediate ELL students in ESL or English language development classes. The assessments also incorporated the unique feature of allowing students to collaborate with peers on certain tasks before completing items individually.

A first-year pilot study was conducted with 295 ELL students in 15 middle-school ESL classes across the U.S. Both quantitative and qualitative data were collected, including classroom observations, teacher interviews, and assessment scores. These data provide empirical evidence of ELL students’ reading comprehension difficulties, which is used to inform the refinement of the assessment model. The data was expected to provide insight into what assessment-derived information is most useful to teachers for integration into their instructional planning. The pilot study was also intended to examine the psychometric properties of the developed items and the feasibility of the use of the assessment for formative purposes in classroom settings.

Preliminary results suggest that both foundational and higher-level reading comprehension skills are important constructs to be measured for intermediate ELL students. During classroom observations, collaborative tasks appeared to create more opportunities for engaging discussion and scaffolded opportunities to acquire and exercise reading comprehension skills. It was also notable that teachers had varied perspectives on formative assessment and the application of the pilot assessment, indicating the central role of teachers in the use of assessments for formative purposes.

In this session, we will present the complete results of our analysis and share our guiding principles for the continued development of the formative reading comprehension assessment tool for ELL students. This information will be useful for those who plan to design and utilize a formative assessment for ELL students.

Authors

• Mikyung Kim Wolf, ETS

• Ian Blood, ETS