Paper Summary
Share...

Direct link:

Measuring 21st-Century Reading Comprehension: How to Measure Higher-Level Skills While Providing Information Useful for Instruction

Fri, April 17, 10:35am to 12:05pm, Sheraton, Floor: Second Level, Huron

Abstract

Objectives: To develop a series of age-appropriate, developmentally-sensitive, and theoretically-based reading comprehension assessments. In the symposium, we will highlight conceptual challenges of measuring higher-level reading comprehension, potential benefits for instruction, as well as practical challenges and lessons learned in developing and administering web-based assessments across pre-K to 12th grade.
Theoretical framework: The assessments developed in this project are based on an assessment framework derived from a review of the theoretical and empirical literature on reading processes and instruction. The theoretical foundations of the assessments are meant to be consistent with the more general features of a number of cognitive models and theories of comprehension (e.g., Graesser, Singer & Trabasso, 1994; Kintsch, 1998; Rouet & Britt, 2011; van den Broek, Young, Tzeng, & Linderholm, 1999).
Methods and Data Sources: students completed the electronically-delivered assessments in their regular 45-55 minute classroom period. To date, two types of assessments have been piloted in 23 states in a mix of urban, suburban and rural areas. We have tested over 90,000 students on our GISA assessments and over 120,000 students on component skills.
Assessments: The system incorporates two types of assessments. Global Integrated Scenario-based Assessments (GISA) are designed to measure a set of integrated skills associated with higher-level comprehension. Students are presented with a realistic purpose for reading that requires them to integrate, synthesize, and evaluate a collection of diverse reading materials (e.g., blog, website). Tasks and activities are sequenced and supported to model complex thinking while simultaneously collecting evidence of partial understanding for developing students. Simulated peer interactions are designed to promote the social nature of reading. Performance moderators including background knowledge, student motivation, self-regulation/metacognition, disciplinary reading, learning, and reading strategies are also measured. To date, we have piloted over 19 GISA forms. Component skills assessments are designed to contextualize and interpret performance on the GISA. Measures include decoding, word recognition, morphology, vocabulary, and sentence processing. To date, we have piloted multiple component forms.
Results: Preliminary analyses reveal that our GISA assessments are demonstrating good reliability (typically α=.80 or higher); component assessments are even higher (typically α>.90). We collected validity evidence and investigated a number of key research questions. For example, analyses are uncovering key relationships between components and comprehension, dimensionality of the measures, the role of background knowledge and motivation in testing, the ability of students to learn new information during a reading test, the added value of constructed response items and potential ways the results can be useful for instruction.
Significance: We believe a more inclusive measure of reading comprehension, such as the GISA, provides several advantages for educators: it provides a more comprehensive construct of comprehension; students evaluate, integrate and synthesize information from multiple sources to solve problems or makes decisions; and task types provide supports for performance and encourage the use of reading strategies. We believe providing the reading component measure adds value, especially for struggling readers who fall behind grade level reading comprehension expectations. We view both assessments as a more integral part of the larger reading literacy construct.

Authors