Developing Theory-Based Literacy Assessments for Pre-K-12 Students: Challenges and Opportunities

• In Event: Assessments for Problem Solving, Literacy, and Learning in Designed, Online Environments

Abstract

Goals: This paper explores the development of a series of age-appropriate, developmentally-sensitive, and theoretically-based reading comprehension assessments. We highlight the conceptual challenges with expanding the construct of reading literacy to be inclusive of a rapidly changing digital literacy environment, as well as the practical challenges and lessons learned in developing and administering web-based assessments across pre-K to 12th grade.

Theoretical framework: The assessments developed in this project are based on an assessment framework derived from a review of the theoretical and empirical literature on reading processes and instruction. The theoretical foundations of the assessments are meant to be consistent with the more general features of a number of cognitive models and theories of comprehension (e.g., Graesser, Singer & Trabasso, 1994; Kintsch, 1998; Rouet & Britt, 2011; van den Broek, Young, Tzeng, & Linderholm, 1999).

Data sources: To date, two types of assessments have been piloted in 23 states in a mix of urban, suburban and rural areas. We have tested over 45,000 students on our Global Integrated Scenario-based Assessments (GISA) assessments and over 115,000 students on component skills.

Assessments: All assessments are web-administered, taking approximately 45-50 minutes. The system incorporates two types of assessments:
Global Integrated Scenario-based Assessments (GISA) are designed to measure integrated skills associated with higher-level comprehension. Students are presented with a realistic purpose for reading that requires them to integrate, synthesize, and evaluate a collection of diverse reading materials (e.g., blog, website). Tasks and activities are sequenced and supported to model complex thinking while simultaneously collecting evidence of partial understanding for developing students. Simulated peer interactions are designed to promote the social nature of reading. Performance moderators including background knowledge, student motivation, self-regulation/metacognition, disciplinary reading, learning, and reading strategies are also measured. To date, we have piloted over 20 GISA forms.
Component skills assessments are designed to contextualize and interpret performance on the GISA. Measures include decoding, word recognition, morphology, vocabulary, and sentence processing. To date, we have piloted multiple component forms.

Results: Preliminary analyses reveal that our GISA assessments demonstrate good reliability (typically α=. 80 or higher); component assessments even higher (typically α>.90). We collected validity evidence and investigated a number of key research questions. For example, analyses are uncovering key relationships between components and comprehension, dimensionality of the measures, the role of background knowledge and motivation in testing, the ability of students to learn new information during a reading test, the added value of constructed response items, and relationships between local and global comprehension processes.

Significance: We believe a more inclusive measure of reading comprehension, such as the GISA, provides several advantages for educators: it provides a more comprehensive construct of comprehension; students evaluate, integrate and synthesize information from multiple sources to solve problems or makes decisions; and task types provide supports for performance and encourage the use of reading strategies. We believe providing the reading component measure adds value, especially for struggling readers who fall behind grade-level reading comprehension expectations. We view both assessments as a more integral part of larger reading literacy construct.

Authors

• John P. Sabatini, ETS

• Tenaha P. O'Reilly, ETS

• Laura Halderman, Educational Testing Service

• Kelly M. Bruce, ETS