Evaluating Item Quality in Large-Scale English Language Arts Assessments

• In Event: 16.023 - Developing Design Principles for Improving State Assessment Items to Support More Valid and Equitable Measurement

Abstract

Purpose & Significance
The 1996 NCTE/IRA Standards for the English Language Arts are widely accepted and have repeatedly been affirmed as representing central values of the discipline. The standards emphasize the importance of students encountering cognitively complex texts, applying “knowledge of language structure, language conventions . . . media techniques, figurative language, and genre to create, critique, and discuss print and non-print texts”. Students should “conduct research…generating ideas and questions, and by posing problems. They gather, evaluate, and synthesize data from a variety of sources...to communicate their discoveries” And students should “read a wide range of literature from many periods in many genres.”
Despite widespread agreement on these core values of the discipline, many large-scale English and literacy assessments, textbooks, and curricula have failed to provide opportunities for students to demonstrate the complex skills and dispositions that reflect these values. Instead, assessments have tended to focus on low-level reading comprehension, speculative rather than evidence-based interpretation of text, and identification of literary elements in decontextualized passages of text (Applebee & Langer, 2013; Applebee, 1993; Lynch & Evans, 1963; Mihalakis & Petrosky, 2015).
The goal of our assessment item review was to select items of varied cognitive complexity that measure authentic disciplinary ways of thinking, reading, and writing, with varied item formats and content foci, and to analyze these items to generate a set of design principles for developing high quality ELA assessment items that more validly measure key disciplinary understandings and skills. These design principles contribute to the knowledge base for assessment design, and have the potential to strengthen the design of state assessments of ELA and to drive more authentic teaching and learning that benefits diverse learners.
Theoretical Framework
We applied Hess’s 2013 Matrices for Close Reading Across Content Areas and for Written and Oral Communication and Hillocks and Ludlow’s 1984 Taxonomy of Skills in Reading and Interpreting Fiction to evaluate the cognitive complexity of items. We also generated our own set of analytic questions, informed by the NCTE/IRA standards, the PISA Reading Literacy Framework, and the Key Shifts in ELA from the Common Core State Standards, to evaluate each item’s potential to measure understandings, skills, and modes of thinking that are authentic and significant to the discipline.
Methods & Sources
To find high quality items among those publicly available from large-scale ELA assessments, we reviewed a wide range of state and consortium assessments from our organization’s 50 State Assessment Collection, as well as national and international assessments (e.g., Advanced Placement, PISA). The six items we selected for analysis represent three content foci: reading informational text, reading literary text, and writing. Two ELA assessment specialists applied the frameworks described above to the evaluation of item quality.
Results
We identified four core features of high-quality ELA assessment items, which we explain and illustrate through our item analyses:
• Items that are text-dependent and measure more than literal comprehension
• Items that measure cognitive and meta-cognitive competencies for reading literacy
• Items that require evidence-based writing
• Items that call for authentic disciplinary uses of literacy.

Author

• Nicole Renner, Stanford University