Structured Scenario-Based Assessments: Explicit Task Sequences to Identify Barriers to Effective Strategy Use

• In Event: 29.035 - Structured Scenario-Based Assessments as Tools to Support Best Practices in Reading/Writing Instruction

Abstract

Objectives: To develop assessments that capture critical steps in key literacy practices such as research and argumentation, emphasizing the need to integrate reading, writing and reasoning skills. In the symposium, we will highlight ways in which SSBAs can encapsulate appropriate strategy use while measuring key component skills. We will examine theoretical and methodological challenges to this approach, present potential instructional uses, and analyze results from multiple administrations that highlight research and argumentation skills that need to be developed as students transition from traditional literacy instruction to the Common Core.
Theoretical framework: The assessments developed in this project draw on a literacy framework derived from a review of the cognitive and learning sciences literature. It emphasizes interconnections between reading, writing, and thinking, and identifies integrated practices that are commonly targeted for instruction in K-12 education. The framework is designed to be consistent with cognitive analyses of writing skill (McCutchen, 2006), best practices in writing pedagogy (Graham & Perin, 2007) and similar results in reading (Kintsch, 1998; Goldman, 2004; Vellutino et al., 2007)
Methods: Students completed online assessments during regularly scheduled class periods, typically in two consecutive class periods to sample both reading and writing skills.
Data sources: Between 2009 and 2013, ten middle school reading and writing assessments were administered in more than 30 states, in a mix of urban, rural and suburban schools. Three major studies were conducted, in 2009 (>2600 students), 2011 (>3500 students), and 2013 (>3800 students), along with smaller pilot studies and field tests. In addition, we have piloted formative use of CBAL assessments in schools, with accompanying professional development, and have surveyed the teachers involved in these studies to examine teacher responses to assessment results and content coverage.
Assessments: The assessments focused on several key practices (argumentation, research, informational reading/writing, and literary analysis). Each assessment follows a task sequence that reflects effective reading and writing strategies, leading up to an appropriate performance task. Items are grouped to simulate realistic tasks, but designed to provide measurement of the ability to accomplish specific substeps necessary to accomplish the full performance task.
Results: Analysis indicates that CBAL assessments are generally reliable (typically α=.75 or higher). Dimensionality analysis indicates that scenario effects are relatively small. The relation between preparatory reading and writing steps and the final writing task is moderate to strong. Over the three studies, middle school students performed relatively poorly on a range of critical sub-skills necessary for research and argumentation, including framing research questions, evaluating sources, summarizing source content, and evaluating arguments.
Significance: We believe that there is significant educational benefit to creating SSBAs that model key practices. Patterns of student performance suggest specific targets for educational intervention directly tied to effective reading and writing strategies. Teacher response indicates that SSBAs may provide significant support to teachers as they work to prepare students for independent performance on complex literacy tasks.

Authors

• Paul Deane, Educational Testing Service

• Yi Song, Educational Testing Service

• Jesse R. Sparks, Educational Testing Service