Designing Scenario-Based Technology-Enhanced Assessments

• In Event: Building and Validating Cognitively Based, Technology-Enhanced English Language Arts Assessments

Abstract

A major focus of the CBAL ELA research initiative has been to design, develop, and pilot test scenario-based assessments that incorporate evidence-based learning progressions, technology-enhanced item types, and extensive teacher/student feedback to ensure that the assessments are instructionally valuable and engaging for students. The assessments are delivered online, and technology plays an essential role throughout their design, development, and administration. This poster will be designed to document the way we discipline assessment design to reflect an appropriate scenario and to test appropriate learning progressions in a technology-enhanced environment.
Conceptualizing a scenario-based assessment. A team of ELA assessment specialists begins designing each prototype assessment by targeting a cluster of integrated reading/writing skills and a narrow range of grade-appropriate proficiency levels in the CBAL ELA learning progressions and Common Core State Standards. They identify authentic contexts in which students might use these skills, taking to account audience and purpose for writing, the ways in which students might work (alone, collaboratively, or in a whole-class context), and the kinds of activities that might best prepare students to perform well on the complex culminating task. The team collaborates with classroom teachers and other educational partners to confirm that the proposed sequence of activities is engaging, realistic, and meaningful. Then the activities are transformed into a carefully structured sequence of selected-response and constructed-response items with texts and other relevant resources for the progression of targeted skills. Technology is integral to these deliberations.
Using technology to help confirm the appropriateness of selected texts. All texts used in CBAL ELA assessments are analyzed by SourceRater and other text-complexity evaluation software, although it is expert judgment informed by text-evaluation data that determines whether or not a text should be used at a particular grade, for a particular purpose, and in a particular way.
Creating technology-enhanced items. Many of the most effective CBAL test questions are possible only because of the computer. Selective-response item types include asking students to construct graphics (e.g., to outline information or show hierarchical or chronological relationships), to improve texts (e.g., by striking out redundant phrases or irrelevant ideas), and to highlight textual evidence that supports their answers. Constructed-response items benefit from computer support by presenting click-on tips, writer’s checklists, guidelines, rubrics, and other resources to support a writing activity.
Using technology applications to storyboard the assessment. Once the sequence of tasks and items are texts, they collaborate with a graphic artist to create a PowerPoint presentation illustrating the flow of activities that comprise the assessment. This format facilitates further review and refinement, often with teacher/student feedback, before the assessment is programmed for usability studies and pilot testing.
Online usability and pilot testing to validate the assessments. Each prototype assessment is administered locally to small groups of students who comment on the clarity of the directions, the items and texts, and overall navigational flow. Additional changes are programmed, if necessary, and the assessment is administered in a large-scale study to document individual item characteristics and their contribution to overall scenario results.

Authors

• Mary E. Fowles, ETS

• John P. Sabatini, ETS

• Paul Deane, Educational Testing Service