Paper Summary
Share...

Direct link:

NAEP Survey Assessment Innovations Laboratory Virtual World for Assessment of English Language Arts Inquiry

Fri, April 13, 4:05 to 6:05pm, New York Marriott Marquis, Floor: Fourth Floor, Odets

Abstract

The National Assessment of Educational Progress (NAEP) Survey Assessment Innovations Lab (SAIL) initiative aims to advance state-of-the-art large-scale survey assessments by leveraging cognitive science research and technological developments to explore new approaches to what constructs we measure and how we measure them. One construct on which NAEP has not previously reported is the ability to conduct research and inquiry from multiple sources in digital, online environments (Author, 2015). Theoretical and empirical work in the cognitive and learning sciences underscores the importance of inquiry and multiple-document reading comprehension skills as central to success across many academic domains, including science, history, and literature (Britt & Rouet, 2012; Goldman, Lawless, & Manning, 2013). Specifically, students need to develop skills in defining information needs; locating sources; evaluating sources; processing, analyzing, and synthesizing sources; and clearly communicating the results of one’s research to an audience, using relevant citations and evidence to support one’s conclusions.

The Virtual World for ELA project aims to develop and research a flexible “virtual world” platform that provides evidence of eighth-grade students’ information gathering, processing, and evaluation skills in the context of English language arts (ELA) inquiry tasks. We developed a prototype virtual world platform consisting of a social context and suite of digital inquiry tools and associated specifications for data capture. Within the platform, we developed two extended scenario-based tasks that engage individual students in a complete research project, requiring them to locate, evaluate, read, and to synthesize information from multiple sources by incorporating evidence from those sources in a final written product addressing an overall inquiry question. Each task is organized into three phases: Setup (scenario and task introduction), Free Roam (exploring, gathering and evaluating available resources), and Conclusion (apply information from collected sources to construct an overall response). As students interact with the tools and available information resources, logfiles capture evidence of their behaviors, including specific sequences of actions, keystrokes, and item responses.

An iterative research and development approach was used to ensure that tasks were usable, comprehensible, and engaging to students, including several rounds of small-group and one-on-one playtesting (n=30) and individual cognitive laboratories (n=15). These early empirical efforts yielded two main conclusions. First, students’ performance on a culminating task was related to their engagement (or not) in various actions during the inquiry process, as revealed by logfiles. Second, balancing students’ self-regulated exploration of the virtual world with provision of constraints and supports to help them remain on track is critical for capturing students’ unfolding inquiry processes. Complex research activities require extended time to complete, and students with limited prior experience with inquiry skills, or who took longer to read and evaluate sources, failed to complete the task. This finding resulted in task design revisions to address this issue (e.g., time-out warnings). Results from a small-scale tryout study to be conducted in Fall 2017, where 100+ students complete the inquiry task under normal, uninterrupted testing conditions, will also be presented. Implications for developing engaging assessments of online inquiry will be discussed.

Authors