Search
Program Calendar
Browse By Day
Browse By Time
Browse By Person
Browse By Room
Browse By Unit
Browse By Session Type
Search Tips
Annual Meeting Housing and Travel
Personal Schedule
Sign In
X (Twitter)
In Event: Developing Innovative Assessments of 21st-Century Skills: Online Inquiry and Collaboration
The National Assessment of Educational Progress (NAEP) Survey Assessment Innovations Lab (SAIL) initiative aims to advance state-of-the-art large-scale survey assessments by leveraging cognitive science research and technological developments to explore new approaches to what constructs we measure and how we measure them. One construct on which NAEP has not previously reported is the ability to conduct research and inquiry from multiple sources in digital, online environments (Author, 2015). Theoretical and empirical work in the cognitive and learning sciences underscores the importance of inquiry and multiple-document reading comprehension skills as central to success across many academic domains, including science, history, and literature (Britt & Rouet, 2012; Goldman, Lawless, & Manning, 2013). Specifically, students need to develop skills in defining information needs; locating sources; evaluating sources; processing, analyzing, and synthesizing sources; and clearly communicating the results of one’s research to an audience, using relevant citations and evidence to support one’s conclusions.
The Virtual World for ELA project aims to develop and research a flexible “virtual world” platform that provides evidence of eighth-grade students’ information gathering, processing, and evaluation skills in the context of English language arts (ELA) inquiry tasks. We developed a prototype virtual world platform consisting of a social context and suite of digital inquiry tools and associated specifications for data capture. Within the platform, we developed two extended scenario-based tasks that engage individual students in a complete research project, requiring them to locate, evaluate, read, and to synthesize information from multiple sources by incorporating evidence from those sources in a final written product addressing an overall inquiry question. Each task is organized into three phases: Setup (scenario and task introduction), Free Roam (exploring, gathering and evaluating available resources), and Conclusion (apply information from collected sources to construct an overall response). As students interact with the tools and available information resources, logfiles capture evidence of their behaviors, including specific sequences of actions, keystrokes, and item responses.
An iterative research and development approach was used to ensure that tasks were usable, comprehensible, and engaging to students, including several rounds of small-group and one-on-one playtesting (n=30) and individual cognitive laboratories (n=15). These early empirical efforts yielded two main conclusions. First, students’ performance on a culminating task was related to their engagement (or not) in various actions during the inquiry process, as revealed by logfiles. Second, balancing students’ self-regulated exploration of the virtual world with provision of constraints and supports to help them remain on track is critical for capturing students’ unfolding inquiry processes. Complex research activities require extended time to complete, and students with limited prior experience with inquiry skills, or who took longer to read and evaluate sources, failed to complete the task. This finding resulted in task design revisions to address this issue (e.g., time-out warnings). Results from a small-scale tryout study to be conducted in Fall 2017, where 100+ students complete the inquiry task under normal, uninterrupted testing conditions, will also be presented. Implications for developing engaging assessments of online inquiry will be discussed.
Jesse R. Sparks, Educational Testing Service
Colleen Appel, Educational Testing Service
Jie Gao, Educational Testing Service
Mengxiao Zhu, ETS