The Online Research and Comprehension Assessment Project: Designing Technology-Based Assessments for Online Research, Comprehension, and Communication

• In Event: Assessments for Problem Solving, Literacy, and Learning in Designed, Online Environments

Abstract

Goals: This paper presents technology-based assessments and results from the ORCA (Online Research and Comprehension Assessment) Project. The project has developed valid, reliable, and practical performance-based assessments of students’ ability to conduct online research and write a short report of their results, skills central to the Common Core State Standards Initiative (2010) and an online age of learning (Wiley, Goldman, Graesser, Sanchez, Ash, Hemmerich, 2009).

Theoretical framework: Research into online research is informed by broadly conceived theoretical work in New Literacies (Gee, 2007; Lankshear & Knobel, 2006). Within this broader context, a more specific theory of online research and comprehension has also emerged (Leu, Kinzer, Coiro, Castek, & Henry, 2013). During online research and comprehension, both traditional offline reading skills and new online reading skills are required, often in complex and interrelated ways. These include additional skills in: locating, evaluating, synthesizing, and communicating.

Data sources: The ORCA project took place with over 100 schools in two states, over five years. The data in this paper come from seventh-grade students in two northeastern states, one where every student in seventh grade had a laptop computer and one where students did not. Representative state samples (n = 1,584), completed two out of 16, randomly assigned, Internet research tasks in science about human body systems in one of two formats: Closed Internet and Multiple Choice Internet. The ORCA-Closed format assessed performance within the confines of a simulation of the Internet. A student avatar presented the initial problem and guided each student through the research process within a social network containing text messaging, a notebook, email, and a wiki. The ORCA-Multiple Choice format paired screenshots of key decision points for each skill with prompts and multiple-choice items. Classical test theory analysis, Item Response Theory (IRT), exploratory and confirmatory factor analyses were used to analyze the psychometric properties for each assessment.

Results: The project has developed initial, performance-based assessments of students’ ability to conduct online research. These assessments will be presented.
In addition to demonstrating good levels of reliability and validity for each of the assessments and formats, the following preliminary results were obtained and will be presented:
1. While both formats demonstrated good levels of reliability, generally higher levels of reliability and validity were demonstrated by the ORCA-Closed format.
2. When prior knowledge and SES were controlled, the online research performance of students in the laptop state was greater than students in the non-laptop state, but the pattern was not consistent across all items.
3. Students in both states performed least well with the critical evaluation and communication of information, demonstrating correct responses for only about 30% of the items in these areas.

Significance: The information from this session may be used to inform the scant research available about how best to effectively measure the online research and reading abilities required to function successfully in a globalized, information society (Educational Testing Service, 2003; Quellmalz & Haertel, 2008).

Authors

• Donald J. Leu, University of Connecticut

• Jonna M. Kulikowich, The Pennsylvania State University

• Nell Sedransk, National Institute of Statistical Sciences

• Julie Coiro, University of Rhode Island

• Elena Forzani, University of Connecticut

• Clint Kennedy, University of Connecticut

• Cheryl Maykel, University of Connecticut