Using Process Data (Eye Tracking and Keystrokes) to Validate Scenario-Based Summative Assessments

• In Event: Building and Validating Cognitively Based, Technology-Enhanced English Language Arts Assessments

Abstract

1. Purposes. More and more educational assessments today use scenario-based tasks to elicit knowledge, skills, and abilities (SKAs) similar to those necessary in real-world applications. Nonetheless, scenario-based assessments are also more complex and may introduce construct-irrelevant variance. Here we introduce a method of using eye-tracking and other process data to validate new scenario-based educational assessments and illustrate its applications in English language arts and mathematics assessments.
2. Theoretical framework. CBAL (Cognitive Based Assessment of, for, and as Learning; Bennett, 2010; Bennett & Gitomer, 2009) is a model of innovative educational assessments that document student achievement (of learning), facilitate instruction (for learning), and are a worthwhile learning experience in and of themselves (as learning). CBAL assessments not only measures students’ achievement of curriculum standards, but also provide information about student cognitive processes. Because of this emphasis on how students learn and problem-solve, part of the validity argument for CBAL needs to address the cognitive processes involved in solving assessment tasks and how they relate to the constructs intended to be assessed. Measuring student response processes becomes an essential aspect of the validation process.
3. Methods. We use eye-tracking and keystroke logging methods to obtain rich information about student response processes. Students conducted CBAL scenario-based assessment on the computer, while their eyes were tracked with an EyeLink eye-tracking system and their keystrokes and mouse movements were also recorded. In addition, we also conducted cued retrospective verbal protocols in some of the studies, in which students’ self reports further substantiated results from the process data analyses.
4. Data sources, evidence, objects, or materials. We report data from two empirical studies. Study 1 is an eye-tracking study of a middle school math assessment. Study 2 uses a combination of eye-tracking and keystroke logging on a reading and writing assessment. Phase one of the data collection has completed, with sample sizes of 30 and 15 in the two studies, respectively. Additional data are being collected.
5. Results. Available data show evidence that the assessment tasks engaged cognitive processes consistent with the design of the assessments. In the math assessment, for example, students who answered an item correctly tend to show eye movement patterns consistent with the logical requirement of the solution. In contrast, students who were mistaken showed difficulties in encoding information and/or making decisions. Our initial analysis of the ELA task suggests different levels of editing behaviors that may reflect students’ metacognition and engagement with the writing task.
6. Scientific or scholarly significance of the study or work
Validating the new generation of innovative educational assessments is challenging because they often involve complex assessment design, as illustrated by scenario-based tasks. In addition, some assessments such as CBAL tests also make claims about student cognitive processes, which require empirical validation using process data. The current program of research aims to develop a systematic approach to validating such assessments. The methods developed here, as well as some of the lessons learned, help to improve the design of the next generation educational assessments.

Authors

• Anita M. Sands, Educational Testing Service

• Gary Feng, Educational Testing Service

• Paul Deane, Educational Testing Service