Strengthening NAEP Interactive Computer Tasks Through Increased Student Engagement

• In Event: The Impact of Item, Test, and Test-Taker Characteristics on Engagement During NAEP Assessments

Abstract

The National Assessment of Educational Progress (NAEP) is an assessment of academic competencies regularly administered to nationally representative samples of students in grades 4, 8, and 12. NAEP’s primary purpose is to measure student achievement in different subject areas and to collect contextual data for the reporting and interpretation of assessment results. However, as students do not receive a NAEP score, the assessment is considered “low stakes” for them. There are long-standing questions about their level of engagement and effort. For assessment results to be valid, students should be sufficiently engaged with the assessment tasks to demonstrate their knowledge and skills.

In this study, we investigated the effects of engagement-enhanced features (EEFs) on the performance of 8th-graders on digitally based assessments. The goal of implementing EEFs is to help reduce the likelihood that assessment scores underestimate students’ knowledge and skills. During the first stage of the study, the research team identified low-engagement interactive computer tasks (ICTs) from tasks developed for the 2015 8th-grade NAEP science assessment but dropped after the pilot test due to item performance issues. The identification of low-engagement items focused on events observed in the process data, including missing responses, correctness of responses, and time spent on each item and the entire task. One task was selected as the candidate for adding EEFs.

In the second stage, EEFs used to augment this task focus on three primary design features: the social contextualization of the task scenario, the reduction of cognitive load, and the inclusion of metacognition opportunities. Items were augmented with more than one feature if needed. The team interviewed three groups of students through virtual cognitive labs. Two groups of students experienced both the original task and the version augmented with EEFs, in reverse order, and the third (control) group experienced only the augmented task. The team evaluated the impact of the engagement-enhancing features on student performance and preferences, including a focus on high- and low-performing students, across the three groups when possible.

Our findings indicate that reducing cognitive load by changing item types, removing dependencies, breaking grouped items into separate items, and modifying the mode of data generation had a positive impact on performance as well as student preference in terms of making the task seem more engaging and achievable. We did not see a direct impact of social contextualization enhancements and metacognition-based enhancements on performance; however, we did see a positive impact of social contextualization on students’ preferences in terms of making the task seem more engaging (fun), collaborative, realistic, and achievable (easier) and a positive impact of metacognition-based enhancements on students’ preferences in terms of improved understanding and perceptions of the task as more engaging and more achievable (easier). Overall, we found that the EEFs improved performance, with some indications that this effect might be greater for lower performing students than for higher performing students. These findings provide insight into the design and development of future interactive or scenario-based tasks that can be applied across a range of domains and assessments.

Authors

• Kristin Stoeffler, ACT, Inc.

• Yan Wang, American Institutes for Research

• Emmanuel Sikali, U.S. Department of Education