Machine Scoring by Design for New Digitally Based Item Types and Innovative Observations in NAEP

• In Event: 79.017 - Technology-Enhanced Items in Educational Assessments: Challenges, Findings, and Implications

Abstract

Technology-enhanced questions and tasks in STEM assessment (science, technology, engineering, mathematics) have taken on many new formats in the last several years. From a content analysis perspective, such formats can potentially provide better avenues for collecting evidence on what students know and can do in authentic STEM activities. However the psychometric performance of many new formats is not well understood, especially at scale and in regard to promising practices for automated scoring rules in new types of observations. In this session we will share investigations from the U.S. National Assessment of Educational Progress (NAEP) Science Pilot 2018 Assessment. The new technology-enhanced (TE) questions and tasks in NAEP can include zone, matrix, and other drag-and-drop formats, as well as a variety of other intermediate-constraint innovations that fall between fully constructed and fully selected item types. For the NAEP investigations, probabilities of response combinations that students can provide by random guessing were identified for each new format type in the science assessment. Permutations of decision rules were explored. This resulted in the identification of three factors in determining automated scoring approaches: (1) statistical probability information for the specific TE item type, (2) historical performance data if the item was previously assessed and the scoring guide did not change, and (3) content considerations. This paper will share some of the large-scale pilot investigations of the new machine scoring approaches.

Author

• Kathleen Scalise, ETS