Nonconstruct Item Features and Response Processes in Computer Science Assessments: Evidence From Think-Alouds and Sequence Analysis

• In Event: Process Data Modeling and Application in Educational Assessments

Abstract

This paper proposes a framework of non-construct related item features for Computer Science items and explores whether and how they influence participant response processes. The framework draws on research in computer science education and psychometrics, and is investigated using thinkaloud interviews with university students studying computer science. Three different analyses of transcribed and coded thinkalouds were undertaken: based on the frequency of codes, differences in sequences of codes, and the most discriminating sequences. Preliminary results found that varying the item features varied response processes, specifically the proportion of planning and reading processes participants engaged in. Implications are discussed for CS item validity and design, as well as further use of sequence analysis methods to analyze process data.

Authors

• Matt Davidson, University of Washington - Seattle

• Min Li, University of Washington