Evaluation of Code Manipulation in Coding Games

• In Event: Game-Based Measures of Learning Processes: Challenges, Methods, and Solutions

Abstract

Block-based coding games can provide information on how players construct and manipulate their code before executing it. Although this code-manipulation telemetry can be used to provide valuable real-time feedback to players and to make inferences about their thinking processes while coding, it is seldomly evaluated.

To evaluate code manipulation, we collected gameplay telemetry from 24 third-grade students playing the puzzle levels of the block-coding game codeSpark Academy. This game covers sequences, parameters, loops, events, and conditionals. The player “codes” by adding a block (or command), changing its parameter (e.g., specifying the count for a loop block), reordering blocks, or removing a block. Optimal solution(s) were defined for each puzzle level.

The code manipulation sequences and the optimal solutions were encoded in a way to allow use of Damerau–Levenshtein edit distance. The edit distance accounted for command insertions, deletions, substitutions, and transpositions. This edit distance was used to compare players’ programs, step-by-step, to their associated optimal solution state. Thus, each code update was “productive” if the distance to the optimal solution decreased, otherwise “unproductive.”

Plotting code manipulation quality as a function of time and visual inspection of the plots suggest patterns associated with debugging, iteration, and problem decomposition. The patterns correspond with our qualitative analysis of students’ moment-to-moment programming. Our next steps are to develop algorithms to automatically detect instances of debugging, iteration, and problem decomposition and evaluate how well the algorithms (a) detect and miss researcher-identified instances of these programming processes, and (b) detect new instances not identified by researchers. We will evaluate the quality of the algorithm using the signal detection framework, which has been widely used in psychosocial research to describe the accuracy of a diagnostic instrument, classification, and decision making (Chung et al., 2018; Lynn & Barrett, 2014). These results will be reported at AERA.

Authors

• Markus Iseli, University of California - Los Angeles

• Tianying Feng, National Center for Research on Evaluation, Standards, and Student Testing

• Gregory K.W.K. Chung, University of California - Los Angeles

• Emily Relkin, Tufts University