Extracting Debugging Indicators Based on Distance to Solution in a Block-Based Programming Game

• In Event: Game-Based Indicators of Learning Processes: Extraction Methods, Validity Evidence, and Applications

Abstract

Objective
To develop game-based indicators based on distance to solution and examine students’ debugging processes in a block-based programming environment.

Background
Block-based programming involves changing at least two dimensions--content and structure distance to solution--of the block commands. One advantage of considering both dimensions is better differentiation of students who reach the same solution but differ in their approaches. Decomposing programming into these two components enables us to identify where difficulties and bugs arise, and track students’ debugging processes--valuable diagnostic insights.

Methods
We used Johnson et al.’s (1983) bug classification scheme to identify four types of coding bugs—Missing, Redundant, Misplaced (wrong, weak, or strong form), and Malformed (weak or strong form)—based on the content and structure distance. The content distance describes how closely individual blocks in the student’s codes match those in the solution. The structure distance describes how different structurally (e.g., order and form) the student’s code sequence is from the solution sequence. We then developed indicators of students’ debugging processes, capturing trends of numeric changes, forms of Malformed or Misplaced, and students’ code states. Lastly, we conducted correlational analyses to examine the relationships between the indicators and students’ test of computational thinking and other common game-based outcomes (e.g., mean number of solution attempts).

Data Sources
Twenty-five students (10 first- and 15 third-graders) played a block-based programming game codeSpark Academy remotely for one day a week for six weeks. The students worked on in-game puzzles under the supervision of two teachers during a 45-minute Zoom session. A pretest and a posttest were administered at the start and end of the study. Both tests used items from TechCheck, a 15-item multiple choice assessment targeting computational thinking (Relkin et al., 2020). Fine-grained telemetry data were collected to capture students’ moment-to-moment gameplay.

Results
Spearman's nonparametric correlation procedure was used for analyses. Results showed the number of non-increasing trends for Missing was positively correlated with the pre- to posttest change scores (⍴ = .48, p < .05). No significant correlations were found between the non-increasing trends for Redundant and Misplaced, and test scores. The number of non-increasing trends for Missing and Redundant were also negatively correlated with mean number of attempts (⍴ = -.64 and -.42, p < .05). Such relationships were expected given that non-increasing trends reflected students’ code content that was not diverging much from the solution. When we examined changes in debugging processes, we found the number of times students transitioned from having poorly structured codes to reaching the solution in the next submission was positively related to their posttest (⍴ = .45, p < .05).

Significance
This work demonstrates the utility of first determining general indicators of learning in the task from which secondary indicators are derived. The decomposition of students’ coding into content and structure indicators enabled us to derive indicators of different coding bugs. This work also demonstrates the use of theory to guide algorithm development (vs. unsupervised machine learning approaches). The benefits of theory-based algorithm development are traceability, interpretability, and more generalizable (e.g., to other block-based programming languages--Scratch).

Authors

• Tianying Feng, National Center for Research on Evaluation, Standards, and Student Testing

• Gregory K.W.K. Chung, University of California - Los Angeles