Using Dynamic Time Warping and Cluster Analysis to Analyze the Learning of Computer Programming

• In Event: Roundtable Session 98
  In Roundtable Session: Using Learning Analytics and Educational Data Mining to Understand Scripted and Exploratory Learning Environments: Toward a Common Theoretical and Methodological Framework to Investigate the Trajectory to Expertise

Abstract

Educational data mining and Learning Analytics (EDM; Amershi & Conati, 2009; Baker, Corbett, Koedinger, & Wagner, 2004) has grown rapidly over the last years, profiting from the vast availability of logfile data from e-learning systems, instrumented computer applications (Blikstein, 2009, 2011, Berland, 2008), computer vision systems, and web logs. The majority of the work focuses on standardized tasks delivered by cognitive tutors or clickstreams of online courses (Baker & Yacef, 2009). However, there is increasing interest in promoting types of learning activities and outcomes that are non-standardized, engaging students in constructionist (Papert, 1980) open-ended tasks such as designing robots or programming computers. In this work, we present a machine-learning-based framework to predict the performance and level of expertise of students based on their coding style. We will show results from a study in which we capture and analyze logs of undergraduate students (n=150) learning to program in an introductory Java course. In addition to logfiles, we also had students fill a questionnaire about their programming style, motivation, and previous experience.
As students learn to write program, they develop their own distinctive style of coding (top-down vs. botton-up, “planners” vs. “tinkerers” (Turkle & Papert, 1991)). The capture of these styles and their evolution during a computer science course could point to cognitive changes as well. Therefore, we analyze code fragments written by students for various assignments, based on fundamental concepts such as recursion, functional programming, object-oriented programming, error-handling, looping, etc. The code fragments committed to a global repository, giving us access to the each incremental time-stamped step in the process. First, we calculate differences between each successive commits (or “diff” statistics) by a student, in terms of the number of lines and characters added/deleted/reordered/modified, control-flow blocks added/deleted, and comments added. This gives us an approximate idea of the increments made by the student in each successive step and the time interval between steps. The individual code fragments are separately compiled and executed to capture their runtime errors and compiler errors. This results in 33 time-series of various diff statistics at each commit step, along-with the compiler or run-time errors. Using Dynamic Time Warping to calculate the distance between the time-series, we hierarchically cluster the time-series data of students, and use silhouette values to decide the cutoff for the number of clusters. We then analyze the clusters for certain underlying “coding trends” which all students in that cluster shared. Each student progresses with an assignment to a different extent in his/her own time and this changes his/her programming style over the course of time (due to errors, output, time constraints, etc.). The idea here is to first identify the features influencing the coding style of a student, and then use them is features here to cluster students. Finally, we cross those results with self-reported questionnaires on style, motivation, and experience. Preliminary results show distinctive changes in the style and learning pattern for self-declared “top-down” and “bottom-up” programmers, as well as connections between the evolution of coding styles and expertise.

Authors

• Paulo Blikstein, Stanford University

• Mustafa Safdari, Stanford University

• Marcelo Worsley, Stanford University