Leveraging Educational Data Mining and Human Judgment to Understand Moment-by-Moment Patterns in Student Learning

• In Event: Roundtable Session 98
  In Roundtable Session: Using Learning Analytics and Educational Data Mining to Understand Scripted and Exploratory Learning Environments: Toward a Common Theoretical and Methodological Framework to Investigate the Trajectory to Expertise

Abstract

Recent advances in student modeling, driven by educational data mining methods (cf. Baker & Yacef, 2009; Romero & Ventura, 2010), have led to the creation of methods that estimate a student’s learning at a moment-by-moment level (Baker, Goldstein, & Heffernan, 2010, in press), while the student works within educational software. The outputs of these methods can be distilled into a visual representation, termed the moment-by-moment learning curve, showing an individual student’s learning on a specific skill over time.
In some cases, the curve can be quite flat (Figure 1 Left), indicating steady learning/steady improvement in performance, whereas in other cases the curve has a spike (Figure 1 Right), indicating a sharp improvement in performance which can be thought of as a “eureka” moment during learning (cf. Lindstrom & Gulz, 2008).

[fugure will be present in the submission document]
Figure 1. Examples of the moment-by-moment learning curve

It has been found that the degree of spikiness of the moment-by-moment learning curve is a significant predictor of later student performance on a paper test of preparation for future learning (Baker, Gowda, & Corbett, 2011), a construct discussed in (Bransford & Schwartz, 1999). Earlier work also found that spikiness predicted final knowledge in the tutor software (Baker, Goldstein, & Heffernan, 2010, in press). In this work, spikiness was defined simply as the ratio between the maximum moment-by-moment learning and the average moment-by-moment learning.
However, it has been observed that graphs can be spiky in multiple fashions; for instance, a graph can have a single spike (indicating a sudden jump in knowledge), double spikes (potentially indicating acquiring a partially-correct skill before acquiring a fully correct skill), or a plateau (indicating a period of steady improvement in knowledge).
In this talk, we examine the functional form of the moment-by-moment learning curve, and its implications for robust learning. We have developed software that visualizes a student's moment-by-moment learning curve on a specific skill. Two coders coded these curves with reference to seven potential functional forms; first, they conducted an inter-rater reliability check on 100 curves (each curve representing a student’s complete performance on each opportunity to learn a specific skill), resulting in inter-rater reliability – Cohen’s Kappa – of 0.71. Afterwards, they labeled a stratified sample of 554 curves from 72 undergraduates using a Cognitive Tutor for Genetics (cf. Corbett et al., 2010).
Each student’s prevalence of each functional form across skills was computed, and correlated to post-tests of problem-solving skill, retention of knowledge over time, transfer, and preparation for future learning. False discovery rate corrections were used to control for the possibility of spurious results due to multiple comparisons. Overall, it was found that plateaus and flat curves in learning were associated with poorer performance on a range of post-test measures, while functional forms indicating rapid learning at the beginning of practice were associated with better performance on a range of post-test measures.

Authors

• Ryan S. Baker, Worcester Polytechnic Institute

• Adam B. Goldstein, Worcester Polytechnic Institute

• Lisa M. Rossi, Worcester Polytechnic Institute

• Albert T. Corbett, Carnegie Mellon University