Measuring the Learning Quality of Open Educational Resources in Mathematics


• In Event: 63.059 - Emerging Perspectives on Understanding Learning Behaviors in Digital Environments

Abstract

Objectives
A number of tools exist for evaluating the diagnostic quality of items authored for a test (e.g., van der Linden & Hambleton, 1997). Online learning resources such as Kahn Academy and ASSISTments produce thousands of items with hints and scaffolding added. In these systems, the learning quality of the items is at least as important a concern as their diagnostic quality. The objective of this work is to demonstrate how the learning quality of items can be measured in these scenarios to enable an iterative cycle of improvement of the pedagogical dimension of these open educational resources.

Theoretical Framework
Learning in this work is described in the context of mastery learning where students practice answering items of a shared construct, receiving feedback and help along the way, and stop practicing once a threshold level of proficiency is demonstrated. The practice stopping criterion for both Kahn Academy and ASSISTments is N correct in a row; however, both systems have additional tasks requiring successful completion in order for the student to be declared as mastered. The assumption of learning employed in this analysis is that of atomic skill acquisition (Anderson, 1993) where students learn a particular skill through practice, hints, and correctness feedback from items of the same skill. While Anderson’s approach uses very fine-grained skill definitions derived from cognitive task analysis, more relaxed, coarser definitions are used in the compositions of items in exercises in Kahn Academy and ASSISTments.

Methods and Data Sources
Data were collected from Kahn Academy from a two week period in the Fall of 2013. The exercise with the most data points, “comparing with multiplication”, was used for the analysis which consisted of 43,000 participants. A Bayesian Networks model from previous work on item effect modeling (Pardos & Heffernan, 2009) was employed using computational scaling techniques from work in preparation (Pardos & Johnson, in prep). A 10-fold cross-validation was used to investigate if learning quality parameters from one set of students generalized to another.

Results and Conclusions
In our preliminary analysis of the data, statistically significantly higher learning rates were measured among particular item templates in the exercise. Modeling separate learning quality parameters per template resulted in a higher cross-validation accuracy than modeling only a single average learning quality parameter. This suggests that a signal related to learning can be measured from the items. A study to examine if higher learning quality items lead to higher rates of success on future transfer tests is the next step in this research.

Significance
This work is significant because generalization of item learning parameters has not previous been observed in these or other digital learning environments. The aim of this research is to enable iterative improvement of pedagogy in digital learning environments by filling in the missing psychometric capability of measuring learning quality. This technique of learning quality inference from high volume data can also be applied to passively consumed open educational resources that are interleaved with assessments, such as in open online courseware.

Author

• Zachary Pardos