Using Action Sequences as Evidence of Proficiency

• In Event: 63.059 - Emerging Perspectives on Understanding Learning Behaviors in Digital Environments

Abstract

Purpose
Data from games provide information about the process a player used to arrive at a final product, providing great potential for generating new insights regarding student actions as they relate to complex knowledge, skills, and attributes (Mislevy et al., 2014). However, the potential of games as assessment tools can be met only if replicable methods for aligning game play with learning objectives can be developed. This paper demonstrates two methods by which information about player action sequences can be used to inform estimates of player proficiency.

Perspective
Understanding learning is fundamentally about designing situations which elicit evidence about aspects of what learners know and can do. Evidence-Centered Design (ECD; Mislevy, Steinberg, & Almond, 2003) provides a framework for specifying these arguments. It defines a student model (what we want to know about the learner), the task model (what activities the learner will undertake), and the evidence model (how we will identify evidence in the learners’ work product (scoring model) and use measurement models to link it to the elements in the student model (measurement model)). This paper will focus largely on the scoring model, or the identification of the important elements in the record of player actions to extract and pass to measurement models.

Methods
This study used two methods of analysis to extract sequence information: n-gram frequencies and network analysis. N-gram frequencies were developed by counting the number of occurrences of two-, three, or more (n) action sequences found in the log file. Next, network diagrams were developed (see Figure 1), with each action as a node in the network, and metrics of network density, node centrality (how much other nodes connect through that node), and node degree (total number of connections to that node) were computed. These were then evaluated as potential pieces of evidence for inclusion in a Bayesian Network that estimates the probabilities of student proficiency.

Data Sources
Data for this work consisted of log files from SimCityEDU, which offers players various challenges to solve problems facing a city, requiring them to balance elements of environmental impact, infrastructure needs, and employment. The game scenarios are designed to assess and teach systems thinking.

Results
Analysis of the frequencies of trigrams of key actions revealed that trigram frequencies occurring at the beginning of an attempt in particular improved estimates of proficiency. Evaluation of the network analysis metrics revealed that centrality metrics of a two actions also improved prediction of proficiency.

Significance of the Work
In order to understand learning in digital environments, we must be able to measure constructs of interest. The explosion of availability of data allows us to use interactions in the environment, rather than external measures, as evidence for inferences. This paper offers two ways to consider sequences of actions in the environment as part of this evidence.

Author

• Kristen E. Dicerbo, Pearson