Agents in Collaborative Problem Solving in Games and Simulations

• In Event: Symposium: Games and Simulations: Learning Analytics and Metrics

Abstract

Objective
This presentation describes how conversational computer agents can play a role in both tracking and assessing collaborative problem solving in high stakes assessments, simulations and games.

Perspective
Agent-based systems are designed to (a) assess the students’ knowledge, skills, actions, and various other psychological states on the basis of the students’ actions and the conversational interactions, (b) generate discourse moves that are sensitive to the psychological states and the problem states, and (c) advance a solution to the problem. We describe how this is accomplished in two environments. The first is the Programme for International Student Assessment (PISA) for Collaborative Problem Solving (CPS) in 2015 (OECD, 2013). Second, there is a multiparty game called Land Science in which the system tracks progress and knowledge states of small groups of 3-4 students who attempt to learning urban and rural planning (Bagley & Shaffer, 2015).

Methods and Data Sources
In the PISA CPS 2015 assessment, a single human test taker (15-year old student) interacts with 1, 2, or 3 agents that stage a series of assessment episodes. During the course of each episode there are a small number of interaction turns (2 to 5) between the agents and the test taker and a small number of chat or action options (3-5 typically) for the test taker at each turn. Performance is mapped on to theoretical categories of problem solving and collaboration. In the virtual internship Land Science (Bagley & Shaffer, 2015), students in the Massachusetts Audubon Society play the role of interns at Regional Design Associates, a fictional urban and regional planning firm. Their problem solving task is to prepare a rezoning plan for the city of Lowell, Massachusetts that addresses the requests of various stakeholder groups (business, environment, industry, or housing) that have views on socioeconomic and ecological issues, some of which are incompatible. During the course of planning, students discuss options with their project teams through online chat.

Analysis
In PISA, each student’s responses in the CPS environment are categorized into the theoretical categories of collaboration and problem solving. Then the responses are scaled. In the land science game, the log files of these interactions were tracked, mined, and categorized on theoretical categories. The categories extracted from data mining procedures and evidence-centered design are identified and significantly correlated with solutions to problems and other performance measures. Data are reported on these computational analyses and correlations with outcome measures.

Significance
The PISA framework has been successful in scaling test takers on the theoretical constructs (collaboration, problem solving) specified in the PISA CPS 2015. PISA as a high stakes test influences educational policy in multiple countries. These results will be described, with an eye to describing how the PISA categories also map onto those in other data sets, i.e., land science game. The results of the Land Science simulations uncovered a number of discourse patterns that predict learning. These metrics also will be useful to measure 21st century skills in other countries.

Author

• Arthur C. Graesser, The University of Memphis