Affect Detection in Physics Playground

• In Event: 18.057 - Physics Playground, a Game for Learning: Demonstration and Posters

Abstract

In this poster, we discuss work to detect student affect within the context of Physics Playground.
Affect detectors were developed using a combined approach involving interaction logs, video data, and quantitative field observations. This combined data was obtained for 133 students who used Physics
Playground and took pre-tests and post-tests in authentic classrooms over the span of four days. The quantitative field observations were conducted by two BROMP-certified coders; BROMP certification requires Cohen’s Kappa of 0.6 or higher on observations conducted in a real context of use. BROMP observations are conducted according to a specific protocol where students’ affect is coded according to a pre-determined schedule and coding scheme. In this case, students were coded for the affective states of boredom, confusion, engaged concentration, frustration, delight, dejection, and ? (other or impossible to code). A total number of 2,374 observations of student affect were made. 1,838 of these observations were made during use of Physics Playground (as opposed to the pre- and post-tests) and had valid data. Some observations were lost or removed due to computer crashes and students logging into each others’ accounts.

Interaction logs from Physics Playground were distilled through a feature engineering process, capturing 113 features regarding the number of objects of specific types the student drew, the presence of behaviors related to other strategies (such as stacking lines, a way to game the system), the time the student took between actions, and other actions and events that occurred as the student played each level. Video features were created using the FACET facial recognition tool, which probabilistically identifies 19 faction action units, and also identifies head positioning and orientation.

The detectors were built using standard classification algorithms, with downsampling and upsampling methods used on the training sets (but not the test sets) to adjust for class imbalance in the data
(i.e. some affective states were rare). Data was cross-validated at the student level, so that each model was repeatedly built on one set of students and tested on a different set of students.

For most affective states, the detectors based on video data performed moderately better than the detectors based on interaction data (although confusion was a notable exception to this pattern; it performed better on the interaction data). A combined detector, using both facial and interaction data, typically performed slightly better (but not for delight, where performance was excellent to begin with).
Another advantage of the combined detector was that the interaction detectors were able to function in a larger number of situations than the video detectors; the video detectors sometimes could not make inference due to factors such as hats, water bottles, or chewing gum. As such, a combined detector was able to leverage the higher-quality of video-based detection when it was available, but make predictions about affect in a broad range of situations, leveraging the greater availability of interaction data.

Authors

• Ryan Shaun Baker, University of Pennsylvania

• Sidney K. D'Mello, University of Notre Dame

• Shimin Kai, Teachers College, Columbia University

• Philip Nigel Bosch, University of Illinois at Urbana-Champaign

• Jaclyn Ocumpaugh, Worcester Polytechnic Institute

• Valerie J. Shute, Florida State University