Adapting a Problem-Solving Experiment From In-Person to Online Testing During COVID-19

• In Event: Lessons Learned From Research During COVID-19: Strategies for Incorporating Remote Data Collection Beyond the Pandemic

Abstract

Experiments featuring game-like tasks are often used to study children’s learning. While experiments are typically conducted in-person in laboratories, museums, or schools, testing was shifted online during COVID-19. To meet this challenge, we created an online computer game version of an in-person problem-solving task that tested how children’s exploration supports learning. The experiment was run in synchronous video conference sessions with an experimenter. Creating an online computer game and testing children in synchronous sessions yielded unexpected challenges and benefits.

In the live experiment, 99 six-year-olds (mean age = 6.50 years) learned rules to unlock locks using keys (Figure 1). Children participated in one of three conditions that varied whether they explored the locks and keys independently (Exploration) or received instructions about the unlocking rules (Instruction): Exploration before Instruction, Instruction alone, or Exploration alone. Children’s immediate rule learning did not differ by condition, but children who explored prior to receiving instruction were better able to generalize the rules they learned to novel locks and keys. Data collection is ongoing to test 120 children in an online computer game version of this experiment to examine whether exploration prior to instruction similarly benefits generalization in a virtual environment.

Unexpected challenges emerged when creating the online game. Practically, we chose a platform to host the game (web browser rather than tablet app) and programmed participant gameplay data to save on university servers. Theoretically, it was challenging to replicate the gameplay environment and difficulty level: The live game had 3D objects that children manipulated, and the online version had 2D images that children clicked and dragged. Through piloting children of different ages with gameplay varying in difficulty, we ensured the online game was age-appropriate and sufficiently challenging.

The primary benefit of the online game was that gameplay data were tracked automatically. To quantify children’s behaviors in the live version, research assistants coded video recordings for behaviors of interest (e.g., strategy use). Here, gameplay data were used to calculate similar measures as those that were coded, which was faster and more precise. Further, new versions of the game can be easily programmed to test additional research questions.

Children tested in the online experiment participated in synchronous video conference appointments with an experimenter. Relative to in-person sessions, children may be more distracted in their homes and can more easily walk away from the game if they become bored or frustrated. However, testing children at home is more ecologically valid and less burdensome for families who would have had to travel to a testing site. Still, unlike asynchronous testing, an experimenter was present to keep children focused. This provided an intermediate level of control over sessions compared to more controlled in-person testing and less controlled asynchronous testing. In sum, online games played in synchronous sessions are a promising method for running experiments with young children during COVID-19 and beyond.

Authors

• Natalie Brezack, University of Chicago

• Mia Radovanovic

• Laura Shneidman

• Amanda Woodward