Remote Evaluation of Digital Games for Social and Emotional Learning

• In Event: Lessons Learned From Research During COVID-19: Strategies for Incorporating Remote Data Collection Beyond the Pandemic

Abstract

In our work studying learning and measurement in digital games for young children, we rely on relationships with school districts and preschools for recruitment and in-person data collection. When the pandemic necessitated a completely remote study of social and emotional learning (SEL) games, we found that despite challenges, the remote design had unanticipated benefits.

We conducted a nation-wide analytics validation study of three PBS KIDS games created to measure children’s SEL. With the goal of validating in-game indicators of SEL with external measures, we collected gameplay data from 144 children (aged 4-5) and measures of their SEL. Families were mailed tablets on which children played the games for >15 minutes for six days and completed one direct measure of SEL. Parents completed two indirect measures of their children’s SEL. Data from the tablet activities were uploaded to an online server.

The greatest challenge was that data collection was not overseen by researchers. Gameplay and completion of measures took place in families’ homes, without researchers present, which introduced concerns about (a) data quality and (b) possible technical difficulties. We were able to mitigate these concerns in several ways. We created a clear communication plan (including written instructions) to convey the type and amount of assistance parents could give their children with the games and SEL measure. We met via Zoom with each participating parent twice: once to obtain consent/permission, and again after they had received their tablet to explain the study activities and answer questions. To preempt possible technical problems, we helped them connect the tablet to wifi during the meeting and provided data plans for those without wifi. Using a device management program, we could verify the tablet was online before we ended the meeting. If the tablet was used offline, gameplay data were queued and uploaded upon reconnection and the SEL measure data were saved locally on the tablet as a backup. We also provided assistance via email, phone, and text message. We found the vast majority of our sample played the games each day as requested, and parents reported providing little assistance. Because most of the data were collected online, this study was a great candidate for remote research, and participation was more ecologically valid given that it took place at home, at their leisure, rather than in a school setting.

We found recruitment logistics simpler than with in-person studies. Usually, our recruitment involves extensive coordination with the district, schools, and teachers, all before sending home consent forms. Recruiting families through email blasts and social media meant that we worked directly with interested families and scheduled participation at their convenience. Snowball sampling allowed us to reach even more prospective participants. However, this strategy resulted in a sample that was not representative of the general population (e.g., 56% of the parents held a graduate/professional degree). Nevertheless, with an adapted recruitment strategy, we plan to continue to utilize remote design in future studies to recruit a more inclusive sample, without geographic and school district restrictions.

Authors

• Elizabeth Redman, University of California - Los Angeles

• Joanne Michiuye, University of California - Los Angeles

• Charles Parks, University of California - Los Angeles