Exploring the Measurement of Learning Outcomes and Self-Regulated Learning in Game-Based Learning Environments

• In Event: AERA Graduate Student Research in Progress Roundtable Session Three
  In Roundtable Session: Learning With Games: Implications for Cognitive Flexibility, Self-Regulated Learning, and Self-Confidence (Table 9)

Abstract

Various educational games have been designed and developed to help learners learn relatively challenging subjects (e.g., biology, science, math, etc.) in more effective ways with high engagement and enjoyment (Plass et al., 2020). Learners can acquire complicated instructional material with less boredom and higher engagement in game-based learning environments (GBLEs), especially those with narrative-based content and engaging visualizations compared to traditional teacher-centered classroom environments where learners can become easily disengaged. However, beyond the increase of motivation and engagement in game-based learning environments (GBLEs), it is critical to understand and improve effective learning processes as a means of assessing if students learn valuable knowledge related to the contents of GBLEs. Since assessment methods for measuring the quality and quantity of valuable knowledge students learn related to the contents of GBLEs vary considerably, it is not easy to understand SRL processes' contributions to GBLEs. To evaluate the degree of learning in GBLEs, diverse methods can be utilized including game scores, post assessments, and embedded assessment. For instance, game scores measuring in-game progress or achievement (e.g., students completed tasks during games) can be utilized and test scores implementing after playing games can be used to assess students' domain knowledge (Taub, Azevedo, et al., 2020). However, despite efforts to establish best practices for evaluating the effectiveness of GBLEs, there is still a lack of scientific rigor (All et al., 2021). Since assessment after learning in GBLEs which frequently concentrates on the outcome and not process may neglect significant changes during the learning process (All et al., 2021), evaluating GBLEs with only post assessment might not be sufficient. If students achieve high game scores, does this mean that students acquired more valuable knowledge compared to other students with low game scores?

Moreover, how can we connect the learning outcome with self-regulated learning strategies in GBLEs? Although educational games have been designed to promote effective self-regulated learning (SRL; Dever et al., 2020; Taub, Azevedo, et al., 2020), there is still a lack of understanding of SRL processes during serious games. However, how can we measure students' SRL in terms of CAMM (cognitive, affective, metacognitive, motivational) processes in GBLEs which have a multitude of system elements (Azevedo et al., 2019)? If students succeed with GBLEs, does this mean that they use more SRL strategies that translate to non-game-based learning environments (e.g., virtual reality)? If students achieve higher scores on posttests after playing games, does this mean that they utilize greater SRL strategies? If students complete educational tasks during games but they get poor scores on posttest compared to pretest, how can we understand their SRL processes in GBLEs based on these divergent results?

In terms of these contentious issues, we would like to discuss an ongoing study with Crystal Island (Taub, Sawyer, et al., 2020). In Crystal Island, students are required to learn microbiology and engage in scientific reasoning while gathering clues from reading materials and interacting with diverse game items, formulating hypotheses, and testing evidence to solve a mystery, requiring SRL to foster content knowledge about microbiology and scientific reasoning skills (Dever et al., 2020). In this study, data from 26 high school students (M_age=16.04; SD_age=0.34; 65% female) and 26 undergraduates (M_age=19.73; SD_age=1.49; 69% female) were collected and analyzed. For each participant, we collected pre-test and post-test learning outcomes data. To assess learning gains, pre-test scores and post-test scores were calculated as normalized change score (Marx & Cummings, 2007). We also collected if the participant solved the mystery following game play.

We focus on learning outcomes (test scores vs solving mystery) based on learners' developmental levels with Crystal Island. Based on previous studies showing that learning outcomes were different across developmental levels in traditional learning environments (Mayer, 2019; Veenman et al., 2004), we expect that undergraduates will perform better on post-test and solving a mystery during the game compared to high school students. However, a t-test using participants' normalized change scores did not show any significant differences (t=-1.92, p=0.061) between high school students (M=0.12, SD=0.23) and undergraduate students (M=0.23, SD=0.22). Despite no significant difference at pre-test score, there was a mean difference in normalized changes scores for each group, but the huge variabilities in both groups lead to no statistically significant difference between developmental levels. That is, both developmental levels showed evidence of learning about the same with Crystal Island. When it comes to a difference in solving the mystery based on developmental levels, a 2X2 chi-square test found a significant difference in the distribution of students who solved the mystery correctly across development levels (?2=16.65, p<.01). Specifically, 24 undergraduates (92%) solved the mystery compared to only 10 high school students (39%). This suggests that learning outcomes and performance on scientific based reasoning were different, but microbiology content knowledge acquisition was not.

In the RiP roundtable, we will review and discuss the validity of measuring learning outcomes in GBLEs using various sources of data (i.e., learning outcomes, trace data). For example, although undergraduates and high school students did not show any differences on posttest scores, since they were likely to solve the mystery compared to high school students, should we interpret undergraduates learned more in Crystal Island? We will also discuss how we can understand SRL processes based on the learning outcomes with various trace data since these data (e.g., eye tracking, facial expressions of emotions, and log-files) can be utilized to understand students' SRL processes while using Crystal Island. For instance, how can we measure SRL processes in GBLEs when they are dynamic in nature and so is the GBLE? Which methods are most appropriate and accurate in measuring SRL within GBLEs (e.g., eye tracking, facial expressions of emotions, physiological sensors)?

Author

• Saerok Park, University of Central Florida
  Presenting Author