Using Log File Data as Evidence of Problem-Solving Skill in Technology-Rich Environments

• In Event: 41.025 - Learning, Engagement, and Assessment in Technology-Rich Learning Environments

Abstract

Objectives. Problem solving is an important competency throughout life, defined as the ability to engage in cognitive processing to understand and resolve problem situations where a method to solve the problem is not immediately available. One issue with teaching problem solving skills in a classroom context is that the problems presented in formal education tend to be qualitatively different from those encountered in the real world. That is, problems presented in assessment situations in schools are typically clearly defined and structured, whereas problems in real life are often ill-structured. Well-designed digital games offer a viable alternative to assessing and developing complex problem solving skills that are needed to succeed in the real world (Greiff et al., 2014; Shute & Wang, 2016). In this structured poster session, I will describe the design, development, and validation of an assessment embedded in a video game to measure the problem solving skills of middle school students.
Methods and Results. The game we employed was a slightly modified version of the popular game Plants vs. Zombies 2 (Popcap Games and Electronic Arts) called Use Your Brainz (UYB). This game served as the vehicle for our stealth assessment of problem solving. Our sample consisted of 55 7th grade students enrolled at a middle school in Illinois. They each played the game UYB on an iPad for about three hours across three consecutive days. On the fourth day, all students completed two external tests of problem solving skill: (a) Raven’s Progressive Matrices (Raven, 2000), which measures simple problem solving skills, and (b) MicroDYN (Wüstenberg, Greiff, & Funke, 2012), which measures complex problem solving skills. Each student also completed a demographic questionnaire concerning his or her age, gender, gaming history, and so on. Together, these assessments took about one hour to complete.
To establish convergent validity, we tested the correlations among our stealth assessment estimates of problem solving skill and the scores from our two external measures. Despite the relatively small sample size (n = 47), the results showed that our game-based assessment of problem solving skills significantly correlated with both Raven’s (r = .40, p < .01) and MicroDYN (r = .41, p < .01). Thus our problem solving stealth assessment appears to be valid.
Significance. Today’s students are expected to develop 21st century skills such as problem solving. Assessments can help educators determine not only students’ current levels of this competency, but also their strengths and weaknesses on particular facets of problem solving. This information can assist educators in supporting their students’ development of problem solving skills. Traditional formats for assessing learning are usually stripped of the context in which knowledge and skills are applied (Shute, Leighton, Jang, & Chu, 2016). Stealth assessment (Shute, 2011) was proposed as a promising method for assessing such a complex skill.

Author

• Valerie J. Shute, Florida State University