Developing and Evaluating Stealth Assessments for 21st-Century Competencies

• In Event: Knowing What Students Know and Feel: Innovative Technology-Rich Assessments

Abstract

Our research involves developing and evaluating evidence-based stealth assessments within games to estimate a player’s current level on valued competencies. During gameplay, students produce rich sequences of actions while performing complex tasks, drawing on a variety of competencies. Evidence needed to assess the competencies is thus provided by the students’ interactions with the game itself. The stealth assessments will measure three focal competencies: creativity, persistence, and conceptual physics in the game Crayon Physics Deluxe. Crayon Physics Deluxe (CPD) is a computer-based game that emphasizes two-dimensional physics simulations, including gravity, mass, kinetic energy, and transfer of momentum. Data will be collected in CPD from students’ interactions in the game to inform our three focal competencies. The stealth assessments will be developed based on the evidence-centered design (ECD) framework, which requires a systematic analysis of the assessment argument, including the claims to be made about the learner(s) and the evidence that supports those claims (Mislevy, Steinberg, & Almond, 2003).
Stage 1 will involve creating three stealth assessments that will be embedded in CPD. We will be using problems (i.e., game levels) that already exist in CPD, and we will also create new ones with the game's editor. We plan to pilot test the problems to determine if they're suitable for our population and for our methodological requirements. Pilot work will be conducted with 200 middle school students at the Florida State University School. In stage 2, we plan to conduct two studies to evaluate the validity of the stealth assessments, examine learning from the game, and test the scalability of the stealth assessments to other games. Study 1 (n = 120) will evaluate the validity of our three stealth assessments in CPD. Students will (a) complete a pretest battery of traditional tests on our focal competencies, (b) interact with 15 CPD problems over three 1-hour sessions in the computer lab at their school, and (c) complete a posttest on conceptual physics understanding. Students' competency levels will be estimated from their gameplay in CPD, and the competency estimates will be correlated with scores from the traditional tests. The results of the study will inform us as to the validity of the stealth assessments for the three focal competencies (creativity, persistence, and conceptual physics) and provide us with preliminary evidence for conceptual physics learning in CPD. In Study 2, we will employ our ECD models of the stealth assessments in a different digital game (e.g., World of Goo) and a subset of the students in Study 1 will be used in Study 2 (n = 80) to evaluate how persistence can be assessed in this second game. Students will interact with the second game over two 1-hour sessions in the computer lab at their school. At the end of the sessions, competency estimates for persistence will be compared to the Study 1 estimates and to scores from the traditional tests to evaluate the (a) validity of the assessments, and (b) scalability of the models (i.e., developed for one game and reused within another one).

Authors

• Valerie J. Shute, Florida State University

• Matthew Ventura, Florida State University

• Yoon Jeon Kim, Florida State University