A Game-Based Assessment to Measure Persistence and Response to Failure After Design-Focused Summer Camp Programs

• In Event: Transfer for Learning Outside the Classroom: Informal Contexts to Examine STEM Learning Skills and Strategies

Abstract

Design thinking and making are becoming increasingly common in formal and informal educational spaces, where they are believed to encourage productive approaches to problem solving (Martin, 2015; Carroll et al, 2010). Here, we describe the development process of an online game called Farmlet, which aims to examine how summer camps focused on making and design influence campers’ approaches to problem solving.

Farmlet was developed through an iterative top-down and bottom-up process in collaboration with the summer camp organization. We worked with the organization to identify learning strategies they thought the design activities were helping campers develop. Through on-site observations, several discussion cycles with the camp staff and curriculum developers, and consultation with the research literature, we narrowed these down to the construct of response to failure as the target for measurement. This was further broken down into three sub-constructs:
1) Persistence after failure
2) Willingness to share incomplete or imperfect ideas
3) Examining unsuccessful attempts at a solution in order to learn from them

We worked together to identify behaviors that someone who had learned productive strategies in response to failure might take in a new problem-solving situation and built opportunities to engage in these strategies into the game mechanics.

In the game, learners plant seeds to feed various animals. The game goes through a series of increasingly challenging levels. A feedback screen shows how well the garden grew and if it met the needs of the animals. Primary measurement points include learners choices whether or not to:
1) Try again to replant their farm or quit the level and choose a different animal to feed.
2) Share their results with their growing community, which is made of other characters they have chosen.
3) Look in the community book, which displays successes and failures that have been shared, along with some notes
4) Re-observe the garden growth animation in slow-motion to try to understand what happened.

Farmlet was initially tested with one cohort of campers, and relevant strategy metrics were identified through an exploratory process. These metrics were then used a priori in later studies. Participants were rising 3rd -5th graders at two kinds of summer camp programs across three years. Approximately 900 campers were part of a private camp network. Approximately 500 were part of a public school program for academically at-risk students that used a curriculum developed by the private camp network. Campers engaged with various design projects, such as designing and building a car. For the private camp, these kinds of activities were the focus of a full day, week-long experience. For the school camp, they were the afternoon’s activities spread across four weeks.

After attending camp, children showed increased persistence in response to failure. Results relevant to sharing were less clear. One reason was that our measure of sharing involved canned characters, which may not reflect response to a real community. Additionally, the camp realized they did not focus on sharing as much as they initially believed. We are continuing to refine our sub-construct measures, as well as testing with other design-focused programs.

Authors

• Kristen Pilner Blair, Stanford University

• Doris B. Chin, Stanford University

• Daniel L. Schwartz, Stanford University