Achievement Systems and Learning: Methods for Analyzing Digital Badge Systems

• In Event: Democratizing Learning Through Digital Badges: Theoretical and Analytical Frameworks to Advance Design and Research

Abstract

Objectives
In this poster we present the analysis of digital badge systems created in partnership with two informal science-learning organizations and a university to award college credit in science-focused programs. We introduce methods for evaluating and measuring badge systems and to better understand how badge systems position learners, support learning, and implications for equity. The methods include; 1) analysis of digital badge criteria and, 2) analysis of digital badge systems as a whole. Our research question is, in what ways do digital badges impact youth science learning, identity, and equity?

Theoretical Framework
We draw from Design-based Research (Barab, 2006) to inform an iterative design and implementation approach. We reference Pinkard et al (2008) to consider how badges support developing digital literacy and Deterding’s (2011) Gameful Design to identify desired badge qualities.

Method
We use the cultural learning pathways model (Bell, et al, 2012) to understand the ways in which badge criteria position learners in certain ways, make actions visible, and open up scopes of possibility for learners who can be in designed spaces. We use open and axial coding to identify themes and analyze badge criteria and badge systems.

Data
For this poster we analyzed data collected from design documentation, platform and badge analytics, and program developer and participant interviews.

Conclusions
We learned that the design and implementation of badge systems is a complex process that is not easily replicated across different organizations or programs. Therefore, close analysis of badge criteria, systems' design and structure, become critical for understanding how badges can support identity development, learning, and inclusivity.

In badge criteria, we see evidence of how the learner is positioned as a participating expert within 6 of a set of 8 badges for one program. These badges require youth include “personal observations” as part of their scientific observations. Another theme found in badge criteria is authentic scientific practice. A subset of badges requires youth to complete work of an environmental scientist.

In the badge system we identified attributes such as badge sequencing and progression, badge relationships and interdependency with each other, frequency of badge distribution, and linearity of badge pathways. We found badge system attributes such as frequency, sequencing, and progression of badge distribution impact youth’s agency and choice within the program. We also found badge systems provide an opportunity to formalize the evaluation of youth work (e.g. peer feedback) that normally would not be recognized in the programs’ formal assessment structures. Lastly, we found that badge systems can embody stated program values and culture. For example, combining playfulness and scientific content knowledge in a subset of badges (e.g. bog bounce).

Significance
Concrete methods for evaluating and understanding the scope of digital badge influence and affect on learning is needed in order to better design future educational programs that include badges. In this poster we address this gap as we consider which methodologies are best for researching and evaluating digital badges in educational programs and whether these methods are adequate for measuring learning and inclusivity.

Authors

• Theresa Horstman, University of Washington - Bothell

• Carrie T. Tzou, University of Washington - Bothell