Storylet: (NOT) Measuring Change in Students' Interpretations of and Dispositions Around Data Representations

• In Event: When Nothing Is Something: Null Results in Informal Learning Research

Abstract

Objectives: Real World, Real Science (RWRS) is a collaboration of educators, scientists, exhibit designers, and researchers that worked to build interactive learning experiences, curriculum, and novel assessment tools around climate science and data literacy for 5th and 6th graders using NASA Earth systems data. In service of this goal, we developed Storylet -- an online game-based assessment to measure the impact of these learning experiences on students’ ability to interpret data representations and their willingness to use data as a form of communication.

Perspective: The learning technology discussed here is an example of a choice-based assessment (CBA). CBAs are interactive technologies grounded in the theoretical framework of preparation for future learning (PFL) assessments, which afford students opportunities to learn during the evaluation by presenting them with a problem and learning resources (Bransford & Schwartz, 1999). By design, CBAs provide multiple pathways to success, allowing students to choose how they learn and to demonstrate their use of disciplinary practices and strategies. Pintrich and De Groot (1990) describe this as “skill vs. will” - students must not only know a learning strategy, they must also choose to apply it. CBAs are frequently designed as short, online games, making them ideal vehicles to cross between formal and informal environments (Authors, 2019; Authors, 2016).

Methods: In Storylet, students assume the role of new editor for the school newspaper. They must read two stories (~300 words each; on one ecology- and one non-ecology-related topic), and then choose images which best support the stories from a variety of photographs and data representations (e.g., bar graphs, scatterplots, choropleths). Several iterations led to modifications to 1) encourage learners to more carefully study each prospective image and 2) limit learner fatigue during gameplay.

Data: In this study, we employed Storylet as a pre-post assessment for 175 students. The post was administered after a 2.5 hour informal museum experience and at least one curriculum lesson. We analyzed each student’s game clickstream, including: time spent reading the passages, number and duration of views per possible image, sequence of images viewed and removed, etc.

Results: While we found some change in behavior for metrics related to the ecology stories (e.g., number of final data representations), most of our results (e.g., time spent viewing representations, number of representations chosen reflecting trends in the story) showed no change. We hypothesize that several factors could have contributed, including a mismatch between the strength of the interventions and the targeted learning skills; uneven curriculum implementation across educators; and limitations of the assessment itself, including the nested design which restricted sample size.

Significance: The results from this work demonstrate layered nuances in the PFL and “skill vs. will” frameworks. Based on an external post-test, we believe the learning experience did in fact improve students’ data literacy skills. However, Storylet did not successfully measure the “will” (choice) to apply this new knowledge. While results were not statistically significant, we measured positive trends in behaviors, showing promise for this type of work in future studies.

Authors

• Rachel Wolf, Stanford University

• Doris B. Chin, Stanford University