Evaluating the Efficacy of Real-Time Scaffolding for Data Interpretation Skills

• In Event: Cognitive Models for Assessment

Abstract

Cultivating science inquiry skills is increasingly seen as important to students’ science education (NGSS Lead States, 2013). However, learning these skills can be difficult (cf. de Jong & van Joolingen, 1998) because they are complex (e.g. Gobert et al., 2013). Effective scaffolding can support students’ inquiry, resulting in a deeper understanding of these processes (Sao Pedro et al., 2013; Sao Pedro et al., 2014). In prior work we showed that in an intelligent tutoring system scaffolding can help students who did not know two inquiry skills – testing hypotheses and designing controlled experiments – acquire these skills and transfer them to a new science topic. This scaffolding approach was then applied to support students’ data interpretation. This work (1) analyzes the efficacy of the data interpretation scaffolds using an extension of the Bayesian Knowledge Tracing (BKT) framework accounting for scaffolding as a cognitive modeling approach for approximating mastery learning of the inquiry skills of interest and (2) discusses modifications to this framework which allow it to be applied when student data, practice opportunity, and evaluated skills are not clearly delineated.

The scaffolds used (Moussavi et al., 2015) were designed to support students in real-time with data interpretation skills as informed by research documenting data interpretation difficulties (Klahr & Dunbar, 1988; Schunn & Anderson, 1999; Kuhn et al., 1992) and our research on students’ inquiry behaviors. Consequently, these scaffolds address such skills as: matching variables in the claim to those in the hypothesis, selecting controlled data for warranting, creating an accurate claim reflecting the data, and stating whether the claim supports the hypothesis. Furthermore, a successive scaffolding strategy was used allowing students to receive increasingly targeted support, similar to Cognitive Tutors (Corbett & Anderson, 1995).

Our data was collected from 160 eighth-grade students using three Density activities in the tutoring system. Students were randomly assigned to either the “Scaffolding during Interpretation” (n=78) or “No Scaffolding during Interpretation” (n=82) condition. BKT was used to approximate student learning of the data interpretation skills. However, because the data logged here differed from typical data logs due to how the data interpretation scaffolds were integrated into the system, it became important to consider how the BKT framework defined skill demonstration, scaffolding condition, and practice opportunity to create an accurate model.

Using this BKT extension, we analyzed students’ error rates for the data interpretation skills and found that students who receive scaffolding improve faster than students who do not. BKT analyses also indicated the scaffolds were effective in supporting the learning of this inquiry skill as the probability of learning, p(T), was higher for students receiving scaffolding for all but one of the evaluated skills, which may be due to the high probability of initial knowledge attributed to that skill. These findings suggest scaffolds that target common procedural errors in data interpretation can help students learn and apply these skills, as such this work represents an advance in real-time scaffolding of students’ data interpretation skills.

Authors

• Raha Moussavi, Worcester Polytechnic Institute

• Michael A. Sao Pedro, Worcester Polytechnic Institute & Apprendis LLC

• Janice D. Gobert, Worcester Polytechnic Institute