Using Educational Data Mining to Assess Students’ Experimentation Skills During Inquiry Within Complex Systems

• In Event: Developing Evidence of Understanding Complex Systems

Abstract

Here we address how computational techniques, namely, data-mining, can be used to assess students’ skills at designing controlled experiments (Sao Pedro et al., 2013), a lynchpin sub-skill of inquiry (NRC, 2011), within complex systems. This builds on our prior work assessing inquiry in Physical Science microworlds (Gobert et al., 2012; Sao Pedro et al., 2012, 2013), in which phenomena (at the middle school level) typically have one independent and one dependent variable and a linear relationship between them (cf. Greiff et al., 2012). However, in Ecosystems, the domain studied here, a number of interconnected, non-linear elements interact in a complex causal system (Yoon, 2008), making them difficult for students to understand (Hmelo-Silver & Pfeffer, 2004; Jacobson & Wilensky, 2006). As regards inquiry, this added complexity increases the hypothesis search space (Klahr & Dunbar, 1988; van Joolingen & de Jong, 1997), and understanding the effects of the independent variables on dependent variable(s) is much more challenging because of the interaction of the independent variables in the causal system. This, in turn, makes assessing students’ experimentation within complex systems very complicated because the simple control for variables strategy (cf. Chen & Klahr, 1999) cannot be applied in a straightforward manner.
In Study 1 by our group (Boschetto, 2012), middle school students used the Ecosystems microworld to hypothesize what change(s) in the initial populations of big fish, small fish, shrimp, and seaweed would cause the ecosystem to become stable, and then collected data to test their hypothesis. In Study 2, using the same microworld and a classroom-based data set of 101 middle school students from Central MA, we address whether our algorithms to evaluate whether students design controlled experiments built for Physical Sciences (in which variables do not interact in a complex system), can be validated and generalized to assess of this skill in Ecosystems -- a particularly strong test of generalizability since the variability in how students collect data during inquiry in complex systems is wider.
Our results revealed strong evidence that our algorithm can evaluate whether students are designing controlled experiments in the Ecosystems microworld based on their experimentation patterns. The algorithm can distinguish when a student designs controlled experiments in Ecosystems from when they do not 75% of time (A’ = .75). Agreement with human judgment was also quite high, with a kappa of .61. The performance of this algorithm is on par with that for our physical science microworlds (Sao Pedro et al. 2013); thus, it can be used successfully to evaluate new students’ performance in the Ecosystems activities. This is of particular importance both because there is no one single right or wrong way to design controlled experiments within complex systems, such as ecosystems, and our algorithm can handle these nuances and complexities of how students conduct experimental trials in this domain. Our methods for building algorithms of this type will be discussed, as will implications for how data such as these can be used to scaffold students in real time when conducting inquiry in complex domains.

Authors

• Janice D. Gobert, Worcester Polytechnic Institute

• Michael A. Sao Pedro, Worcester Polytechnic Institute & Apprendis LLC

• Cameron Betts