Paper Summary
Share...

Direct link:

Automatic Assessment of Students' Data Analysis Skills Across Physical Science Simulations

Sun, April 19, 4:05 to 5:35pm, Marriott, Floor: Fifth Level, Chicago FGH

Abstract

Computer-based activities like simulations and microworlds (e.g. Quellmalz et al., 2009; Gobert et al., 2012) are touted as a way to overcome the face validity limitations associated with assessing students’ inquiry skills with multiple choice-style items (e.g. Clarke-Midura et al., 2011; NRC, 2011). These activities require students to “do” inquiry, resulting in rich log traces that can be the basis for evaluating both their work products and the processes followed to generate those products (Rupp et al., 2010; Gobert et al., 2012; Sao Pedro et al., 2014). Though promising, assessment is still challenging because inquiry is complex, multi-faceted (Williamson et al., 2006), and can be demonstrated in many correct and incorrect ways.
To address these challenges and demonstrate the potential for using simulations to assess inquiry, we present our approach to evaluate and track sensemaking skills related to interpreting data and warranting claims with data (Kuhn, 2005; NRC, 2011) within the Inquiry Intelligent Tutoring System (Inq-ITS; Gobert et al., 2012). Inq-ITS is a web-based virtual lab environment in which students conduct inquiry with interactive simulations aligned to middle school Physical, Life, and Earth Science content. As students work, their inquiry is automatically assessed. The assessments provide the student-targeted, in-the-moment support (Sao Pedro et al., 2013a) and the educator actionable information on student progress.
We leverage our prior work on assessing data collection skills using Educational Data Mining (EDM) techniques (Sao Pedro et al., 2013a,b, 2014; Gobert et al., 2012) to assess data analysis skills as follows. First, we identified nine key subskills that could be measured from students’ usage of an inquiry support tool to analyze data they collected with a simulation. Example subskills include: identifying relationships between variables in their collected dataset, using controlled trials to warrant claims, and indicating if their hypothesis was supported based on their data analysis. Next, we developed knowledge-engineered evidence rules (cf. Almond et al., 2013) to automatically score these subskills. We authored rules, as opposed to learning rules via data mining (Sao Pedro et al., 2013b), because demonstration of these skills is relatively well-defined (Gobert et al., 2012). We then used the rules to score students’ data analyses, yielding observables of performance. Finally, we leveraged the observables to build latent models of each skill across science topics using Bayesian Knowledge Tracing (BKT; Corbett & Anderson, 1995).
In this presentation we determine if Bayesian Knowledge Tracing (BKT) can provide consistent estimates of students’ latent knowledge of data analysis. Our dataset contained 11,448 evaluations of subskills from 154 middle school students’ experimentation with three physical science simulations. Analyses showed that under student-level cross-validation, Classic BKT (Corbett & Anderson, 1995) provides overall consistent estimates of latent knowledge, meaning that high estimates of latent knowledge are associated with students’ positive demonstration of skill (A’ for each BKT model ranges from .58 to .83, where A’ = .50 indicates chance level performance). The present work on assessing data interpretation skills contributes to our existing work on using EDM-based methods to assess students’ inquiry skills.

Authors