Using Model-tracing to Conduct Performance Assessment of Students’ Science Inquiry Skills Within a Microworld

• In Event: New Measurement Paradigms: Psychometric Methods for Technology-Based Assessments

Abstract

The frameworks for science emphasize inquiry skills (NRC, 1996), however, their assessment is problematic (Fadel, Honey, & Pasnick, 2007; Quellmalz, Kreikemeier, DeBarger, & Haertel, 2007; Black, 1999; Pellegrino, 2001).
Science Assistments (www.scienceassistments.org) learning environment assists and assesses (hence, “assistments) middle school students on inquiry so teachers can assess their students’ skills during instruction--in the context in which they are developing (Mislevy et al, 2002). We describe a proof-of-concept for performance assessment of a set of inquiry skills based on model-tracing (Corbett & Anderson, 1995; Koedinger & Corbett, 2006). Additionally, we present Cronbach’s alphas as reliability measures for each of our variables, and correlations with other inquiry tasks as additional construct validity data.

Method
78 eighth grade students (aged 12-14 years) from a public school in Central MA participated. Students belonged to one of six class sections and had one of two science teachers.

Materials. Pre- and post-tests for inquiry skills (n=12) and domain knowledge (n=7) were used. A Phase Change Microworld activity was used with which students engaged in a series of inquiry tasks.

Data Collection and Analysis: By applying model-tracing to students’ log data from their interactions with microworlds, we use production rules to code for: 1) CVS-relevance for each of the trials (using the control for variables strategy and relevant to the student’s hypothesis), 2) tested-and-true hypotheses for each of the trials (whether a claim is supported based on data), and 3) lastly, an average of these scores, referred to as %cvs+true-tested for each of the trials.
Our model tracer tracked whether: students’ initial hypotheses were scientifically accurate, whether the experimental trials they ran were relevant to their hypotheses, whether their trials used the control for variables strategy (Chen & Klahr, 1999), whether their final analysis entered was supported or unsupported by their data, and whether they had collected appropriate experimental evidence that supported their final conclusion (relevant controlled trials).
Using data from the model-tracer, we calculated Cronbach’s alpha for our variables to ascertain the reliability across the 4 trials on each of the measures. The Cronbach’s alpha for the 4 CVS-relevant scores was 0.683; the Cronbach’s alpha for the 4 true-tested scores was 0.741; and lastly, the Cronbach’s alpha for the aggregate of the 2 inquiry scores across the 4 trials, %CVS+true-tested, was 0.774, indicating a high degree of internal consistency for each of the three measures.
Correlations were calculated between our auto-scored performance measures of inquiry with specific post-test inquiry items that should, in theory, be related. We obtained moderate correlations between our performance measures of inquiry and our post-test items for identifying an independent variable, identifying a dependent variable, and demonstrating the control of variables strategy (CVS).

Conclusions: In this paper we have shown that model-tracing can be used as a method of performance assessment for science inquiry skills, an ill defined domain. Additionally: 1) we can reliably capture students’ inquiry performance on these rich inquiry tasks, and 2) our measures are moderately correlated with post-test measures of inquiry performance for analogous concepts.

Authors

• Janice D. Gobert, Worcester Polytechnic Institute

• Kenneth R. Koedinger, Carnegie Mellon University