Assessing Scientific Inquiry Skills Based on Multiple Sources of Evidence

• In Event: 63.057 - Automated Assessments in Science Inquiry

Abstract

Theoretical perspective and objective
The Next Generation Science Standards (NRC, 2012) require students to understand multiple aspects of scientific inquiry skills. For example, students may need to analyze and interpret data, construct explanations as well as obtain, evaluate and communicate information. Therefore, a formative assessment that aligns with NGSS needs to include aspects of content knowledge combined with science practice(s). Interactive simulations provide a natural platform for combining the application of knowledge with a particular science practice. Therefore, researchers are investigating how formative assessments may be enhanced by combining data from simulations with information gathered through more traditional test formats (i.e. multiple-choice and open-ended responses), which may provide rich information about student’s knowledge of the material (Forsyth et al., 2016). The purpose of this investigation is to examine the relationship between features of students’ open-ended responses and behaviors within an interactive simulation designed to assess scientific inquiry skills.
Methods
Participants were recruited from Amazon Mechanical Turk (N=309) and interacted with an altered PHet Simulation for assessment.
Data sources
Altered PHet assessment. An existing PHet simulation (Wieman, C. & Perkins, K., 2005) was augmented for assessment purposes. Specifically, students investigate the phenomenon of concentration in a simulated environment where they are able to manipulate aspects of the environment to run trials and collect data. Students then refer to their data when answering 7 different multiple-choice and open-ended questions interspersed throughout the interaction (see Figure 1).
TextEvaluator. TextEvaluator is a computational linguistic tool that analyzes hundreds of features of text. These features can be reduced into eight dimensions that indicate text complexity (Sheehan, Kostin, Napolitano, & Flor, 2014).
Analyses
Previous research with this assessment simulation environment found connections between actions in the simulation and students’ answers on multiple-choice questions (Forsyth et al., 2016). We posit that information about the nature of students’ open-response answers will provide an additional source of evidence that may correlate similarly with actions from the simulation. The open-ended responses were analyzed via TextEvaluator to generate linguistic indices for each student response. Furthermore, human raters scored responses on three features (e.g. providing evidence, correct reasoning, and logical reasoning) with preliminary Kappas ranging from .78-.84. Preliminary analysis suggests that indices from TE may correlate with the human scores. For example, the argumentation index from TE is highly correlated with the scores for correct reasoning (r =.405, p <.001) but less with providing evidence (r =.118,p =.04). These relationships may provide valuable insights when correlating TE indices to other sources of evidence which will be presented at the conference.
Significance
We posit that bringing together multiple sources of evidence will strengthen inferences about student understanding of scientific inquiry skills. Recent work has demonstrated that simulations can provide fine-grained information about student knowledge of scientific inquiry skills (Gobert et al., 2012; Forsyth et al., 2016). Therefore, the current study aims to extend previous findings and examine how these multiple sources of evidence can potentially be used to bring about a deeper understanding of student science inquiry skills.

Authors

• Dr. Carol McGregor Forsyth, Educational Testing Service

• Stephanie Peters, Educational Testing Service

• Jung Aa Moon, Educational Testing Service

• Diane Napolitano, Educational Testing Service