Using Virtual Labs in NAEP to Assess Students' Scientific Inquiry Skills

• In Event: 46.027 - Issues and Developments in NAEP Technology-Rich Assessments: Representative Work of NAEP–Educational Testing Service Summer Undergraduate Research Experience/Summer Predoctoral Research Experience Fellows

Abstract

Research has shown that in traditional lab environments students tend to follow recipes and gather and record data without a clear sense of the purposes and procedures of their investigation and their interconnections (NRC, 2006, p.3). This talk will introduce a tablet-based virtual science lab currently under development that allows students to design their own experiments instead of following procedures. We explain how we use principles of Evidence-centered design (ECD, Mislevy, Steinberg, & Almond, 2003) in this virtual science laboratory to measure students’ scientific inquiry skills. The evidence model provides specific expected performance to ensure that the design of the virtual lab is appropriate to capture actions that elicit student ability in target areas.

The Evidence Model
Our evidence model is built off the 2003 TRE (Technology-Rich Environments) Simulation scenario evidence model with a particular focus on understanding scientific inquiry exploration and synthesis skills. Core fundamentals in our evidence model include: the degree to which students use science help, plan experiments to solve problems, understand experimental design through variable manipulation, analyze and interpret data, and communicate results after reflection and revision of initial claims. Within each of these core areas, we will develop specific features that will allow us to investigate and quantify students’ scientific inquiry skills into different levels. For example, one particular feature under planning could be selecting relevant tools, where we look at the percentage of tools that the student acquires that are actually related to the experiment. In other features, we can distinguish high and low performance based on the specific actions that the students take. For example, students can repeat an experiment while keeping some variables fixed and manipulating other variables.

We have developed a list of features that are designed to capture the actions of 8th grade middle school students in a way that reflects their understanding of science and scientific inquiry. The evidence model records the sequence and the method in which the student attempts to solve a problem. The amount of time the student spends in specific sections of the lab environment is observed and compared to the total time spent in the experiment. We will also use the timing data to look for correlations between the students’ scores in the lab and the relative time spent on specific tasks. The resulting variables will be analyzed along with the students’ overall score or grade in the lab to build an understanding of where students’ struggle and consequently what areas of learning teachers need to emphasize in their curriculum. Based on the evidence model, modules of lab activities will be developed and qualitative focus studies will be conducted to assess whether the evidence model captures the major features of students’ actions and whether the task provides sufficient and appropriate evidence to differentiate students’ scientific inquiry skills. Results from the focus studies will provide implications to revise the evidence model and the design of the task.

Authors

• Anthony Tucker-Bartley, Wagner College

• Jack Weiyu Hu, University of North Carolina - Chapel Hill

• Lei Liu, Educational Testing Service