Assessing Science Practices Using a Bayes Net

• In Event: 66.030 - The State of the Art in Automated Scoring of Science Inquiry Tasks

Abstract

The Voyage to Galapagos (VTG) project, in which we are investigating how best to support students with system-driven assistance, takes a novel approach to scoring of interactive tasks as students engage in an exploratory science learning environment in which they follow in the footsteps of Charles Darwin to learn about evolution. In VTG, students progress through a sequence of three levels in which they complete a series of tasks. For example, in Level 1 students are asked to collect a sample of iguanas from the islands, which shows the range of variation among the iguana populations. As they undertake the data collection task by exploring the islands, taking photos of iguanas that they see and saving them to their logbook, the system logs their actions.
The novel scoring approach in VTG is that rather than monitoring only the correctness of their actions and modeling their content knowledge, a Bayesian Network is used to collect data about student actions and assign probabilities of students having acted in such as way that suggests they are struggling with the task. The Bayes Net contains a decision node that, when the probability exceeds a threshold value, turns on the assistance. The Bayes Net has three top layers that range from the most general to most specific – the Knowledge, Skills, and Abilities (KSA) Layer, the Error Evaluation Layer, and the Error Diagnosis Layer. The specific nodes at each of these layers have associated error feedback and hints that are triggered when the nodes at the associated level reach a certain threshold. Whether a student receives the feedback or hints is configurable according to (a) what experimental condition they are in and (b), in the case of hints, whether they request help. By allowing the assistance to be configured in this way, we are able to create the conditions of assistance that are the focus of our experimental design in which we will examine five levels of assistance: (1) no support, (2) error flagging only, (3) error flagging and text feedback on errors, (4) error flagging, text feedback on errors, and hints, and (5) preemptive hints with error flagging, error feedback, and hints.
Initial pilot testing of the VTG system has been conducted and preliminary results indicate that there are differences in how well students are supported under the different conditions of assistance. As expected, conditions 2 through 5 provide better support than condition 1 (no support) but it is not yet clear which of the other conditions supports which students best and it seems that students with different levels of knowledge at the start might respond differently to the types of assistance. These data are being analysed and will inform adjustments to the assistance system before VTG is used in a randomized controlled trial of the system that will take place in late 2014. From this study we will be reporting which levels of assistance work best in the exploratory science learning environment.

Author

• Michael Timms, Australian Council for Educational Research