Enabling Students to Self-Assess With Guidance From a Multi-Agent Environment for Supporting Scientific Inquiry

• In Event: Measuring Self-Regulated Learning With Multi-Agent Learning Environments

Abstract

Our Inquiry Island software provides a community of software advisors who guide students as they engage in empirical research projects, such as science fair projects. Each advisor embodies a different area of cognitive, social, or metacognitive expertise that is needed for scientific inquiry, such as Ivy Investigator, Cahlil Communicator, and Pablo Planner. The advisors recommend goals to purse, provide motives for pursuing each goal and suggest strategies for achieving them. To support monitoring and reflection, each advisor provides a set of self-assessment sliders, which typically map onto the goals the advisor advocates pursuing. For Ivy Investigator, for example, one goal is to design an experiment that tests your competing hypotheses. The corresponding sliders for each goal have several settings, each corresponding to the value of a qualitative variable. For example, the slider for Tests Hypotheses asks, “Have you designed your investigation so that it will test your hypotheses?” and it provides several possible responses:
• Definitely: We've figured out what experimental conditions are needed to test our hypotheses, including what things to vary, control, and measure.
• Not clear: We have only a general idea of what to do and how to control for variables not related to our hypotheses.
• Probably not: We don’t think that our investigation will test our hypotheses.
Such self-assessment is doubly useful in the Inquiry Island system; it helps students judge their own work and also serves as a source of input to the advisors' production systems. Students are made aware that they are not only reflecting on their progress, they are also communicating that progress, or lack thereof, to the advisory agents, leading to advice from the agents on how to improve their work.
The focus in this study is on such self-assessment. It investigates how middle-school science students use the Inquiry Island software to self-assess, how the software records the various ways in which they self-assess, and how students self-assess in interesting and potentially important ways that are not captured by the software.
The software was deployed in several middle-school classrooms in the support of various inquiry science curricula. As students used the software, we collected keystroke and interaction logs, most importantly when and for how long the students interacted with the self-assessment tools and advisory agent system. Additionally, each computer was outfitted with a web-camera that recorded a video stream of student conversations with peers and teachers. That video record was synchronized with the keystroke log to afford qualitative researchers a replay of everything every student did using the software and everything they said to each other as they worked.
We then developed several metrics of self-regulation, applied both automatically (software pattern recognition) and manually (human coder) to our data sources. Those metrics were shown to correlate with important indicators of student learning, such as standardized test scores and pre-post gains on a test of inquiry skills. This research adds to our understanding of self-assessment and of how to facilitate and track it through software tools, advisors, records and data displays.

Authors

• Eric M. Eslinger, University of California - Berkeley

• Barbara Y. White, University of California - Berkeley

• John R. Frederiksen, University of Washington