SimScientists Assessments

• In Event: Genres of Formative Assessment

Abstract

Technology-enhanced assessments that teachers may insert at key points in a learning sequence may serve as powerful resources for classroom formative assessment (Stiggins, 2006). Technology can support teachers’ needs to align, deliver, and score assessments within rich task environments that measure deep understandings in a feasible, and cost-effective manner (Quellmalz & Haertel, 2004).
This presentation will describe the design, development, and validation of the SimScientists Assessments, simulation-based science assessments that have been designed as curriculum-embedded assessments to be used formatively by teachers and students. As students work through the simulations, the embedded assessments provide immediate, individualized feedback and three levels of coaching. The simulation-based assessments provide progress reports with recommendations for grouping students into simpler or more complex activities in an offline reflection activity the following day. Student groups collaborate and “jigsaw” their work to communicate their findings.

Objectives
This presentation will describe the theory and research guiding the design, development and validation of the simulation-based assessments, the evidence-centered design development process, and studies in three states of the usability, feasibility, and technical quality of the assessments. Implications for the integration of simulations into state science assessment systems will be proposed (Quellmalz &Pellegrino, 2009).

Perspectives
The SimScientists assessments are shaped by frameworks of model-based learning and evidence-centered design (Buckley 2012; Mislevy, Almond & Lucas, 2004). The science simulations represent the cross-cutting organizational model of all science systems--their components, interactions, and emergent system behavior. Evidence-centered design was employed to specify the student models (targeted science knowledge and skills), evidence models (how evidence will be scored and reported), and task models (features of tasks and probes that will elicit evidence of the targeted outcomes.

Methods
The SimScientists assessments proceeded through a systematic design and development process. National science frameworks and standards shaped the student models. Storyboards specified representations of the dynamic science phenomena to be provided in the tasks and items. An iterative process of cognitive labs with students and teachers talking aloud as they worked through the assessments provided early evidence of construct validity. The assessments were tried out in pilot test classrooms, then field tested across schools and classes in three states.

Data
An evaluation of three classroom implementations was conducted by UCLA CRESST in a set of case studies. Student responses to the simulation-based assessments were analyzed for their psychometric properties and technical quality. An additional field test compared the effects of use of the curriculum-embedded assessments as measured by an external posttest of AAAS calibrated multiple-choice items and a simulation-based benchmark assessment with classrooms not using the embedded assessments.

Results
Findings from the studies indicated that the simulation-based assessments were feasible for implementation across a wide variety of schools’ technical infrastructures, useful for understanding student learning progress and adjusting instruction, and of high technical quality (Quellmalz, Timms, Silberglitt, & Buckley, 2012).

Scientific or scholarly significance
The studies of the SimScientists assessments provide strong evidence of the quality and effectiveness of the simulation-based assessments and their suitability as credible components of a state science assessment system.

Authors

• Edys S. Quellmalz, WestEd

• Barbara C. Buckley, WestEd

• Mark Loveland, WestEd