Exploring Automated Scoring of Student-Generated Model in Science Assessment

• In Event: 63.057 - Automated Assessments in Science Inquiry

Abstract

Assessment experts have noted that new reforms in science education require innovative assessments to probe multiple dimensions of scientific knowledge such as core ideas and science practices (Pellegrino et al., 2014). The Next Generation Science Standards (NGSS) explicitly identifies modeling as one central and valued practice. The visual models constructed by students can serve as rich vehicles of information for educators interested in supporting and assessing what students know and can do in science. A few researchers have used students’ hand-written model drawings as a rich source of evidence to explore what students know about the structure and behavior of matter (Liu, Rogat, & Bertling, 2013; Merritt, 2010). However, there are still many challenges to score student-generated models at scale as human scoring is expensive and has potential reliability issues. In our study, we explored possible automated scoring techniques to score object-based drawings generated by students.
The student-generated models were collected through a pilot study in two science classroom settings. In both classroom settings, teachers used the prototype assessment task to help students learn about the core idea of Matter. In this prototype, modeling items involve the use of a computer-based drawing tool in which students use a virtual pen or select from a pool of predefined objects, including abstract objects (e.g., dots or squares) and concrete representations (e.g., fish, water drops), to allow students to express their idea of structure of matter. The drawing tool also allows students to change the size or color of selected objects, add arrows to represent motion, and label objects. In total, we collected about 500 student drawings of particle models of matter.
We developed a scoring rubric to human score student models of matter. The scoring scheme includes four dimensions, namely scale, material identity, behaviors, and distribution of particles that are mapped with the Matter LP (see Table 3). The scale dimension provides information about students' understanding of the smallest unit that composes matter (e.g., macroscopic, microscopic, or nanoscopic). The material identity dimension looks at how many identities students include in their drawings for pure water and ocean water models. The behaviors dimension is related to if and how students represent the particle behaviors in their drawings. The distribution of particles dimension examines how students represent the positions of individual particles and space between them in liquid matter state.
We deployed two steps key to most automated scoring approaches: (1) extracting meaningful features from students’ models, and (2) learning statistics models to represent the latent relationship between the features and human-rated scores. Table 4 summarizes the marginal correlations of individual features with human scores. The results show that the macro-object types feature was negatively correlated with the understanding levels. In addition, the number of arrows and the randomness feature almost exclusively account for student understanding of the particle behaviors when compared to other features. These preliminary findings are promising in that it is possible to build scoring models that are interpretable. Future work will focus on validating the constructed models.

Authors

• Lei Liu, Educational Testing Service

• Chee Wee Leong

• Rutuja Ubale

• Lei Chen, Educational Testing Service