Designing and Applying Scoring Rubrics for Automatically Scored Knowledge-in-Use Assessment Tasks for Instructional Decisions (Poster 4)

• In Event: Artificial Intelligence–Augmented Assessment Tools to Support Instruction in STEM

Abstract

Objectives
Knowledge-in-use requires individuals to develop the ability to apply knowledge to solve real-world problems in uncertain situations (Pellegrino & Hilton, 2012). Using knowledge-in-use assessments will allow teachers to adjust teaching and plan lessons to better support students’ 3D learning (Zhai, 2021). However, these assessments are usually performance-based, which needs evidence-based scoring rubrics to provide diagnostic information for teachers’ instructional decision-making. Second, designing rubrics is essential for developing algorisms to automatically score students’ responses (Krajcik, 2021; Zhai et al., 2021). This study has two purposes: 1) designing scoring rubrics for knowledge-in-use assessment tasks; and 2) examining experience science teachers’ perceptions regarding the usefulness of the rubrics for making instructional decisions.
Perspective
The knowledge-in-use assessment tasks and their rubrics are anchored in situated cognition theory (Greeno, Collins, & Resnick, 1996). Situated cognition emphasizes real-world situations that are meaningful for learners to make sense of phenomena and motivate them to figure out solutions for problems in new contexts actively. The theory of situated cognition informs the critical principles for designing rubrics for knowledge-in-use assessment tasks.
Methods
This study employs design-based research (Barab & Squire, 2004) to design the scoring rubrics for assessment tasks. We designed initial scoring rubrics and used the rubrics to train humans and developed machine learning algorisms to score students' responses. We used the information collected from experts, human scorers, and machine learning algorisms to refine scoring rubrics.
Data Sources
This study designed scoring rubrics for ten NGSS-designed knowledge-in-use assessment tasks, developed by the Next Generation of Science Assessment project (NGSA, 2022). Using such scoring rubrics, we presented the human-machine process for scoring students' responses and interpreting student performances. We received and analyzed six expert teachers’ feedback via semi-structured interviews on the usefulness of the rubrics to inform their instructional decisions.
Results
Our preliminary results show that the critical features of the scoring rubrics for knowledge-in-use assessments include meeting the task standard-based learning goals, integrating practices with science content, eliciting critical aspects of scientific practices, and creating scoring criteria for each single meaningful performance statement. The scoring rubrics informed teacher instructional decisions as follows. First, the scoring rubrics inform the interpretations of student performance by individual and grouped students. Second, students' performance can be categorized into four distinct groups: achieved the tasks, achieved the DCI aspect but needed support in SEP and CCC, achieved SEP and CCC but need support for the DCI, and not yet achieved. Third, the analysis of the six expert teachers’ interview data indicates the usability of the scoring rubrics to inform their instructional decisions to support students in the four groups.
Scholarly Significance
This study shows the critical features of scoring rubrics on complex construct-response assessment tasks. This study also presents the use of the scoring rubrics to inform the development of ML algorisms and inform teacher instructional decisions.

Authors

• Peng He, Michigan State University
  Presenting Author

• Namsoo Shin, Michigan State University
  Presenting Author

• Holly Kennedy Amerman, University of Georgia
  Presenting Author

• Xiaoming Zhai, University of Georgia
  Presenting Author

• Joseph S. Krajcik, Michigan State University
  Presenting Author