Using Classroom-Based Assessment Scenarios to Design Score Reports That Support Science Teaching and Learning (Poster 6)

• In Event: Artificial Intelligence–Augmented Assessment Tools to Support Instruction in STEM

Abstract

Purpose
Assessment score reports are common features of teachers’ lives, but few teachers report confidence in making instructional decisions from the plethora of score reports provided to them (Zapata-Rivera et al., 2019). Increasingly, technology-based assessments include automatic score reports (i.e., AutoRs). To develop AutoRs that are usable and meaningful for instructional decision-making, we need usability information regarding how teachers use score reports. This poster illustrates how classroom-based assessment scenarios can be used to elicit this information from teachers and refine score reports to better support teachers’ decision-making.
Theoretical Framework
Scenarios are short vignettes about a classroom that lead to a prompt about teacher decision-making (Cobern et al., 2014; Furtak et al., 2016). According to Hutner & Markham (2017) and Magnusson et al. (2006), scenarios provide details that teachers would typically already know about their own classrooms and information about students’ proficiency relative to a learning goal. In the case of assessment, scenarios can provide a relevant real-world context for teachers to reflect and think aloud about the instructional decisions they make based on student data (Datnow & Hubbard, 2015). Teachers’ responses to scenarios are of high value because they can inform the design of assessment tools, such as score reports, to increase their usability and effectiveness for teachers.
Data Sources and Methods
We developed three types of classroom-based assessment scenarios that each describe a specific instructional situation where a teacher was motivated to assess students. Each scenario includes a three-dimensional learning goal, a contextualized classroom occurrence that gives rise to enacting formative assessment, and a characterization of student proficiency in the AutoRs. A semi-structured protocol was developed to accompany the scenarios. A panel of seven expert science educators from the middle grades with deep knowledge of three-dimensional instruction were presented with a scenario to elicit their feedback and perspectives on the scenario + AutoR and what instructional next steps they would take to support learning based on the scenario and the AutoR. The scenarios were also used to invite reflection of teachers’ formative assessment practices.
Results
Our initial use of the assessment scenarios confirmed the authenticity of these scenarios to teachers’ experiences in the classroom. We found that the expert teachers were able to use the scenarios to reflect on how they would use the information from AutoRs to make next-step instructional decisions. Moreover, we found that the scenario prompts effectively oriented teachers toward reflecting on using assessment data for supporting learning, rather than toward grading alone. The full paper and poster will further expand on our use of scenarios with science teachers to inform the design of actionable score reports.
Scholarly Significance
Teachers are increasingly faced with using score reports. As technology-based assessment systems become more sophisticated in aggregating and displaying data, there is an opportunity to build into these systems “just-in-time” guidance for using assessment information formatively to improve teaching and learning. Thoughtfully crafted scenarios can help us better understand how score reports can be designed to be instructionally supportive for teachers and students.

Authors

• David McKinney, WestEd
  Presenting Author

• Christopher J. Harris, WestEd
  Presenting Author

• Lauren Stoll
  Presenting Author

• Lindsay Tanner, WestEd
  Non-Presenting Author

• Katherine Joy Nilsen, WestEd
  Presenting Author

• Jessica Triant, WestEd
  Presenting Author

• Gary Weiser, Bill & Melinda Gates Foundation
  Presenting Author