Teacher Interpretation of Artificial Intelligence–Based Assessment Reports (Poster 1)

• In Event: Artificial Intelligence–Augmented Assessment Tools to Support Instruction in STEM

Abstract

Objectives
Artificial intelligence (AI) has the potential to enhance assessment practices by targeting complex constructs, augmenting the evidentiary reasoning process, and easing teacher workload (Zhai et al., 2020). However, challenges arise when using AI-based assessments, as teachers have limited experience using complex assessment results (Zhai, 2021). We know little about how teachers attend to, interpret, and use assessment results to make instructional decisions. We employ cognitive labs to uncover how teachers interpret AI-generated assessment reports (AutoRs). We explore two questions: (a) How do teachers interpret AutoR? (b) How do teachers use AutoR information to make instructional decisions?
Theoretical Framework
This study adopts the goal-oriented theory that accounts for teachers' decision-making (Schoenfeld, 2010) as teachers (a) enter into a context (i.e., teaching scenario) with specific goals, resources, and orientation; (b) orient to the situation (i.e., read student reports, notice students' 3D performance, interpret students' learning), (c) obtain information (from student reports) and draw on knowledge (e.g., PCK), (d) refresh goals, and (e) make decisions consistent with the goals regarding which directions to pursue and what sources to use.
Methods
Participants. This study recruited three middle school teachers with more than five years of teaching experience. Teacher A is a black male; Teachers B and C are white females. They taught in highly diverse schools, with more than 80% of students identifying as African American or Spanish and with all students qualified to receive reduced/free lunch.
Procedures. We surveyed and interviewed teachers' everyday assessment practices and employed think-aloud interviews to examine how they approached three constructed response items. We randomly selected 20 student responses to science assessments and presented three types of information on mobile dashboards: a) students’ response time, b) individual automatic scores, and c) grouping information. We asked teachers to talk aloud when interpreting data and making instructional decisions and recorded the screens. The data collection lasted up to six hours for each teacher.
Analysis and Results
We used an inductive data analysis approach (Stern, 1980). Screen-recorded video and audio data were transcribed and analyzed along with teachers' decisions, surveys, and interviews. Preliminary results suggest that Teachers A used assessments "as learning activities" while Teachers B and C used assessments "for evaluation." Although Teachers A and B used assessments for different purposes, they used similar interpretation procedures. They focused on "connections" among the three types of information in the AutoRs. The "connections" enabled them to gain an understanding of each student. In contrast, Teacher C used the information individually and gained superficial knowledge. We found that teachers' instructional decisions relate to their context and backgrounds, with significant differences between Teacher C and Teachers A and B. The paper will present the time curves of each teacher's cognitive activities and detail how teachers utilize the contextualized information to interpret and use scores.
Scholarly Significance
This study contributes to the field by uncovering teachers' interpretation and decision-making cognitive processes using AutoRs. The findings will guide the design of AutoRs, dashboards, and instructional scaffolds to support assessment practices.

Authors

• Xiaoming Zhai, University of Georgia
  Presenting Author

• Xinyu He, University of Georgia
  Presenting Author

• Ehsan Latif, University of Georgia
  Presenting Author

• Peng He, Michigan State University
  Presenting Author

• Joseph S. Krajcik, Michigan State University
  Presenting Author

• Yue Yin, University of illinois at Chicago
  Presenting Author

• Christopher J. Harris, WestEd
  Presenting Author