Automated Collaboration Assessment Using Behavioral Analytics (Poster 2)

• In Event: Artificial Intelligence and the Future of STEM Instruction: Designing New Models to Automate Feedback to Teachers

Abstract

The Automated Collaboration Assessment Using Behavioral Analytics project measures and supports collaboration as students engage in science-based learning activities. Collaboration clarifies misconceptions and promotes deeper understanding of concepts in STEM which helps prepare students for future employment in STEM fields. In many K-12 classrooms, teachers use instructional approaches to promote and inculcate collaborative skills, such as project-based learning (Krajcik & Blumenfeld, 2006) or problem-based learning (Davidson & Major, 2014) to monitor developing student proficiencies in science.
Collaboration is an important learning skill in K-12 STEM education, yet teachers have few consistent ways to measure and support students’ development in this area. This project furthers our understanding of productive collaboration as we develop an automated instructional tool that can help teachers identify nonverbal behaviors and assess overall collaboration and engagement quality. We focus on behavior, instead of discourse and content-based dialogue, because teachers rarely have the time or the resources to commit to monitoring student groups and provide accurate and differentiated feedback on their collaborative skill development. Rather, teachers often briefly observe students to make assessments of progress.
We designed a preliminary automated collaboration assessment tool based on our collaboration conceptual model (Alozie et al., 2020a, 2020b). Our rigorous stratified, hierarchical, multi-level collaboration framework describes how well students are collaborating individually and in groups. Through our collaboration conceptual model, we created a rubric for human annotation using behavioral cues that can identify individual and group contributions to the overall collaboration in face-to-face and online settings. We used behavior-based learning analytics to train machine learning models to evaluate collaborative interactions at various levels (Som et al., 2020, 2021). The input to our machine learning models consisted of video (without sound) or audio+video (with sound) data recordings of student groups performing a collaborative task. We compared audio and audio+video data to test if visual behaviors alone could be used to estimate collaboration skills and quality. Our machine learning models showed 80-85% F1-Score performance when compared to human annotations, showing a high level of precision and recall. We also designed three kinds of machine learning systems that offer different levels of detail, such as important instances during the task and important student interaction that influenced or contributed the most towards the machine learning model’s decision. These sophisticated machine learning systems were used to design our recommendation system. Our recommendation system can provide individual student level or group level recommendations that promote better overall collaboration. The stratified nature of the collaboration conceptual model created opportunities for teachers to give students differentiated feedback.
This project addresses a time-sensitive and critical need for vetted classroom-based technological instructional tools that can assess and give differentiated feedback on important collaboration skills in science. This study is designed to increase knowledge by creating an automated tracking system that can identify behaviors that contribute to collaboration and provide differentiated information about individual and group level behaviors. Our work provides new ways to collect data on classroom interactions, providing a new understanding of collaboration and other interpersonal interactions in learning spaces.

Authors

• Nonye M. Alozie, SRI International

• Anirudh Som