Artificial Intelligence for Formative Assessment: A Systematic Review (Poster 9)

• In Event: AI and Formative Assessment: The Train Has Left the Station

Abstract

Objectives
AI is increasingly applied in supporting formative assessment practices to facilitate customized learning. Various AI-based applications have been developed for formative assessment (AI4FA) in diverse disciplines. For instance, c-raterML provides revision suggestions after students submit their written responses to science questions (Gerard & Linn, 2022). Systematically reviewing the applications of AI4FA is critical to identify the status of research and future directions. This study addressed two questions.
• What AI applications have been used for formative assessment in educational settings?
• How have the AI4FAs been evaluated (e.g., reliability, validity, and effectiveness)?

Theoretical Framework
This study adopted a framework specifically for innovative assessment practices, including four components (Zhai, 2021): (a) Identifying Learning Goals. AI4FA helps to achieve the most challenging learning goals, such as practice-based science learning. (b) Eliciting Performance. With AI, formative assessment can engage students in complex performance and evaluate them immediately. (c) Interpreting Observations. AI may facilitate assessment interpretation by providing structured and immediate scores to teachers. (d) Decision-Making and Instructions. Contingent on the assessment's purpose, decisions based on assessment outcomes can be substantially disparate.

Method
Data Source. We used ProQuest Database (including APA PsycINFO and ERIC database), ScienceDirect, and IEEE Xplore to search AI for formative assessment literature in the last five years. After the search and filtering, 15 empirical studies that applied AI4FAs were included.
Data Analysis. The authors developed the coding rubrics and practiced coding training together till the rubrics were finalized. Two authors coded four randomly selected articles and reached an interrater agreement over 0.90. The first author coded the rest of the articles independently. We conducted descriptive analyses of the codes to answer the RQs.
Results
RQ1. the majority of studies (60%) were conducted in postsecondary educational settings, while K-12 educational environments accounted for approximately one-third (33%) of the studies. One study examined MOOC providing feedback to both high school AP course takers and college students. More studies were conducted in formal (60%) than in informal (33%) education settings, reflecting the expanding role of AI in traditional learning environments. Lastly, the AI4FAs were used in a wide scope of subjects, including STEM, English as a second language, history, and physical education.
RQ2. The review shows that AI4FAs need a more robust evaluation. Only three studies offered a comprehensive examination, including the reliability, accuracy, and effectiveness of the AI4FAs. Four studies sought to validate the accuracy of the results by comparing AI4FAs scores against human experts. Besides, five studies validated the accuracy of ML scores with other existing models. Lastly, eight studies explored the effectiveness of AI4FAs using mixed methods, such as interviews and surveys.

Significance
This study summarizes what AI applications have been implemented to facilitate formative assessments and identifies the gaps in this promising function of AI. (a) practitioners can locate AI applications for teaching. (b) technical experts can create AI applications for underdeveloped fields, such as science. (c) researchers can further develop the full potential of AI applications to facilitate formative assessment.

Authors

• Ying Chen, University of Illinois System

• Ruiping Huang, University of Illinois at Chicago

• Yue Yin, University of illinois at Chicago

• Xiaoming Zhai, University of Georgia