AI-Based Evaluation of Intervention Effects in Single-Case Designs: A Study of Claude’s Analytical Capacity

• In Event: Innovative Statistical Methods and Computational Tools for Complex Research Designs

Abstract

This study evaluated the analytical capabilities of Claude large language model in interpreting AB graphs. Using 80 simulated graphs with known parameters, Claude was assessed on its ability to identify data characteristics and detect intervention effects. The results showed that Claude demonstrated high accuracy in identifying phase lengths, trends, and true effects, with at least 85% accuracy. However, performance declined under conditions of the presence of trend and high data variability. Claude’s Type I error rate was .33, while statistical power was .93. Results from chi-square analyses confirmed data complexity influenced accuracy. The study provides insights into the opportunities and challenges in using AI to analyze single-case design graphs, contributing to the broader fields of using AI for graph analysis.

Authors

• Isaac Akuoko-Mensah, University of Nevada - Reno

• Li-Ting Chen, University of Nevada - Reno