Outlining a Qualitative Methodological Approach to Studying Medical AI Chatbots

• In Event: 20220 - Section on Communication, Information Technology, and Media Sociology Roundtables
  In Refereed Roundtable (60 minute): Section on Communication, Information Technology, and Media Sociology Roundtable I: Artificial Intelligence I

Abstract

While digital ethnographic approaches are well-suited to studying online spaces as diverse as forums, social media platforms, and search results (Forberg and Schilt 2023), relatively little methodological work is tailored to chatbots specifically. There has been limited, but timely work in developing qualitative methods to study generalist AI models like ChatGPT. However, the particular affordances of medical AI chatbots, particularly “AI doctors” marketed for use by laypeople, require a retooling of these approaches. This is because these “AI doctors” seek to emulate (more or less closely) a particular kind of conversation: the traditional offline doctor-patient consultation, and, as such, are both coded and approached by the user with an explicit and clear primary purpose in mind: to record a user’s symptoms, interpret them, and present the user with a list of potential diagnoses. I argue that a qualitative approach to studying medical AI chatbots must be attuned to the particular boundaries and constraints of the medical knowledge, authority, and conversation and consider how both the chatbot itself and the online, text-only format of the conversation may be either reinscribing or pushing against those norms.
In this paper, I review existing qualitative and digital ethnographic methods for analyzing generative AI chatbots, outline a framework tailored to medical chatbots specifically, and suggest future directions for qualitative methods for analyzing generative AI chatbots across various purposes and applications. I propose employing a two-pronged qualitative/digital ethnographic framework for studying medical chatbots that emphasizes the particular technological and medical aspects of medical AI chatbots. I demonstrate a brief application of this approach by presenting an excerpt of a conversation with CodyMD, a consumer-facing “AI doctor” chatbot.

Author

• Caroline Petronis, University of California San Diego