Topic Modeling: Exploring the Pitfalls of Automated Coding in Research and Evaluation Data Apps

• In Event: Marriott Roundtable Session 29
  In Roundtable Session: Capitalizing on the Advantages of Combining Methods to Enhance Educational Research

Abstract

This paper presents the results of a mixed methods study which examined the use of tools for the auto-coding of qualitative data. The study sought to better understand how researchers, who are increasingly using apps, use them effectively. MonkeyLearn services were integrated into Dedoose and used to critically explore the effectiveness of automated sentiment analysis, keyword, and entity extraction and subsequent auto-coding. This processing focused on 17,000 journal abstracts. Results framed the discussion which focused on what these technological features can offer the end user and where automated routines might be misused. Findings inform social science researchers regarding the challenges and pitfalls of these features and draw greater attention to the methodological implications and limitations of current technologies.

Authors

• Eli Lieber, University of California - Los Angeles

• Dan Kaczynski, University of Canberra

• Michelle Salmona, University of Canberra