Poster #48 - Visualizing and coding qualitative data with AI language models

• In Event: Poster Session 05

Abstract

Qualitative methods are critical for investigating developmental processes within contexts (Lerner, 2013). Particularly, exemplar methodology involves studying individuals with high levels of developmental phenomena of interest (King et al., 2014). Such qualitative approaches, though, have known limitations. Foremost is the time required to code and analyze qualitative data. Cutting-edge advancements in machine learning have untapped potential to make these processes efficient and effective, including impressive data visualization. Novel psychological constructs provide ample space for such qualitative and visual exploration. Thus, the purpose of this study was to demonstrate the utility of language models for examining a novel construct in the psychology of religion, indebtedness to God. Indebtedness to God is related to yet distinct from gratitude to God (Nelson et al., 2022). We conducted exemplar research to better understand indebtedness to God and analyzed the data using the GPT-3 language model.

The dataset consisted of 574 paragraphs of free-form text from interviews with religious exemplars. The religious exemplars (N=15) were recruited from a sample of young adults at a religious university (N=1099). Exemplars scored in the top 5% of a composite religiousness variable. The semi-structured interviews included open-ended questions regarding exemplars’ experiences surrounding gratitude and indebtedness to God, among other things.

The data were human coded and then coded using the language model. Human coding discovered five themes and six outcomes relating to indebtedness to God. Next, to compare the human themes with language model themes, we used GPT-3 to generate short themes for each paragraph of text, called “tags”, which we then used to construct visualizations. Several high frequency tags were “gratitude”, “love”, “family”, “faith”, and “forgiveness”. We also examined co-occurring tags with our primary topics of interest, including gratitude and indebtedness. These analyses provided evidence for the breadth and depth of the data. Then, we created a semantic plot of each sentence in the dataset, clustered by semantic similarity. We used the sentence-transformer package with the sentence-t5-xxl model to create a multidimensional feature vector for the semantic content of each sentence. Next, using PCA, we extracted the top 25 dimensions and employed the U-Map projection algorithm to create a descriptive sentence cluster. Refer to Figure 1 for the results.

These data visualizations allowed us to see themes not readily observable by the human coding process. Namely, the semantic clustering revealed a cyclic growth process that human coders did not detect, as well as new themes. Because clusters were organized by semantic correlation, the cluster’s topographical distance presents an ordering to the concepts, allowing researchers to infer a connected process within the lived experiences described. Further, by analyzing the qualitative religious exemplar interviews using the language model, we generated clear, precise, and comprehensible visualizations to aid the discovery of new insights into a complex developmental phenomenon. In short, machine learning can enhance qualitative research by 1) providing a more time efficient approach to coding and analysis, 2) providing sophisticated visual exploration options, and 3) enabling greater breadth and depth of analysis of qualitative data.

Authors

• Russ Amelang, Brigham Young University
  Presenting Author

• Jenae M Nelson, Baylor University

• David Wingate, Brigham Young University

• Sam A Hardy, Brigham Young University