Analytic Methods for Analyzing Expert Coded and Computationally Processed Data From Constructed Responses (Poster 3)

• In Event: Leveraging Constructed Responses to Understand Text Comprehension

Abstract

1. Objectives. Our research adopts an approach to using CRs that involves a quantification of the data extracted from them (Chi et al., 2004). This approach necessitates the use of data analytic approaches that are applied in order to make inferences about comprehension strategies, individual differences, and the nature of comprehension. These approaches range from linear modeling (e.g., Authors, 2020), machine learning (e.g., Feller et al., in-prep), and dynamic systems analyses (e.g., Likens et al., 2018). These different approaches tend to be applied differently to expert coded and computationally analyzed data. For example, linear modeling is typically applied to expert coding, whereas machine learning and dynamical systems analyses tend to be applied to computationally analyzed data. The purpose of this poster is to provide a primer on these different analytic methods used to analyze CRs.

2. Perspectives. Each of these analytical methods (linear modeling, machine learning, and dynamic systems analysis) predicts readers’ comprehension under a different lens. More common inferential statistics are used to assess the linear relationships between expert coded holistic and subscale scores and various individual differences or features of the text. However, the machine learning of computationally processed texts allows researchers to consider language as multidimensional (Öncel et al., 2021) and, in many instances, nonlinear (Elman, 1995). Dynamic systems analyses are used to examine the nonlinear patterns of coherence-building processes over time and view comprehension as a ‘system’ in which many factors (characteristics of the reader and the text) interact to influence it.

3. Method. Linear modeling (linear regressions) can be used to look at the linear relationships between comprehension scores and individual differences. Machine learning is often performed to examine the more complex/nonlinear relationships between comprehension scores and linguistic features of the text. Recurrence quantification analyses (RQA) can provide dynamic indices related to recurring linguistic patterns found in the CRs to reveal information about their comprehension process and how they unfold over time.

4. Data Sources. The wide range of analytical methods used applies to the analyses and results of all other projects/posters presented. Individuals’ CRs, the reading source texts, NLP spreadsheets, expert scores, and individual differences are used.

5. Results. Examining CRs using these various approaches allows for more factors to be included and lends itself easily to a range of comprehension theories.

6. Significance. Overall, these analytical methods give insights into how classroom and automated interventions can be more precisely attuned to the different interactions between the student (individual differences), task (instructions), and text (genre) as they dynamically change across time and relate to the linguistic features and strategic processes used during comprehension.

Authors

• Lauren Flynn, University of Minnesota
  Presenting Author

• Danielle S. McNamara, Arizona State University
  Presenting Author

• Laura K. Allen, University of Minnesota
  Non-Presenting Author