Computational Analyses of Constructed Responses Reveal Levels of Comprehension (Poster 8)

• In Event: Leveraging Constructed Responses to Understand Text Comprehension

Abstract

1. Objectives. Comprehension theories differentiates between two aspects of a mental model, the textbase and the situation model (Kintsch 1988). The textbase refers to memory for explicit content that was conveyed in a text. The situation model refers to inferences to elaborate on the meaning of the text. The current study examined how linguistic features of CRs are predictive of the performance of comprehension assessment tapping into different levels of the mental model. Specifically, we examined how different linguistic characteristics of self-explanations uniquely contribute to the comprehension of the textbase and bridging inferences questions beyond individual differences.

2. Perspectives. This study is motivated by theories of discourse comprehension that specify the different aspects of a mental model (Kintsch, 1988). Researchers and assessment builders aim to develop comprehension items that assess the textbase and situation model as the models are assumed to reflect different levels of comprehension. Specifically, constructing the situation model is assumed to reflect a deeper level of comprehension than constructing a texbase (Kintsch, 1994).

3. Method. Participants read science texts, self-explained while reading, and answered comprehension questions that required leveraging content from their textbase (i.e., content from one sentence) or situation model representation (i.e., questions that required integration across sentences). We derived linguistic features using the Tool for the Automatic Analysis of Cohesion (TAACO; Crossley et al., 2016) and Tool for the Automatic Analysis of Lexical Sophistication (TAALES; Kyle & Crossley, 2015). We conduct a Principal Component Analysis PCA to extract the underlying patterns of linguistic dimensions reflected within the CRs that explain the variability in readers’ responses.

4. Data Sources. The data was collected from a study by Authors (2018) examining the effects of metacognitive prompts on learning in iSTART (Authors, 2006), an Intelligent Tutoring System in which students (n=116) learn how to write quality self-explanations while reading.

5. Results. Results of the PCA with Varimax rotation returned four components that collectively accounted for 78.81% of the variance in the linguistic features of the CRs. The four components that emerged are local cohesion, global cohesion, academic words, and lexical sophistication. Subsequent regression analyses demonstrated that local cohesion in CRs was predictive of surface comprehension beyond reading skills and science prior knowledge. However, only global cohesion was predictive of deep comprehension beyond individual differences. The overall model consisting of the principal components and individual difference measures significantly accounted for 38.1% of variance in textbase comprehension performance. Adding linguistic features significantly explained an additional 9% of variance in textbase comprehension. Finally, both local and global cohesion features of CRs were related to overall comprehension.

6. Significance. The degree to which the principal components depicting measures of global cohesion, local cohesion, and lexical sophistication within CRs, above and beyond individual difference measures, predicted overall comprehension, textbase comprehension, and bridging inference comprehension demonstrates that linguistic dimensions represent coherence of the mental representation or situation model. Our results confirm that cohesion and lexical features are significantly related to comprehension and meaning-making (Authors, 2021).

Authors

• Linh T. Huynh, Arizona State University
  Presenting Author

• Michelle Banawan, Arizona State University
  Non-Presenting Author

• Katerina Christhilf, Arizona State University
  Non-Presenting Author