Sourcing Strategy Use in Constructed Responses Within Multiple Document–Integrated Reading and Writing Tasks (Poster 9)

• In Event: Leveraging Constructed Responses to Understand Text Comprehension

Abstract

1. Objectives. While much of the research that will be reported in this structured abstract focuses on comprehension strategies, the research team has also focused on other strategies in the context of processing multiple documents (MD). One such strategy involves sourcing, which involves strategies focused on who wrote a document, their credentials, and where a document was published (Braasch, et al., 2013; Bråten & Braasch, 2017). In this study, participants produced CRs while reading MDs and then wrote argumentative essays and prompt-based CRs. The CRs were coded by experts to determine the presence of source strategies and the essays were scored. Computational approaches were used to detect sourcing strategy use. This study investigates the practical application and theoretical implications of the automated detection of sourcing strategies within CRs in MD tasks.

2. Perspectives. Within MD integrated reading and writing tasks, one key strategy is sourcing, or identifying and maintaining source information (Strømsø & Bråten, 2014). Thus, researchers often look for evidence of sourcing in students’ reading behaviors. One way of capturing such processing behaviors is through CR. However, evaluating CRs for the presence of different strategies can be both challenging and time-consuming, most especially within MD reading and writing tasks.

3. Method. While there is a developing body of work examining computational approaches to identifying and evaluating sourcing in post-reading essays, there is little work exploring computational approaches to identify sourcing strategies in readers’ concurrent CRs . This study addresses the gap by leveraging natural language processing (NLP), machine learning (ML), and state-of-the art Explainable-AI techniques to explore the language dimensions that predict the presence of sourcing strategies.

4. Data Sources. Participants (n = 86) produced CRs in the context of a multiple document integrated reading and writing task (Authors, 2021). Expert raters coded the 1,296 CRs for evidence of source presence and source evaluation and achieved good reliability (weighted kappa = 0.71).

5. Results. Random Forest classifiers for source presence and source evaluation were trained, tested and cross-validated using three repeats of 10-fold cross-validation (Presence: F1-score =0.93; Evaluation: F1-score = 0.96). The strongest predictors of sourcing were categorized into coarse-grained dimensions and revealed that sourcing is marked by differences in lexical diversity and sophistication, academic language use, syntactic complexity, and lexical semantics. Further probing into the predictive quality of the actual words (bag-of-words) used in the CRs confirmed our earlier results and strengthened the findings on lexical semantics as discriminatory of MD CRs that used sourcing strategies. When students write their CRs, the choice of words is tied to maintaining and integrating word meanings from the sources.

6. Significance. We found that sourcing is scarcely used across CRs but, when present, sourcing has predictive linguistic features that reflect consideration of the source texts and integration across texts. Our findings suggest that CRs that include sourcing strategies can be efficiently determined by word-level linguistic dimensions of lexical diversity, academic language use, syntactic complexity, and lexical semantics that reflect the complexity of MD CR writing tasks.

Authors

• Michelle Banawan, Arizona State University
  Presenting Author

• Kathryn McCarthy, Georgia State University
  Non-Presenting Author