Innovative Approaches to Assessing Multiple Source Use via Expressive Tasks

• In Event: Synergy of Reading and Writing Within Multiple Source Tasks: Novel Approaches and Innovative Techniques

Abstract

Objectives
The forthcoming NAEP Reading Framework (NAGB, 2020) expands notions of reading assessment to involve reading with peers to solve problems and reading to use and apply information from multiple sources. Broader K-12 standards also expect students to understand, evaluate, and use multiple sources while reading, writing, and speaking (CCSSO & NGA, 2010). We report findings from a large research project intended to inform the design of collaborative multiple source inquiry tasks called for in the 2025 NAEP assessment by exploring a novel scenario-based task (SBT) integrating reading, writing, and speaking.

Theoretical Framework
This project builds on work in multiple document comprehension (Braasch et al., 2018) and new literacies of online reading comprehension (Leu et al., 2013) to emphasize using and communicating information as a product reading across multiple digital sources (Coiro et al., 2018; Sparks & Deane, 2015). Studies of multiple text integration (MTI) frequently use expressive tasks (e.g., source-based essays) to measure this construct (Primor & Katzir, 2018), but rarely include spoken expression (e.g., Wolfe & Goldman, 2005). This study explored how students communicate and work collaboratively on an oral presentation task, an expressive measure of MTI within an SBT (Coiro et al., 2019).

Method
The SBT included Setup (planning), Free Roam (locating, evaluating), and Conclusion (MTI) phases. The oral presentation task asked ninth and tenth graders to jointly construct a written plan (6 points) specifying their overall conclusion and identifying key sources and relevant supporting evidence; students then delivered the presentation aloud. The ninth and tenth graders completed the SBT in either face-to-face (F2F) or remote modes. F2F pairs (n = 5) jointly worked around a single computer to complete an individual SBT variant. Remote pairs (n = 16) worked remotely on a collaboration-enabled SBT via a digital platform with embedded audio/video and chat capabilities surrounding the task (Hao et al., 2017). Dialogue was transcribed and evaluated qualitatively; item responses received numerical scores.

Results
Nonparametric correlations revealed that presentation task completion (ρ=.53, p<.05) and performance (among 12 dyads responding; ρ=.78, p<.001) were related to Conclusion scores. Interestingly, dyads’ Free Roam evaluations of source relevance (ρ=.44, p<.05) and final position (ρ=.59, p<.01) were related to presentation plan scores (Table 2).
In both F2F (n=5) and Remote (n=6) playtesting sessions, deeper qualitative analyses of task processes provided insights into stronger and weaker collaboration (Coiro et al., 2019; Sparks et al., 2019). Stronger pairs engaged in multiple rounds of negotiating, clarifying, monitoring and elaborating their ideas. In contrast, exchanges among weaker pairs indicated that they engaged in simple agreement/disagreement with minimal discussion.

Significance
Multiple kinds of evidence of students’ literacy skills collected in this SBT yielded an alternate picture of proficiency that permits investigation of meaningful relations among the collaborative processes and final expressive communication products that characterize engagement with inherently collaborative digital spaces (Verbeek-Cowart, 2016). This work raises challenges but also contributes to understanding about the development of operational assessment tasks designed to measure multiple source use and its receptive and expressive correlates in multimodal digital environments.

Authors

• Julie Coiro, University of Rhode Island

• Jesse R. Sparks, Educational Testing Service