Examining Self-Evaluations When Students Compose Written Responses Based on Multiple Texts

• In Event: 28.023 - 21st Century Calibration: Judging Performance in Digital and Classroom Contexts

Abstract

Objectives
To communicate about complex topics, learners today have to be able to compose written responses based on multiple, disparate texts (List & Alexander, in press; Rouet, 2006). Yet, students have been found to struggle to do so, perhaps, in part, because they do not know the criteria for what makes for a quality, integrated written response, composed based on multiple texts. In this study, we investigated this possibility and the criteria that students use to self-evaluate the quality of their writing. We further examine the association between students’ self-evaluations and actual task performance.

Theoretical Framework
We are guided by the Multiple Documents Task-Based Relevance Assessment and Content Extraction (MD-TRACE) model (Britt & Rouet, 2012; Rouet, 2006). The MD-TRACE suggests that the process of multiple text use unfolds through a series of five steps, concluding with learners’ evaluations of the quality of their written responses in Step 5. Although under-examined in the literature, these self-evaluations are critical as they dictate whether students consider a particular multiple text task to be resolved or elect to re-engage in earlier steps of multiple text use (e.g., revisit previously accessed texts). Students’ self-evaluations are investigated in this study. Drawing on the calibration literature, we consider both the accuracy of students’ subjective self-evaluations and their basis.

Methods
Students (N=143) were first were asked to complete a multiple text task, or to research and compose a written response using a library of digital texts. Then, students were asked to assign a grade to their written responses as well as to justify their self-evaluations. Students’ written responses were coded for the volume of arguments they included, as well as their degree of integration and elaboration.
Calibration was computed as the difference between students’ self-determined letter grades and the scores assigned to their written responses. Students’ justifications for self-evaluation were also coded for.

Results
A significant negative correlation between calibration (i.e., the difference between students’ self-assigned grades and objective scores) and writing performance was identified [r(137= -0.77, p< 0.001]. This reflects better calibrated students (i.e., with smaller difference between their self-evaluations and objective scores), performing better on multiple text writing tasks.
Differences in performance and calibration were found to be associated with students’ citing different criteria for their self-evaluations of response quality. A model predicting response quality based on students’ consideration of deep-level source use and response structure in justifying self-evaluations was significant, F(5, 135)=4.74, p=0.001, R2adj=0.12 (see Table 2). Likewise, a model using these same self-evaluation justification criteria to predict calibration accuracy was also significant, F(6,132)=3.17, p=0.006, R2adj=0.09 (see Table 1).

Scholarly Significance
We contribute to the literature on multiple text use by examining students’ self-evaluations of written response quality. We contribute to the literature on calibration by examining students’ self-evaluations within the context of a complex task (i.e., writing based on multiple texts) and by considering the criteria that students use to justify their self-evaluations. The criteria that students use to self-evaluate response quality were found to be associated with both calibration and task performance.

Authors

• Ying Wang, Georgia Institute of Technology

• Alexandra List, The Pennsylvania State University