A Rubric-Based Approach for Analyzing Dimensions of Literary Reasoning

• In Event: Expanding Pedagogical Understandings of Literary Reasoning and Argumentation in Middle and High Schools: Project READI

Abstract

Purpose
Teaching literature engages students in reasoning about interpretations. Developing reliable assessments of literary reasoning is critical to this goal. This paper reports a rubric-based approach to capture how readers reason about literature (Lee, 2011). Prior research on literary reasoning, including studies on expert-novice differences (Graves & Frederickson, 1991), cognitive models (Kintsch, 1998), literary theory (Leitch et. al, 2010), and argument (Hillocks, 2011; Toulmin, 1984) informed an iterative design process that produced rubrics for six dimensions of literary reasoning relevant to symbolic and thematic interpretation.

Rubric score points were defined for the following dimensions:
Claims: Accuracy, number and connectedness of claims about a literary work in terms of what is supportable in the text (0 – 4).

Evidence: Reasons that support claims, connections among reasons, and evaluation of reasons (0 – 4).

Reasoning: Warrants (justifications) used to connect claims and reasons (0 – 2).

Symbolism: Identification of symbols and interpretations in context of story worlds and/or real life (0 – 3).

Theme: Identification and discussion of coming of age, connection to symbolism in story (0 – 3)

Synthesis: Comparison and contrast between stories (0 – 3).

Detailed criteria for each dimension reflected a breadth of literary norms.

Methods
Data Sources. We obtained essays from adolescent students in grades 6 (N = 83), 9 (N = 25), and 12 (N = 46) attending schools in a Midwestern urban area. Students read and then wrote essays about two short stories that featured symbolism and expressed “coming of age” themes. The essay prompt asked them to “discuss the images in the stories and how they helped you understand the characters and the worlds they live in.”
To establish criteria for the rubric score points on each dimension, we selected a random sample of 30 essays, stratified by grade, but then blinded for grade. When reliability greater than .8 in applying these codes was achieved, the remaining essays were scored (grade-blinded) by two independent coders who achieved an intraclass correlation of .82 across all dimensions, indicating high inter-rater reliability (Shrout & Fleiss, 1979).

Findings
Chi square analyses showed that the distributions of score points on each dimension were significantly related to grade level, ps < .001. Most scores on each dimension tended to cluster around the lower ends of the rubrics, although some of the 12th graders achieved rubric score points of 3 or 4. Nevertheless, for each grade level there was a range in performance across at least 3 score points for claims, evidence, and symbolism, and across 2 for reasoning, theme, and synthesis. The findings indicate that the rubrics for these dimensions are capturing expected grade-differences in systematic ways as well as variability within grades, and constitute a conceptually-based and reliable means of assessing literary reasoning.

Significance
This rubric-based approach to assessing literary reasoning captures dimensions important to interpretative arguments about literary texts. Initial work with classroom teachers indicates that the dimensions and rubrics provide a useful tool to diagnose students’ strengths and challenges in literary reasoning.

Authors

• Stephen W Briner, University of Illinois at Chicago

• Kathryn S. McCarthy, University of Illinois at Chicago

• Candice Burkett, University of Illinois at Chicago

• Sarah Levine, National Louis University

• Carol D. Lee, Northwestern University

• Susan R. Goldman, University of Illinois at Chicago

• Joe Magliano, Northern Illinois University