Validating Writing Traits for Classroom Assessment Purposes

• In Event: Automated Writing Trait Analysis to Support Instruction

Abstract

This paper presents a study in which 10 teachers anonymously evaluated Criterion classroom essays written by 33 students. We asked teachers to score each essay on a 6-point rubric, to provide written feedback to each student with advice focused on improving their essays, to assign students to groups for follow-up instruction, and to indicate their intended next steps for the whole class.
We conducted the following quantitative and qualitative analyses on the feedback data from teachers to inform the report mockups:

• We examined the relationship between trait scores and teacher scores by constructing a linear discriminant model. This model resulted in two significant dimensions: with 73.9% of the score variance being accounted for by the first dimension, which loaded on traits related to text macrostructure (e.g., Organization and Text Cohesion, which are strongly associated with essay length), and 22.9% of the variance being accounted for by a second dimension, which mostly loaded on vocabulary-related traits (Vocabulary Length, Vocabulary Difficulty, and Formality). Further analysis showed that students with a rounded mean score of 2 were differentiated from students at higher scores on the first dimension, whereas students slightly higher rounded mean score of 3 were differentiated from students at higher scores on the second dimension. This meant that students who scored a 3 had reasonably elaborated essays, compared to students who scored a 2, but with very simple language when compared to students at score points of 4 and 5.
• We examined the relationship between teacher scores and teacher grouping decisions by conducting a correspondence analysis. More than 80% of the variance in mean teacher scores was accounted for by the first two dimensions of the correspondence analysis, which were correlated with essay length and sentence length. This means that teachers tended to create homogeneous subgroups of students.
• We conducted a qualitative analysis of the teacher feedback to identify themes in their whole-group, small-group, and individual student feedback. We found that teachers were most concerned with the structure of the student essays, and their use of academic language and word choice, which matched the major dimensions that distinguished low and high performing students on our automated trait analysis. Interestingly, teacher feedback seemed to be dependent on student’s overall performance and their score, with different traits coming to the fore as student performance improved. For example, teachers tended to reserve advice about addressing opposing viewpoints for students who had already developed their arguments with reasons and examples.
• We then designed mock reports of an automated system that could provide teachers' feedback along the lines identified in our quantitative and qualitative analyses.

By the time we present, we will have met with the teachers and discussed these report mockups illustrating ways that writing traits can be used to organize student reports, and will be able to indicate whether teachers found such a trait-based method of organizing information about student writing to be helpful.

Authors

• Priya Kannan, WestEd

• Geoffrey C. Phelps, Educational Testing Service

• Paul D. Deane, Educational Testing Service