Toward an Integrated Writing Trait Model

• In Event: Automated Writing Trait Analysis to Support Instruction

Abstract

There are disparate strands of research in literacy, reading, and writing that turn out, on deeper examination, to have deep connections.

Perhaps the earliest developing such strand, work on text readability, goes back to the work of Flesch (1948) and his predecessors. Models of text readability use automatically extracted measures of text complexity, such as sentence length and word frequency, to predict how challenging a text will be for readers at a specific age or level of reading development. As these models have developed, they have expanded to include a much larger battery of NLP features, resulting in systems like Coh-Metrix (Graesser, McNamara, & Louwerse, 2004) or TextEvaluator (Sheehan, Napolitano & Flor, 2014), that use a large battery of NLP features, organized into underlying traits or dimensions, to predict text difficulty.

This work intersects with a different research tradition, dating back to the work of Biber (1988) and his colleagues on genre and stylistic differences in speech and writing. As these models have developed, they have identified a large array of text features that can be identified automatically, organized those features into a set of underlying dimensions, and used the resulting n-dimensional space to identify the registers and styles that characteristically distinguish texts produced in different communities for different purposes and for reception by different audiences.

Finally, there is a separate research tradition, focused at first on automated essay scoring, and then more generally on automated writing evaluation, that began with Page (1966) and led to a variety of commercial and research systems, such as Project Essay Grade, e-rater (Attali & Burstein, 2006), and the Intelligent Essay Assessor (Foltz, Laham, & Landauer, 1999). These systems also seek to identify traits or dimensions of variation that matter for the evaluation of writing quality (Attali, 2011; Shermis, 2002).

Each of these strands of research focuses on a distinct criterion variable: readability, genre classification, or writing quality, but there is considerable overlap in the kinds of features and the traits that such models identify. It thus seems reasonable to develop a unified trait model that can be applied for any of these purposes, and used to track not only student writing quality, but also their ability to flexibly adapt their writing to specific contexts and situations, over time. Such a model has obvious instructional utility – utility that goes beyond assigning essay scores to essays or reading levels to school texts. The instructional goals that teachers set for student writing can be translated into the intention to give students not only the sophistication to produce complex texts, but also the flexibility to produce texts that meet the expectations and requirements of their purpose and audience. A trait model, by identifying the dimensions along which teachers want students to be able to position their writing, also helps to contextualize feedback. A specific expectation – such as the need to use transition words effectively – can be viewed as one of a series of moves that show students how to manage complexity along a specific dimension (in this case, organization).

Author

• Paul D. Deane, Educational Testing Service