Automated Assessment of Writing Proficiency: Can Text Mining of Argumentative Texts Lead to More Nuanced Assessments?

• In Event: Innovative Practices for Assessment in Computer-Based Learning Environments

Abstract

Effective written communication is crucial in both academic and professional settings (Kellogg, 2006). A major bottleneck in the assessment of writing proficiency is the creation of accurate scoring templates that are used by human coders to score written documents (Graesser & McNamara, 2011; Graesser, McNamara, & Kulikowich, 2011).
This paper explores the ways in which computer-based learning environments (CBLEs) can be used to promote the development of proficiency in written communication through automated assessment and personalized instruction. We outline a text mining technique by applying it to the patient case summaries written by medical students and physicians who use BioWorld (Lajoie, 2009), a computer-based learning environment for practicing diagnostic reasoning.
We collected written case summaries from 29 second-year medical students and 5 practicing physicians while they solved 3 cases that showed typical symptoms of diseases, including Pheochromocytoma, Diabetes Mellitus Type 1, and Grave’s disease. The final corpus of written case summaries included 76 case summaries written by novices and 14 by the experts. The average length of case summaries was 58.54 words and 324.68 characters, yeilding a total of 5269 words and 29221 characters included in the analysis.
The goal of the proposed text mining technique is to assist researchers in identifying linguistic features that distinguish between proficiency levels (novice students vs. expert physicians) in written case summaries as well as the quality of these parts (e.g., class and accuracy of diagnosis, symptoms, lab tests, etc.). In doing so, text mining algorithms proceed in a sequence of steps. First, case summaries were segmented through a series of pre-processing algorithms to obtain a document-by-term matrix, where the value of each dimension corresponds to term occurence. Second, the constituent terms are analyzed to extract parts of case summaries, where the output of the first layer of text classification informs the subsequent layer. Third, the linguistic features that characterize expert case summaries are selected in terms of distinguishing the quality of parts of case summaries. Our preliminary analysis of the first two layers of the assessment model demonstrates an average accuracy of 92.32% and 98.33% for the recognition of diagnosis accuracy and class, respectively.
The proposed text mining technique stands to facilitate the design of CBLEs that automate the assessment of writing skills proficiency, with implications for adapting instruction to the specific needs of different learners (see Dai, Raine, Roscoe, Cai, McNamara, 2011). The demonstration and evaluation of the proposed technique could be of practical importance for researchers that attempt to design and evaluate the outcomes of formative feedback interventions in terms of enhancing written communications skills. We will raise several challenges and discuss possible solutions for implementing the assessment model in CBLEs with the aim of delivering individualized formative feedback.

Authors

• Eric G. Poitras, University of Utah

• Laura Naismith, University Health Network

• Susanne P. Lajoie, McGill University