Paper Summary

Direct link:

How Automatic Coding of Short Text Responses Is Able to Enhance Assessment

Fri, April 17, 4:05 to 5:35pm, Marriott, Floor: Fourth Level, Belmont


Automatic coding of short text responses opens new doors in assessment. We propose and implement a collection of baseline methods for automatic coding and software components that are available under open licenses. The methods' accuracy is demonstrated by using data collected by the Programme for International Student Assessment (PISA) 2012 in Germany (for 10 items, with n = 41,990 responses in total). This paper reports preliminary results of four items for which high percentages of agreement (84--90 %) between automatic and human coding were reached. Nevertheless, performance varied by item, as agreement beyond chance differed strongly (.23 <= Kappa <= .70). Overall, the methods appear accurate and feasible, especially for items that are solved by expressing specific semantic concepts.