Keystrokes, Edit Distance, and Grading Rules: Psychometric Properties of Short-Answer Items

• In Event: 21.058 - Special Topics in Test Validity Research: Artificial Intelligence in Test Construction and Scoring

Abstract

This research evaluates the psychometric properties of short-answer response items under a variety of grading rules in the context of a mobile learning platform in Africa. Students receive and answer questions from the system via text message. We find that literal string evaluation performs poorly compared to other grading rules. Partial string matching with an edit distance of two provides the highest reliability across exams. This is a simple yet effective rule, which performs well across a variety of languages.

Authors

• David Lang, Stanford University

• Ben Stenhaug, Stanford University

• Rene kizilcec, Cornell University