Rater Effects vs Robo-Effects: Profiling Automated Classroom Evaluation

• In Event: Writing Assessments and Rater Effects

Abstract

Could a microphone improve the reliability of classroom observation ratings? This paper employs both Generalizability Theory and IRT rater models to contrast human rater effects with the effects observed in automated ratings produced by custom NLP models trained on classroom transcripts to investigate how they might support human observers.

Author

• Michael Hardy, Stanford University