Using Natural Language Processing Methods to Assess Treatment Fidelity

• In Event: Topics in Contemporary Program Implementation, Evaluation, and Measurement

Abstract

A key challenge in education evaluation is assessing fidelity of treatment across settings. Ideally, researchers observe each treatment session and rate the session against the treatment protocol. Unfortunately, this method of assessing fidelity is time-consuming, expensive, and too often infeasible in field settings. In this paper we propose a measure of fidelity created using a set of natural language processing techniques termed document similarity where the researcher calculates the similarity of intervention transcripts to an ideal script representing “gold-standard” implementation of the intervention. The measure applies to highly-scripted interventions and has the potential to provide a quantitative rating of how consistently an intervention is delivered in a way that is low-cost, scalable, and generalizable across treatment contexts.

Authors

• Kylie Anglin, University of Connecticut - Storrs

• Vivian Wong, University of Virginia