Paper Summary

Direct link:

Using Natural Language Processing Methods to Assess Treatment Fidelity

Mon, April 12, 11:10am to 12:10pm EDT (11:10am to 12:10pm EDT), Division H, Division H - Section 2 Poster Sessions


A key challenge in education evaluation is assessing fidelity of treatment across settings. Ideally, researchers observe each treatment session and rate the session against the treatment protocol. Unfortunately, this method of assessing fidelity is time-consuming, expensive, and too often infeasible in field settings. In this paper we propose a measure of fidelity created using a set of natural language processing techniques termed document similarity where the researcher calculates the similarity of intervention transcripts to an ideal script representing “gold-standard” implementation of the intervention. The measure applies to highly-scripted interventions and has the potential to provide a quantitative rating of how consistently an intervention is delivered in a way that is low-cost, scalable, and generalizable across treatment contexts.