Proof of Concept: Topic Modeling as a Method for Focusing an Evaluation Design

• In Event: Research on Evaluation Design, Process, and Outcome

Abstract

A first step in evaluation is to focus the evaluation design based on stakeholder feedback. Stakeholder feedback in evaluation has primarily been analyzed using either qualitative or concept mapping methods. Yet, these approaches are limited in their ability to gather and analyze a large volume of qualitative feedback in a short time frame. The current research on evaluation study proposes the use of Topic Modeling (TM), a method that can analyze large numbers of stakeholder responses to open-ended items, to overcome these limitations. To investigate the applicability of TM in evaluation practice, a stakeholder questionnaire was developed and administered. The results of the study provide promising evidence that TM can be useful to evaluators during the evaluation focusing process.

Authors

• Dakota Wayne Cintron, Neag School of Education, University of Connecticut

• Bianca Montrosse-Moorhead, University of Connecticut

• William Holmes Finch, Ball State University