Mixed-Method Pilot Study: Designing a Survey on Evaluation Capacity in Institutional Research

• In Event: AERA Roundtable Session Saturday 9:45 am - Gold 3
  In Roundtable Session: Application of meta-analytic methods to multi-site randomized controlled trials in educational interventions

Abstract

This pilot study examined evaluation capacity in Institutional Research (IR) offices and developed a valid instrument to assess it. Using the Evaluation Capacity Building framework (Heider, 2011), the study explored IR professionals’ evaluative understanding, current practices, and perceived barriers at individual, institutional, and enabling environment levels. Phase one qualitative interviews identified conceptual confusion, limited leadership support, and structural barriers, highlighting missed opportunities for organizational learning and equity. These findings shaped the development of a 25-item survey in phase two, improving its validity by embedding practitioner language and experiences. The mixed-methods design demonstrated how evaluation research can produce contextually relevant tools. Aligned with Unforgetting Histories and Imagining Futures, the study envisions IR as a strategic partner in institutional learning and equity.

Author

• Giang Huynh, Claremont Graduate University