Evaluating Educational Program Implementation Quality: The Potential of Digital Usage Data

• In Event: AERA Roundtable Session Saturday 3:20 pm Four Seasons Ballroom 1
  In Roundtable Session: Leveraging Technology and Digital Learning: Diverse Evaluation Perspectives (Table 11)

Abstract

Assessing implementation quality is vital for effective program evaluation, as poor implementation can lead to inconsistent educational outcomes. Measuring fidelity through traditional methods (e.g., observations, self-reports, and interviews), however, often face issues with accuracy and generalizability. Digital usage data (DUD), which tracks user interactions within educational programs, offers a promising cost-effective and scalable solution. Despite its potential, research on DUD's effectiveness in measuring program fidelity is limited. To address this research gap, this study examined DUD's role in assessing fidelity in a national math curriculum. Preliminary results suggest that DUD not only corroborated survey data on program usage but also provided a reliable and valid supplement to observational data, showing promise as a cost-effective measure of program fidelity.

Authors

• Jeffrey Yo, Curriculum Associates/University of California - Los Angeles

• Emily Borawski, HumRRO

• Caroline R.H. Wiley, HumRRO

• Rivka Revivo, HumRRO