Balancing Development and Measurement Needs in an Evaluation of a Program Under Development

• In Event: Roundtable Session 15
  In Roundtable Session: Exploring the Methodological Implications of Conducting Research on Evaluation

Abstract

In programs under development, definitions of the intended outcomes typically shift or become more refined. When evaluators of such programs are expected to measure student-level outcomes, they face difficult challenges and complexities. This study describes the steps an evaluation team took to strengthen internal validity claims in the evaluation of a science-teacher professional-development (PD) program while simultaneously engaging in formative evaluation endeavors throughout the program’s development. By including PD providers in item development and sharing the empirical item functioning data with them, we made instrument development decisions that addressed the interests of the program and the needs for psychometric rigor. Using Rasch-analytic equating methods, we conducted group comparisons, meeting our reporting requirements while negotiating feasibility constraints.

Authors

• George M. Harrison, University of Hawaii at Manoa

• Lisa M. Vallin, University of Hawaii at Manoa

• Paul R. Brandon, University of Hawaii - Manoa

• Joanna Philippoff, University of Hawaii - Manoa

• Kanesa Seraphin, University of Hawaii at Manoa