Intervention Effect or Measurement Artifact? Using Invariance Models to Reveal Response-Shift Bias in Experimental Studies

• In Event: Measurement Invariance

Abstract

One aspect usually overlooked in studies evaluating the efficacy of educational interventions is the extent to which posttest responses reflect actual changes in targeted latent constructs versus a more superficial shift in participants’ interpretations of assessment items after exposure to the intervention (i.e., response-shift bias). The goals of this study are to evaluate through a simulation study the impact of response-shift on treatment effects and to offer a framework to evaluate response-shift via measurement invariance across groups and across time. Our results indicate that treatment effect estimates are severely overestimated in some conditions. We discuss statistical approaches researchers could take to mitigate response-shift while at the same time conceptualizing response-shift as a first step in the evaluation of interventions.

Authors

• Margarita Olivera Aguilar, American Institutes for Research
  Presenting Author

• Samuel Rikoon, Educational Testing Service
  Non-Presenting Author