Motivation Validity: Motivational Implications of Measurement Practices

• In Event: 25.024 - Synergies in Studies of Motivation and Learning: An Interactive Workshop

Abstract

Understanding motivation in education requires measurement in some form. However, balancing the need for good quality measurement and feasible usage can be a major stumbling block in research and practice. In our chapter (Chapter 28), we discuss several instances where we glean insights about student motivation despite using what some may view as taboo measurement practices—brief and single-item self-report measures. We refer to pragmatic measurement, using a combination of statistics, theoretical backing, and situational factors, as a method for ensuring reasonable validity evidence.

Contemporary validity theory emphasizes the importance of validity arguments (APA, AERA, NCME, 2014)—the practice of stating the intended uses of a measure and providing evidence to support such stated uses. It is important to acknowledge the benefits and drawbacks of any given measure because few, if any, are perfect. Thus, determining which measurement trade-offs are most advantageous is an important endeavor for researchers and practitioners alike. For example, all things being equal, longer measures are more likely to be more psychometrically sound than shorter measures because of reliability indices and content breadth considerations. Yet, those longer measures introduce time, effort, and attention drawbacks for participants that may ultimately undermine data collection or data quality.

In this presentation, we apply several of the motivational theories discussed in accompanying chapters to the measurement of motivation itself. The motivation-measurement connection already has foundations in test-taking motivation literature (DeMars, Bashkov, & Socha, 2013; Eklöf, 2003; Sundre & Kitsantas, 2004), which suggests students are less likely to provide high quality data when they are in a low-stakes testing environment (e.g., completing a motivational questionnaire). First, we discuss that one motivational construct operating is task value (Chapter 24). Specifically, students may describe surveys as a waste of time (i.e., perceived cost) or as useless (i.e., utility value). Second, boredom has been described as a negatively activating achievement emotion (Pekrun, Goetz, Daniels, Stupnisky, & Perry, 2011), and failure to account for measurement consequences could elicit feelings boredom experiences in the classroom (Chapter 19). Furthermore, it may be that the negative affective experiences students associate with such questionnaires could spill over and negatively affect their motivation for their class as well.

Ultimately, we argue that motivation is an important aspect of validity and the pragmatic measurement perspective was developed with participant motivation in mind. Failing to consider the motivational impact of administering a questionnaire may be just as detrimental as failing to address construct breadth—perhaps even more detrimental. For example, researchers may be able to restrict their interpretation of an underspecified construct, but random or missing responses provide little to no information at all. We conclude by discussing the psychometric and practical tradeoffs of measuring motivation and instances where pragmatic approaches are more and less appropriate.

Authors

• Jeff John Kosovich, Center for Creative Leadership

• Chris S. Hulleman, University of Virginia

• Kenn E. Barron, James Madison University