Iterating Towards Success: Detailing the Developmental Process of an Observational Manual for Teachers’ Motivational Supports

• In Event: Real insights from real classrooms: Motivation scholars’ experiences designing for and working with observational data

Abstract

Objectives. Teachers can help curb motivational declines commonly observed in STEM with effective motivational supports (Benden & Lauermann, 2022; Bureau et al., 2022; Robinson, 2023). To better understand how teachers deliver motivational supports, theory-driven observational methods are needed (Robinson et al., 2024). Our objectives are to describe the development of a multi-theoretical observational scheme for 22 supports, provide evidence for its soundness, and share lessons learned from the process.
Theoretical Frameworks. Motivational climate theory (Robinson, 2023) describes motivational supports as observable teacher behaviors that shape student perceptions of the classroom. To select specific motivational supports (see Table 1), we relied on situated expectancy-value theory (SEVT), self-determination theory (SDT), and achievement goal theory (AGT). SEVT includes supports that promote students’ confidence and valuing of content (Eccles & Wigfield, 2020). SDT focuses on supports for students’ autonomy, competence, and belonging (Ryan & Deci, 2020). AGT includes supports that promote engaging in learning for interest and improvement instead of demonstrating competence (Ames, 1992).
Method. To develop the observational coding scheme, the research team consulted motivation theory and prior research to decide on relevant instructor behaviors for each of the 22 supports. Generation 1 protocols were written based on this review and detailed guidelines for counting instructor behaviors. These protocols were refined across three waves of trial video coding, with each support being assigned two independent coders at each wave. Protocols were applied to lecture recordings of undergraduate STEM courses at a large Eastern Canadian university (n = 18).
Results. After refining the Generation 1 protocols through preliminary pilot testing, Generation 2 protocols demonstrated weak interrater agreement for 20 supports observed (% agreement = 33-67). This was attributable to poor protocol clarity and the coding approach itself, which involved recording specific timepoints that behaviors occurred. Generation 3 protocols (see Appendix for manual) improved clarity and pivoted to a segment-based approach where coders marked whether a support was observed within 5-minute video chunks (n = 60 chunks). With these changes, interrater agreement improved substantially for all 22 supports (% agreement = 65-98). While 13 supports demonstrated good to strong reliability (Gwet’s AC1 = .63-.98), nine demonstrated weak reliability (Gwet’s AC1 = .33-.57), indicating that these protocols require further refinement. Next steps for development include external expert review to ensure completeness of protocols and fine-tuning of coder training regimens to ensure consistency.
Significance. This project offers an open and honest account of the development of a rigorous observational coding system for classroom motivational supports. The process underscores the importance of having standardized coding procedures, thorough coder training, and evidence-based rationales for assigning numbers to observed phenomena (Robinson et al., 2024). Further, this work will produce a detailed, publicly available observation manual that will be useful for other researchers studying motivational supports. This manual marks an important step towards theoretical integration in motivation research (Linnenbrink-Garcia et al., 2016) as it provides the first rigorously developed system for observing a theoretically diverse array of instructor supports (thereby increasing its potential for having far-reaching impacts in motivation research).

Authors

• Jessica Hunter, McGill University

• Cole D. Johnson, McGill University

• Marianne Dubé, McGill University

• Sydney Perkins, McGill University

• Natasha De Cotiis, McGill University

• Lucas Frankel, McGill University

• Romane Monnet, McGill University

• Johana Sava, McGill University

• Shubhangi Bhardwaj, McGill University

• Sanheeta Shankar, McGill University

• Tal Sterling-Eilok, McGill University

• Jing Lin, McGill University

• Kristy A. Robinson, McGill University