Poster #24 - Making better sausage: iterative development of an observational tool assessing emotion-focused teaching

• In Event: Poster Session 19

Abstract

INTRODUCTION
In preschool, the development of children’s emotion skills benefits their engagement with peers, teachers, and learning activities. Teachers support the development of these skills through emotion-focused teaching. The EMOtion TEaching Rating Scale (EMOTERS) is a tool that measures the quality of preschool teachers’ emotion-focused teaching as observed in 10-minute segments. Specifically, the EMOTERS captures how teachers express their own emotions (Modeling), respond to children’s emotions (Responding), and provide information about emotions to children (Instructing).
The EMOTERS was developed via an iterative process using multi-facet Rasch Modeling that accounts for item, rater, and segment differences, which revealed that EMOTERS items were spread across a difficulty continuum (Gordon et al., 2021). The tool also has strong predictive validity. Scores on the EMOTERS are related to a range of child outcomes, including fewer negative expressions and maladaptive emotion-related behaviors, better emotion regulation and learning behaviors (Fatahi et al., under review), and social-skills (Curby et al., 2021).
Despite its psychometrically sound properties and established predictive validity, the tool has features that may be barriers to implementation (e.g., contingent items, length, and inconsistent conceptual clarity). Therefore, this study aims to evaluate a shortened, streamlined version of the tool that reduces rater burden and improves conceptual clarity.
METHODS & RESULTS
The creators of the EMOTERS revised the tool by eliminating items that were redundant in item difficulty, removing display logic, creating parallel wording across similar items, and providing clarity to frequently discussed items.
Pre-service preschool teachers, virtually all of whom identify as Black or Hispanic/Latino women, filmed themselves in their classrooms throughout the 2020-2021 school year as part of a larger study. Each video segment was coded by at least two raters using the published version of the EMOTERS.
Study 1: Three raters used the new version to code 100 randomly selected video segments from 28 teachers (about 35 videos per rater). Analyses revealed a lower-than-expected correlation between the total scores of the two versions (r=.5, p<.001).
Study 2: A follow-up study better controlling for potential rater effects is underway. Ten raters are each coding a new set of 40 video segments from 15 teachers (36 randomly selected, and four selected because exemplify different dimensions of the EMOTERS) with both the published and new version of the EMOTERS, counterbalancing order. Raters will also complete a questionnaire on the tools’ relative clarity and ease-of-use for each version.
Alignment between versions will be assessed via correlations, which we expect to be in the moderate-high range. To identify conceptually confusing items, we will calculate inter-rater reliability for each version and identify items in each version with the most rater discrepancy. To further contextualize the findings, we will compare the rater’s scores on each version’s tool clarity questionnaires with t-tests.
EXPECTED RESULTS
We expect to find improved alignment between versions, and clear next steps for tool revision to improve rater burden and conceptual clarity. This work will share important insights about the iterative development of observational measures that apply broadly to the field of child development.

Authors

• Emma Casey, George Mason University
  Presenting Author

• Timothy W Curby, George Mason University

• Kate Zinsser, University of Illinois at Chicago

• Haoyu Lin, George Mason University