Situational and Individual Validity and Reliability of the Experience Sampling Method: Intensive Longitudinal Data

• In Event: Investigating Optimal Learning Moment to Enhance Student Engagement and Social Emotional Leaning

Abstract

Objectives
This study aims to review several analytical and methodological challenges of Experience Sampling Method (ESM) data, collected in the context of high school students’ daily lives. It presents evidence for short-term (e.g., each data collection wave, location, activity) and long-term (e.g., several years) reliability when using ESM for data collection. We also present several validity indices (e.g., ICC, random variance component) to evaluate emotional state validity, situational validity, ecological validity, intra-individual covariation between states, and within- and between-person variability.

Theoretical Framework
ESM and intensive longitudinal data collections are increasingly used to study students’ engagement and experiences in learning (Bolger & Laurenceau, 2013; Csikszentmihalyi, 2014; Schneider et al., 2016). ESM is a momentary ecological assessment that allows participants to provide data of their feelings, emotions, and behavior multiple times across a wide range of situations. Those repeated measures can be linked to contextual factors such as participants’ daily routines, location, activities, or companions (Mehl & Conner, 2012; Bolger & Laurenceau, 2013). Systematic investigations on the reliability and validity of such measures in school settings could contribute to the use of ESM in the education field (Zirkel, Garcia & Murphy, 2015; Schneider et al., 2016). This study provides a practical evaluation of several psychometric properties, such as intra-individual variation, the variability of emotional states, and covariation-based constructs.

Data Sources and Methods
Participants in ESM data collection respond to signals using a smartphone and answer experience-sampling questions at randomly selected times within certain time intervals. Data was collected from 2013-2019 from 1900 students and 74 teachers in the eight waves of data collection. This study uses intra-individual means (iM), intraindividual standard deviations (iSD), and intra-individual variance (iSD2) to quantify emotional states and constructs. We also describe features of how an individual’s feelings change or fluctuate over time in the framework of multilevel time-series analysis (Nesselroade & Ram, 2004; Molenaar et al., 2009). We use Mplus 8 and several R packages to generate the indices of reliability in the repeated measures. This study will also discuss methodological issues dealing with missing data and unequal time intervals between situations.

Results
Initial results show moderate to good internal consistency across emotional states and within- and between-person variability. Preliminary results, conducted under the multilevel framework, indicate evidence of construct validity and show consistency across data collection waves. Table 1 reports data on the intra-class correlation coefficient in emotional states from w5-w8. The ICC is used to assess variability between and within individuals across waves in emotions/feelings. The higher the ICC, the more emotions/feelings should be considered high school students’ characteristics rather than situational states.

Significance
ESM provide unique data on students’ emotional states in a variety of contexts. These methods include logistical challenges for data collection and analysis, so understanding the psychometric properties will contribute to improved study designs and allocation of resources for data collection. An understanding of emotional states versus traits will allow researchers to ask appropriate research questions that intensive longitudinal data can address in education contexts.

Authors

• Christopher R Klager, Michigan State University

• I-Chien Chen, Michigan State University