Poster #141 - Understanding mental health using multimodal sensing technology and machine learning: A proof of concept study

• In Event: Poster Session 09

Abstract

Background: The percentage of individuals with mental illness who report an unmet need for treatment in the U.S has increased every year since 2011(MHA, 2022). Multimodal wearable sensing technology and machine learning hold great promise for research and as clinical tools for detecting and monitoring markers of behaviors and states associated with mental health conditions (Garcia-Ceja et al., 2018; Mohr et al., 2017). This proof of concept study aims to detect and predict mood states based on verbal (word use, pitch, and tone) and physiological (electrodermal, electrocardiograph activity) data among emerging adults utilizing machine learning models.

Method: Participants include 218 emerging adults (M age = 23.1 years; SD = 3.0). Individuals were outfitted with a wireless biosensor on their wrist, which continuously measured electrodermal activity (EDA) and electrocardiograph activity (ECG) from 10:00 am until bedtime. Individuals were also lent smartphones to report their hourly feelings of happy, sad, nervous, and angry moods over one day. 3-minute audio recordings were collected via the smartphone microphone once every 12 minutes during the day.

Planned analyses: We will fit a supervised classification algorithm to build an algorithm to detect psychological mood states. A function of the R package caret, “rpart” will be used to construct a recursive portioning and classification forest of conditional inference trees. The target state for the algorithm will include eight classes: whether or not mood is happy, sad, angry, or nervous. Feature variables will include 214 variables related to vocal pitch and tone, electrodermal activity, speech, and heart rate. Data will be divided into testing and training sets using leave-one-out cross-validation. We will fit the model to the training data using the rpart function and then generate predictions on the test dataset. We will evaluate model performance via correlation matrices, MAE and MAPE.

Implications: Ecological momentary assessment tools like in our study allow us to discern in real-time, real-life exposure of stress and perceived feelings of support that can interact and influence mood. Results will inform machine learning tools in classifying and predicting mood from multimodal data collected from embedded sensors in everyday tools, such as our smartwatches and smartphones

Authors

• Jacqueline Duong, University of Texas at Austin
  Presenting Author

• Kexin Feng

• Kayla Carta

• Sierra Walters, College of William & Mary

• Dominque Benamu

• Theodora Chaspari

• Yehsong Kim, University of Southern California

• Sohyun C. Han, Children’s Hospital Los Angeles

• Gayla Margolin, University of Southern California

• Adela C Timmons, Florida International University