Applying Video-Based Signal Processing in Developmental Science: A Pediatric Sleep Example

• In Event: 1-009 - Advanced Analytics for Large, Secondary, and Open Source Data

Abstract

Video recordings are a ubiquitous part of developmental research and they are often used for detailed behavioral coding. As a secondary data analysis technique, video-based signal processing can be used to extract additional biological/health information including heart-rate, breathing, and sleep/wake states. Within this paper we first review how video-based signal processing techniques may be applied to existing developmental science data and then present a series of studies on the application and accuracy of these methods within the context of pediatric sleep.

In a study designed to assess sleep and developmental risk (n = 99), existing videosomnography (VSG) data (video recordings of infant/toddler sleep onset to offset) were processed using open-source and novel video-based signal processing techniques to extract sleep/wake states. The signal-processing methods included background subtraction, long short-term memory networks, and 3D convolutional networks (for an example signal-processing system see Figure 1). To assess the accuracy of the signal-processing methods, videos were first coded by trained technicians/research assistants for sleep onset, offset, and wakings – henceforth referred to as behavioral-VSG codes. The signal processing methods were compared with the behavioral-VSG codes of sleep onset, sleep offset, number of minutes awake after sleep onset, number of minor wakings (< 15 minutes), and number of major wakings (> 15 minutes). Analyses included correlations, t-tests, Bland-Altman Plots, and area under the receiver operating characteristic (ROC) curve.

Optimized models using deep learning (i.e., long short-term memory networks) achieved the highest accuracy, precision, recall, specificity, and area under the ROC curve with 95.60, 96.43, 98.87, 75.88, and .95 respectively. In sum, trained models classified sleep with a high degree of accuracy.

This paper demonstrates how video-based signal processing may be applied in developmental science and details the steps used to optimize a model to achieve the desired sensitivity and specificity. The use of automated video-based signal processing in developmental science is uncommon and there is a wealth of untapped data that may be extracted using this secondary data analysis approach.

Authors

• A.J. Schwichtenberg, Purdue University
  Presenting Author

• Jeehyun Choe, Purdue University

• Pearlynne Chong, Purdue University

• Ashleigh Kellerman, Purdue University

• Emily Abel, Purdue University

• Edward Delp, Purdue University