Development of Gaze-detection software using ARKIT for the NIH Infant and Toddler Toolbox

• In Event: An Introduction to the NIH Infant and Toddler Toolbox for infants 1-42 months

Abstract

The NIH Infant and Toddler Toolbox (herein referred to as the NIH Baby Toolbox or NBT) is tasked with assessing six domains of development among children 1 to 42 months, using only an iPad for administration. With direct assessment as a primary goal of the NBT, we set out to develop a method of iPad-based gaze-tracking that would allow infant looking paradigm to be incorporated into the Baby Toolbox. Gaze detection is ideal for assessing infants as it does not require a coordinated motor or verbal response. It was also a primary measure technique mentioned in an initial survey of experts and literature review. Upon completion of the project NBT will be the first standardized assessment to use mobile gaze detection.

NBT gaze detection uses the front-facing camera built into a standard iPad Pro, which makes it non-invasive, inexpensive, and easy to administer. The resulting data can be used to determine the proportion of looking to one of two sides (left/right preference), identify a “look” to trigger gaze contingent responses and determine on from off screen looking (for habituation designs). Data exports include summary scores, x,y coordinates captured at 60 frames per second, as well as images of the participants, taken at 30 frames per second.

In the design of the gaze detection technology, it was critical to ensure that gaze data was captured in real-time without a significant delay between data capture and stimuli presentation. Apple’s augmented reality software, ARKIT, is used to extract estimates of gaze points. The gaze algorithms use a four-point calibration to identify the edges of the screen and adjust the data collected during live items.

To develop this technology, we ran multiple rounds of usability testing and three pilot studies with infants from 1-30 months to determine reliability and calibration success of our gaze approach. The first pilot (N=100) determined reliability between hand-coding and gaze capture algorithms. A second pilot (N=108) found low rates of calibration success (20%), leading us to update calibration images and procedure. Updated algorithms and calibration targets were then incorporated into a third pilot. Within a sample of 84 infants 4- to 24-months (M = 14.3, SD =6.6), 60% successfully calibrated within three rounds, with no difference by race or age.

The current NBT gaze assessments include measures of cognition, language, and numeracy. There are six assessments: habituation, memory, visual delayed response, “Looking while Listening” a measure of receptive language, spatial change detection, and numerical change detection (approximate number system). This technology has unparalleled capacities to bring real time gaze-tracking and post processing capacities to standardized assessment and could very well revolutionize the field developmental assessment.

Authors

• Miriam A Novack, Northwestern University
  Presenting Author

• Aaron J Kaat, Feinberg School of Medicine, Northwestern University

• Rachel Taft Hanrahan, Northwestern University

• Pranav Rajiv, Northwestern University

• Andrey Pankov, Northwestern University

• Anastasiya Shvaykovskaya, Northwestern University

• Steve Glinberg, Blue Moon Learning

• Paula Hom Sievert, Helium Foot

• Cecily Frederick, Glinberg & Associates, Inc.

• Maria Varela Diaz, Northwestern University

• Richard Gershon, Northwestern University