Strengthening Developmental Science with Psychometric Evaluations: An Examination of the Spatial Arrangement Method in Children

• In Event: Measurement Issues in Developmental Research

Abstract

Examinations of a task’s ability to provide valid and reliable measurements of a construct are seldom conducted for cognitive tasks – but this scarcity is costly to the field (Parsons et al., 2019; Shaw et al., 2020). First, it makes it challenging to relate performance in cognitive tasks to other factors of interest (e.g., neural correlates; demographics). Second, it makes it challenging to evaluate the (lack of) replication of experimental manipulations. While the ‘replicability crisis’ has led to important practice changes, the psychometric evaluation of cognitive tasks has remained a less explored – but equally crucial – component of improving the field’s methodological practices. Evaluating cognitive tasks has also become increasingly important as our field addresses current limitations to in-person testing and recognizes the importance of diversifying our samples. Here we focus on the spatial arrangement method to assess semantic structure in children and evaluate its psychometric properties and feasibility for remote data collection.

The spatial arrangement method (Goldstone, 1994) has been increasingly used with children participants to collect estimates of similarity (e.g., Fisher et al., 2015; Vales et al., 2020). Participants are shown tokens of word referents and asked to arrange those items in a two-dimensional space by using physical distance proportionally to similarity. Prior work suggests that this method captures semantic similarity across the lifespan, including some evidence of good construct validity. To date, however, there are no evaluations of the internal consistency and test-retest reliability attributes of this method in children, nor any assessments of the feasibility of using this method for remote data collection. To address the first gap, in Study 1 we developed a new measure that captures the structure of an arrangement in this task and used this measure to examine the psychometric properties of the spatial arrangement method in children. To address the second gap, in Study 2 we used remote data collection procedures to replicate prior work showing that 4-6 year old children more strongly represent across-domain distinctions relative to within-domain distinctions (Vales et al., 2020a; 2020b).

In Study 1, we found that measures derived from the spatial arrangement method show good psychometric properties (Figure 1). Specifically, children’s semantic structure scores (1) were correlated with and predicted performance in another task thought to rely on semantic structure, (2) were correlated across two parallel forms targeting the same domain of knowledge, and (3) were correlated and with strong agreement across two sessions a week apart. In all cases, the observed test statistics were different from those resulting from simulated random item arrangements. In Study 2, which used remote data collection procedures, we replicated the pattern of results found in prior work (Figure 2).

Together, these studies show that the spatial arrangement method has strong construct validity, internal consistency, and test-retest reliability when used to collect estimates of semantic structure in children, and is a promising method for remote data collection. We discuss the implications of these findings for examining semantic structure in children and for strengthening methodological practices in developmental science.

Authors

• Catarina Vales, Carnegie Mellon University
  Presenting Author

• Christine Wu, Carnegie Mellon University
  Non-Presenting Author

• Zach Branson, Carnegie Mellon University
  Non-Presenting Author

• Anna V Fisher, Carnegie Mellon University
  Non-Presenting Author