How “gold” is this standard? Psychometric Properties of Direct and Adult Report ECD Assessment Items

• In Event: Strengthening ECD Monitoring: Psychometrics, Population Surveys, and Integrated Data Platforms

Proposal

Comparing direct and adult-reported modalities of administration in ECD assessment is often complicated by differences in assessment length, the specific constructs items attempt to measure, and how items are scored. This paper attempts to understand item-level psychometric properties with two datasets used in the development of the ECDI2030 (Halpin et al., 2023, Rao et al., 2021). In each dataset, a series of item-pairs are isolated that have construct overlap, [0,1] scoring, and administration in both direct assessment and adult report modality. The first dataset (referred to as BCIM), consists of 956 children from Bangladesh, China, India, and Myanmar and contains 15 item-pairs (Rao et al, 2021). The second dataset, consisting of 1608 children in Mexico, contains 11 item-pairs .Items in both datasets covered Motor, Language/Literacy, Cognitive/Numeracy, and Executive Functioning/Social Emotional domains.

To explore the effects of modality, this paper uses external criteria by assessing how sum scores of tests and item response patterns correlate with age and differ by modality. It also explores internal criteria by estimating the internal consistency of same-length tests by modality and using Item-Response Theory to estimate the information of Direct Assessment and Caregiver Reported modalities. Finally, using a Bayesian implementation of the Explanatory Item-Response Theory model, it investigates how item parameters differ systematically by modality, the age of the respondent, and the domain of the item (Bürkner, 2021; Wilson & De Boeck, 2004).

Sum scores on the Direct Assessment and Caregiver Reported tests have moderate-to-strong correlations with each other of 0.592 in the BCIM dataset and 0.596 in the Mexico dataset. While reasonably well correlated, sum scores in the two datasets differ markedly in their correlations with external and internal criteria. In the BCIM dataset, the correlation with age was 0.502 for the caregiver reported assessment and .608 for the direct assessment. In the Mexico dataset, the correlation was .607 for the caregiver report and .765 for the direct assessment. In terms of internal consistency, the 15-item direct assessment in the BCIM dataset had a Cronbach’s alpha of 0.752, significantly (χ^2=22.308,p<0.001) higher than the 0.656 of the equivalent caregiver reported test (Cronbach, 1951; Diedenhofen & Musch, 2016). In the Mexico dataset, the 11-item direct assessment had a Cronbach’s alpha of 0.812, significantly higher than the 0.683 of the caregiver reported assessment (χ^2=91.212,p<0.001).

Exploring results in an IRT model also reveals a consistent pattern of more information provided by the directly assessed items. Of the 15 item-pairs in the BCIM dataset, the directly assessed item provided more information in 12 cases. For the Mexico dataset, all 11 of the directly assessed items provided more information than the caregiver reported counterpart. Explanatory item response models are used to explore how these results in more detail, and in particular whether the results hold for younger children, the social-emotional domain.

The paper concludes by discussing the implications of differences in item parameters by modality, potential remedies to compensate for differences, and how criteria beyond the psychometric properties of items are important to consider when selecting an assessment to measure ECD.

Author

• Jonathan Seiden, Peabody College Vanderbilt University