Devising assessments based on psychometric principles: Secondary analyses of literacy, numeracy, and social-emotional assessment data in the Syrian response region

• In Event: Innovating assessment: devising a holistic assessment for education cannot wait in Syria

Proposal

The education response in Syria and neighboring host countries has primarily focused on increasing access to formal and non-formal education services, and as a consequence, little is known about the extent to which Syrian children are achieving foundational literacy, numeracy, and social-emotional skills critical for long-term learning and well-being. To address this gap, several organizations have administered a variety of academic and social-emotional learning (SEL) assessments tools, including the Early Grade Reading Assessment (EGRA) and the Early Grade Math Assessments (EGMA; RTI International, 2009); the Save the Children (STC) Learning Assessment; the ISELA (STC, 2017); Children’s Stories (Dodge et al., 2015; 3EA, 2018) and the Pre-School Self-Regulation Assessment (Smith-Donald et al., 2007; 3EA, 2018). However, common approaches to scoring such assessments have assumed that all items or subtests within the assessment provide reliable and valid data on children’s early literacy, numeracy, and social-emotional well-being. This assumption is likely untenable given that (a) many assessments were developed for use in Western contexts; and (b) underwent varying levels of adaptation to capture important linguistic (e.g., diglossia), curricular (e.g., orthography, script), and cultural (e.g., gender norms) factors within the Syrian context.
To address this gap, in the current study we conducted a series of secondary psychometric analyses using 5 unique datasets collected from Syrian children in grades 2-4 enrolled in formal or non-formal schools in Syria and in non-formal retention programs in Lebanon. Analyses were designed to assess the construct validity and reliability of data collected using the literacy, numeracy, and SEL assessments described above, with a focus on the following research questions:
(1) Are correlations among subtest/items scores within each assessment consistent with the construct the assessment is hypothesized to assess (e.g., early literacy, empathy)?
(2) What is the strength of the relationship between observed items/subtests and the underlying latent constructs?
(3) What is the reliability of the best-fitting model identified in RQs 1 and 2?
Datasets were randomly split stratified on grade and gender into (1) an exploratory sample, to examine using EFA techniques multiple versions of data-driven models, of which a final proposed solution was selected based on conceptual and empirical considerations; and (2) a confirmatory sample, to test hypothesized and proposed factor structures, thereby building confidence in the stability of empirically derived EFA estimates (Osborn & Fitzpatrick, 2012). All models used a WLSMV estimator with probit-link function (Beauducel & Herzberg, 2006; Lei, 2009) adjusting for the nested data structure.
Results provided important insights into how to improve the construction of assessments for use with Syrian primary school-aged students. For example, analyses of two versions of the EGRA administered in Syria and Lebanon indicated that the version of the letter sound subtest that separated out diacritics – short vowels that provide guiding information for children – was less related to the underlying early literacy construct than the version that included diacritics with the letter forms. Implications of the results for test construction – when interpreted in the context of evidence and theory from Arab child development scholars and practical feasibility – will be discussed.

Authors

• Carly Tubbs Dolan, New York University Global TIES for Children

• Roxane Caires, Global TIES for Children, New York University

• Ha Yeon Kim, New York University