Preliminary Development of a Mathematics Engagement Composite Using NAEP Process Data of Eighth-Grade Students

• In Event: Advancing Educational Research for All with Large-Scale School Assessment Data: Insights from Innovative Approaches

Abstract

One purpose of analyzing process data is to gain insight into how students engage in digitally based mathematics assessments. A strong approach to utilizing process data is to follow Bergner and von Davier’s (2019) framework, which they developed for process data uses; see the graphic developed by Jiang et al. (2021) in Figure 1.

There are five levels of process data uses according to Bergner and von Davier (2019). These levels were not intended to differentiate between better and worse uses of process data. Instead, it helps scholars clarify how they intend to use the process data, as each level enables us to make different inferences. Extending this framework for the context of this study, scholars can make a range of inferences about a student’s mathematical engagement based on the level. Given this information, this study aimed to extend Kirakosian's (2025) investigation to answer the following question: Are there additional NAEP 2017 Process Data indicators that provide validity evidence when assessing a student’s mathematics engagement?

The NAEP 2017 Mathematics Assessment process data of eighth-grade students were used. In the final sample (N = 2800), after manipulating and merging datasets, 51% of the students were girls, 6% were English Language Learners (ELL), 10% were students with disabilities (SWD), 1% of 6% ELL were also SWD, and 48% were eligible for the National School Lunch Program. Furthermore, 9.2% of the students received accommodations. Validity evidence for internal structure was investigated using the AERA et al. (2014) Standards. The development of a composite indicator, using principal components analysis (PCA), was also initiated based on the validity framework outlined in the Standards. A level of process data use will be assigned upon completion of this study.

Correlations among the indicators varied, with some indicators identified as multicollinear. For example, in Item 1 – VH134366, the correlation between the indicator “used draw” and the indicator “used scratchwork” was r = .91, p < .001. See Table 1 for additional indicator correlations for Item 1 – VH134366. The results revealed KMOs that ranged from .57 to .64, and Bartlett’s tests were also significant (p’s < .001), which suggested adequate sampling for the use of PCA. Typically, four components were extracted for each item, and the percent of variance extracted ranged from 50.06% to 62.62%.

Upon examining the evidence for internal structure, quantitative evidence was available that contributed to the evidence for internal structure. The significant and moderately sized correlations support grouping these indicators. The PCAs also showed highly similar component loadings across items. The Standards might suggest that this evidence was adequate, meaning good enough. Based on the Bergner and von Davier (2019) framework, the process data use of this study would fit within Level 3: Process data is incorporated as essential to understanding the outcome.

More rigorous statistical procedures could have strengthened the preliminary development of this composite indicator. A strongly developed composite indicator could help teachers obtain a snapshot of their students’ engagement in digitally based mathematics assessments.

Author

• Antranik T Kirakosian, Washington State University