Analyzing Process Data in the NAEP Technology and Engineering Literacy Assessment

• In Event: 46.027 - Issues and Developments in NAEP Technology-Rich Assessments: Representative Work of NAEP–Educational Testing Service Summer Undergraduate Research Experience/Summer Predoctoral Research Experience Fellows

Abstract

In parallel with advances in technology, assessment designs are moving away from paper/pencil tests towards computer or technology-rich assessment. The NAEP Technology and Engineering Literacy (TEL) assessment is one such move towards scenario-based tasks (SBTs, National Assessment governing Board, 2013). SBTs departs from traditional paper/pencil assessment items as they allow for a degree of interactivity and manipulation via multimedia tasks. Each interaction between the assessment and examinee provides an additional layer of data that was not accessible in previous generations of assessments. The purpose of this presentation is to illustrate the type of information we can expect to extract from innovative items such as SBTs (i.e., examinee time management strategies and strategies in problem solving), and demonstrate a range of methods to investigate time series data via visual examination and statistical analyses. Two SBTs from the NAEP TEL assessment are illustrated.

The move to SBTs allows us to collect process data of student actions that lead up to the student’s final test response. For example, we can trace examinees’ logic, interaction with the test, and problem solving strategies (Gorin, 2006). In this study, multiple exploratory data analysis methods such as scatterplots, clustering, dendograms, spatial mapping; as well as exploratory statistical analyses such as principal components analysis (PCA) will be used to model process data in two representative NAEP TEL items. The data used are the anonymized raw data representing the examinee’s tracked actions and responses as they progress through the SBT. Two SBTs (items) will be used, specifically one item will be used from the 2014 administration, and one item from the 2013 administration, collected by the Educational Testing Service (ETS). Preliminary PCA results show that for example that, examinees employ different time management strategies. More specifically, in the introductory phase of the assessment, some students spend more time at the beginning of the SBT, whereas in the solving phase, other students spend proportionately less time at the beginning and more at the end. This implies that there are differential profiles of time management strategies. We can further investigate whether the differing groups also employ different problem-solving strategies by mapping their response process using clustering methods. Essentially, what this allows us to do is to test the “empirical evidence that the theoretical processes are actually engaged by respondents in the assessment tasks” (Messick, 1995). By means of triangulation, the richness of the information that can be extracted from process data allows us to strengthen our validity argument on multiple levels. In paper/pencil tests, the bulk of our information revolved around the examinees’ final submitted responses; we had sparse information on the intermittent process and approaches employed examinees to reach this conclusion. Using process data allows us to facilitate formative assessments as we can inform teachers, parents, and students where this is a gap in knowledge.

Authors

• Emma M Sunnassee, University of North Carolina - Greensboro

• Zhan Shu, ETS