Scoring and Aggregating Data From Scenario-Based Assessments to Recover Learning Progressions

• In Event: Building and Validating Cognitively Based, Technology-Enhanced English Language Arts Assessments

Abstract

We focus on the empirical recovery of learning progressions in ELA. In earlier work, we explored several methods to this end for scenario-based writing assessment including Wright maps and Bayes nets. However, in our new study, we collected of data from parallel scenario-bases task sets. This makes it possible to explore the stability of learning progression levels across parallel tasks, compared to the stability of levels across dissimilar tasks. Since the educational research community, and the CBAL initiative in particular, are investing heavily in the development of learning progressions, examining their validity is important for both scientific and practical reasons.
Different methods have been applied to the problem of validating learning progressions. Steedle and Shavelson (2009) used confirmatory and exploratory latent class analysis to validate a learning progression for explaining constant speed. Although they validated two of the four levels from the learning progression, the other two levels did not have a clearcut interpretation. This may have occurred because students did not consistently endorse misconceptions associated with these levels.
Wilmot, Schoenfeld, Wilson, Champney, and Zahner (2011) developed and validated a learning progression for mathematical functions. Student responses to constructed-response items were scored using the levels from the learning progression, and a Wright map based on a partial credit model was used to examine validity. Within items, the data fit the learning progression quite well, but there was too much variability among levels across items to establish cut points. West and colleagues (2010) validated a learning progression for internet protocol addressing. In this case, each item, rather than each response, was associated with one of four levels from the progression. The model was instantiated as a Bayes net with a single parent with n + 1 states, where n is the number of levels in the learning progression (a special case equivalent to a latent class analysis). The learning progression was validated for the most part, though they noted that extending the approach to accommodate multiple learning progressions (i.e., a Bayes net with multiple parents) is among the future challenges.
We focus on a single writing genre: argumentative writing for eighth grade students. Three parallel forms were created, with each form containing a different unifying scenario that includes a mix of selected- and constructed-response items, both dichotomous and polytomous. The tasks containing selected-response items serve as a lead-in, and each assessment culminates in an extended writing task. Two main learning progressions, summarization and argumentation, underlie the assessment design.
Students randomly took two of the three parallel forms within a short time period. The data were collected in the spring of 2013 and are currently being analyzed. The sample size is approximately 1,500 students. The main research question is concerned with the empirical recovery of the learning progression across the three parallel forms.

Authors

• Peter Van Rijn, ETS Global

• Edith Aurora Graf, ETS

• Paul Deane, Educational Testing Service