Cycle for Validating a Learning Progression

• In Event: 47.063 - Validating Theories of Learning for Classroom Assessment Design: Sources of Evidence

Abstract

A learning progression (LP), or learning trajectory (LT), describes the evolution of student thinking from early conceptions to the target understanding within a particular domain (e.g., Clements & Sarama, 2004; Confrey, Maloney, Nguyen, Mojica, & Myers, 2009; Corcoran, Mosher, & Rogat, 2009). The descriptions of student thinking in an LP are often captured as ordered levels. While an LP is ideally research-based, until its interpretation and use are validated it is a provisional theory of learning (Bennett, 2015; Kane & Bejar, 2014). Following at least preliminary validation, an LP may be useful as a guide for both assessment and instruction.
Graf and van Rijn (2016) proposed a cycle for validation of an LP. In its current form, the cycle includes four steps: 1) Theory Development, 2) Examination of Empirical Recovery, 3) Comparison to Competing Models, and 4) Evaluation of Instructional Efficacy. Theory development includes analysis of the logical structure of the domain (Daro, Mosher, & Corcoran, 2011), and of the research literature. An assessment design framework, such as Evidence-Centered Design (Mislevy, Steinberg, & Almond, 2003) can support theory development. An LP assumes that learning levels are both distinct and ordered. Whether these assumptions are tenable can be examined empirically (the second step of the cycle). If the data are consistent with the specification and ordering of the levels, then the levels are said to be empirically recovered. These examinations typically combine both qualitative and quantitative analysis. Early examinations may involve qualitative analysis of cognitive interviews (e.g., Jin & Anderson, 2012), while later examinations based on large samples typically involve psychometric methods such as IRT (e.g., van Rijn, Graf, & Deane, 2014; Wilmot, Schoenfeld, Wilson, Champney, & Zahner, 2011).
Empirical recovery of the ordering of levels is necessary, but not sufficient. Another theory of learning (besides the one embodied in the LP) might account equally well for the specification of levels. Seeking disconfirmation through comparison to competing theories of learning is an important component of validation (Messick, 1989; Shavelson, 2009). Finally, an LP is only useful to the extent that it can be used to effectively guide instruction. This is the focus of the fourth validation step: When compared to other forms of instruction, is instruction guided by a particular LP more effective for enhancing student understanding?
A feature of the LP validation cycle is that it assumes the LP can be used provisionally, for example, to guide the design of tasks that will elicit the evidence needed for an examination of empirical recovery.
During a panel that explored the application of LPs to formative assessment, we discussed the LP validation cycle and how it might be improved. For example, it was pointed out that design experiments, which involve ‘“engineering” particular forms of learning and systematically studying those forms of learning within the context defined by the means of supporting them’ (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003, p. 9), could be used to inform theory development. In this presentation, this suggestion and others to adapt the cycle will be discussed.

Authors

• Edith Aurora Graf, ETS

• Peter Van Rijn, ETS Global