Fostering and Evaluating Coherence in Assessing Learning Progressions

• In Event: 47.063 - Validating Theories of Learning for Classroom Assessment Design: Sources of Evidence

Abstract

Both assessment design and development involve numerous, interconnected decisions that should address the three elements described as the assessment triangle (Pellegrino, Chudowsky, & Glaser, 2001): a theory or set of beliefs about how students think and develop competence in a domain (cognition), the content used to elicit evidence about those aspects of learning and cognition (observation), and the methods used to analyze and make inferences from the evidence (interpretation). In this paper, I will use the framework of the assessment triangle to illustrate the need to explicitly connect and coordinate the three components of the triangle during assessment design and development or the validity of the inferences drawn from the assessment results and the effectiveness of uses of assessment results may be compromised. I use the label “coherent” (National Research Council, 2014) to refer to assessment design and development processes in which the three elements of the assessment triangle are connected and coordinated. The development of the argument for coherence should commence with assessment design and continue following assessment launch.
 
I will present and illustrate the following three criteria for evaluating the extent to which assessment design and development fosters coherence among the three vertices of the assessment triangle when learning progressions are claimed as the intended interpretation target for assessment results.  
 
The extent all design decisions and the rationales for them are explicit and transparent and collecting documentation to support them. Learning progressions, along with relevant empirical evidence supporting those theories and models, inform those decisions and correspondingly offer support for the interpretation and use of assessment results. Thus, part of this criterion is the strength of evidence for the learning progression including the research associated with the theory or model.
 
The degree of coherence between the observation vertex of assessment design and development and the learning progression within which the targets of inference are embedded.  Borrowing concepts from Evidence Centered Design (Mislevy & Haertel, 2006), the observation vertex should explicitly identify features of content that have been identified as effectively eliciting evidence of status with regard to the targets of inference.  These content features may be identified as either characteristic or variable. 
 
The degree of coherence between the evidence vertex of design and development and the targeted learning progression.  Again borrowing from Evidence Centered Design, the evidence vertex should address instructions for interpreting performance that consist of three parts: work product specifications, evidence rules, and the statistical model. Given that paths through learning progressions depend on learning experiences (e.g., formal and informal curriculum), part of this criterion is the role of learning experiences in the identification of relevant features of learners’ performances.
 
I will suggest several structural metrics that can be used to evaluate coherence between learning progressions and the features of content and the features of learner performance incorporated into the assessment.
 
In addition, I will present an argument for the kinds of empirical evidence that would be relevant to collect following test administration that might have the most relevance for evaluating coherence.

Author

• Paul D. Nichols, ACT, Inc.