Establishing Principles for Design-Based Psychometric Theory

• In Event: Roundtable Session 62
  In Roundtable Session: New Approaches to Validity Judgments for Emergent Research Methods

Abstract

Objectives
[a] To provide a critique of the psychometric criteria used for measures in design-based research studies.
[b] To spur community debate on the viability of traditional criteria for judging assessment validity and discuss emerging alternative validity criteria for design-based research studies that recognize the emergent and fluid nature of the constructs studied.

Perspectives
Nitko and Brookhart (2006) identified several sources of evidence for establishing valid and reliable measures, including: (a) Content evidence, (b) Substantive evidence, (c) Internal structure evidence, (d) External structure evidence, (e) Reliability evidence, (f) Generalization evidence, and (g) Consequential evidence.

Evidence for each criterion assumes and depends upon on some pre-established construct against which progress toward learning the construct is gauged.

By contrast, for design-based research, each criterion is open to challenge since the hypothesis about what novel content is teachable, and learnable, is itself an object of research in open-ended, diffuse, fluid, and highly challenging learning contexts.

A revision of validity criteria are necessary since: (a) Subject matter experts’ models of their domains do not adequately capture learning progressions, an important aspect of many design studies (Cobb et al., 2003; Lamberg & Middleton, 2009; Roschelle et al., 2008), thus challenging traditional conceptions of content evidence; (b) Students’ responses to iterative interventions modulate the appropriateness of cognitive challenges, thus impacting traditional conceptions of substantive evidence; (c) Teachers’ dynamic input on the teachability and appropriateness of learning constructs impact criteria for what both internal and external structure evidence; (d) retrospective researcher analyses of instructional interactions can lead to significant changes to the goals and direction of the research (e.g., Cobb et al., 2003; Lesh, Kelly & Yoon, 2008) impacting judgments of reliability evidence; and (e) design research studies tend to be non-representative of normal learning conditions and situations (Fishman et al., 2006) thus questioning criteria for generalization evidence, and consequential evidence.

Methods & Data Sources
Our methods comprise a review of studies in high-impact journals publishing design-based research studies on how they addressed each of the above criteria for assessment validity. In addition to documenting how evidence of learning was treated, we will incorporate the new models of validity being proposed by the other symposium panelists.

Results
This work provides a timely revision of traditional assessment criteria through critique and analysis that will provide the basis for community-led discussions about new criteria that respond to the dynamic and iterative qualities of design-based research.

Significance for Validity and Policy
In response to global economic crises, OECD and other countries are asking researchers in the learning sciences and advanced education technology to deliver research results on innovation in science, technology, engineering and mathematics (STEM) content. In addition, massive investments in the US Race to the Top Assessment consortia (approx. $320M) expect transformative results with novel and experimental methods. Design-based researchers must, therefore, reexamine evidence-based criteria for their work to assure peers, policymakers and practitioners that the evidence for their claims about learning is on the soundest possible footing.

Authors

• Anthony E. Kelly, George Mason University

• John Y. Baek, National Oceanic and Atmospheric Administration

• Brenda Bannan, George Mason University

• Patrick Shane Gallagher, Advanced Distributed Learning