Choice-Based Assessments

• In Event: Roundtable Session 62
  In Roundtable Session: New Approaches to Validity Judgments for Emergent Research Methods

Abstract

Perspective. Educational assessment is a normative endeavor: The ideal assessment both reflects and reinforces educational goals that society deem valuable. A fundamental goal of education is to prepare students to act independently in the world—which is to say, to make good choices. It follows that an ideal assessment would measure how well we are preparing students to do so. The argument of this presentation is that current assessments, which primarily focus on how much knowledge students have accrued, are inadequate. Learning relevant choices, rather than knowledge, should be the construct around which assessments are organized.

Methods. Choice-based assessments present an interesting validity problem. With standard factual and procedural tests, there is little question whether a given answer is right or wrong. However, people can rightly ask how we know that some choices are better than others. We have developed a number of techniques for generating evidence that a choice-based assessment is measuring what we believe they are. For example, given a specific assessment, we “induce” learners into one of two patterns of choice (e.g., using points or tips), and we compare the learning outcomes on a posttest. This experimental method allows us to say which pattern of choice is better for what outcome. We then switch to an individual differences approach, where we use the same assessment without the inducements to evaluate what students choose to do. Finally, we can use convergent validity to show that the assessment is relevant to more traditional outcome measures in school. For example, using one of our assessments, we found that the choice to persist at games predicted 33% of the variance in student science achievement.

Significance. A second major challenge is that practitioners and researchers do not have easy ways to produce and evaluate candidate assessments. This is a problem for the field of learning in general -- investigators will often implement an excellent instructional treatment that fails to show effects because of insensitive, though creative, learning measures. This is not simply a problem that items are too hard or too easy. It can result from a specific word, image, or instruction that might tilt students in unintended ways. Even if one follows an optimal design logic (e.g., Evidence-Centered Design, Mislevy & Haertel, 2006), there is still a legion of micro-details that can only be determined empirically. Arguably, this is one reason that so many assessments focus on factual recall and procedural application – they provide well-worn assessment “scripts” that researchers and students know well. It needs to be easier to produce and progressively improve creative learning assessments.

Results/Outcome. We will describe a crowd-sourcing platform that we have created that makes it possible for researchers to develop on-line, game-like choice-based assessments and have thousands of people play ‘for fun.’ This makes it possible to rapidly and iteratively refine the assessments and determine their psychometric properties.

We will present our rationale for choice-based assessments, including examples of our methods and assessments.

Author

• Daniel L. Schwartz, Stanford University