The Use of Evidence-Centered Design to Reverse Engineer an Existing Concept Inventory and Develop New Questions for Diagnostic Reporting

• In Event: Evaluating and Improving Concept Inventories as Assessment Resources in STEM Teaching and Learning

Abstract

r testing deeper understanding of important and challenging concepts within a typically narrow domain. For example the TTCI (Thermal and Transport Concept Inventory) claims to measure: Rate Processes, First Law of Thermodynamics/Energy Balance, Heat Capacity/Stored Energy, Thermal Equilibrium vs. Steady-State, and Sensation of Hot/Cold. The primary objective of this presentation is to discuss the application of an Evidence Centered Design (ECD) procedure to construct an expanded version of the TTCI that is capable of high quality diagnostic reporting.
Perspectives and Methods: An extensive study of many CIs reveals that beyond properties of individual questions and reliability of total score, most inventories have not provided compelling evidence that supports reporting of diagnostic profiles of individual concept proficiencies (Pellegrino et al., 2013; Jorion et al., in preparation). Previous research indicated that the questions on the current version of the TTCI were of high quality but were too few to support diagnostic reporting (op. cit.). A key question for this research was how to build upon the strengths of the TTCI and expand it to include a sufficient number and breadth of high quality items capable of supporting reliable diagnostic reporting of individual concept proficiencies. We used the existing high quality TTCI questions and its framework of heat concepts to reverse-engineer an ECD design template that articulated the claims intended to be supported by an expanded TTCI and specified the intended focal knowledge and misconceptions to be measured. The template was then used to develop new questions and distractors that were hypothesized to measure each of the concepts and misconceptions.
Results: A new expanded TTCI form was developed and subjected to both qualitative analyses through ( described in presentation #3) and quantitative analysis of a large sample of student performance data on the new TTCI form (described in presentation #4). This presentation provides concrete examples of the design template, describes the expanded TTCI form, and compares the new and earlier TTCI forms to demonstrate why the improved design was expected to provide better diagnostic measurement.
Scientific and scholarly significance: Concept inventories typically result from extensive effort to identify important, cross-cutting and difficult concepts within a given domain. CIs also develop questions and multiple choice distractors by using the research literature and open ended questions for students. Until now there has been little attention to the diagnostic utility of existing inventories. Existing research (op. cit.) has shown that even when total score reliability is reasonably high, CIs are not able to support reliable reporting of diagnostic profiles of individual concept proficiencies for classroom formative use by instructors and students. The approach of using ECD to reverse engineer a design template and then to work forward from that design template to develop new questions and distractors can be adapted to any CI with a well-established conceptual foundation. This has the potential to transform CIs by improving their diagnostic utility and thereby fitting them for formative use within STEM classrooms by instructors and students.

Authors

• Ronald L. Miller, Colorado School of Mines

• Dana Denick, Purdue University

• Ruth A. Streveler, Purdue University

• James W. Pellegrino, University of Illinois at Chicago

• Louis V. DiBello, University of Illinois at Chicago

• Lianne Schroeder, University of Illinois at Chicago

• Katie James, University of Illinois at Chicago

• Natalie Jorion, University of Illinois at Chicago