Developing a New Generation of Concept Inventories to Increase Diagnostic Capability

• In Event: Sustaining Interest in and Preparing Students for Successful Transitions to Professional Programs of Study: Insights From NSF (National Science Foundation)-Supported Engineering Education Initiatives

Abstract

The use of concept inventories (CIs) has recently become fashionable in engineering education. A variety of concept inventories have popped up that have been developed with varying degrees of rigor. Even the most developed concept inventories, the Concept Assessment Test for Statics (CATS) and the Thermal and Transport Concept Inventory (TTCI) are still limited by the right or wrong nature of their present construction. This dichotomous structure does not allow for investigation of shades of right or wrongness and therefore does not provide insights into potential learning trajectories.

This presentation will discuss how one REESE project (Collaborative Research: Integrating Cognition and Measurement with Conceptual Knowledge: Establishing the Validity and Diagnostic Capacity of Concept Inventories) is trying to change this current limitation by developing a new generation of CIs that will provide more diagnostic information to both students and instructors. Using new analysis techniques, we are now in the process of discovering what “skills” or “understandings” are needed to correctly answer any particular concept inventory question. So instead of only knowing that a student answered a question correctly or incorrectly, an instructor administering the new generation of CIs could see a picture of what underlying understandings are missing among students. This will result in profiles of students that can help instructors diagnose, at a more fine-grained level, where more instructional invention is needed.

The presentation will particularly focus on two aspects of the project. First, findings of the project will be briefly presented. Second, and perhaps more importantly for the cross-professional Division I audience, the use of the “Assessment Triangle” as a framework for development of the new generation of concept inventories will be discussed. The Assessment Triangle is an aid for aligning ideas and theories about student cognition in the target (the “cognition corner” of the triangle), with the analysis method (“interpretation” corner), and the construction of the concept inventory itself (“observation” corner). The Assessment Triangle can be used in the development of any measurement instrument in any profession.

Author

• Ruth A. Streveler, Purdue University