Statistical Analyses of Performance on the Concept Assessment Tool for Statics and the Thermal and Transport Concept Inventory: Foundations of Inventory Validity and Utility

• In Event: Evaluating and Improving Concept Inventories as Assessment Resources in STEM Teaching and Learning

Abstract

Purposes & Perspectives: This presentation reviews statistical analyses of data from the CATS and the TTCI and interprets them in terms of claims regarding interpretive uses of each inventory. By evaluating sources of evidence as confirmatory or disconfirmatory of claims about what an inventory is intended to measure and the meaning of the inventory score(s) one establishes elements of a validity argument for the instrument. Three kinds of claims may be supported by CIs: (a) claims about students’ overall proficiency within the conceptual domain defined by the developers; (b) claims about proficiencies with separable concepts, which can be expressed as a diagnostic profile of proficiencies; and (c) claims about incorrect thinking relative to a specified list of common errors or misconceptions. Concept proficiency profiles are required to support formative assessment applications in STEM classrooms. Although nearly all CIs claim to assess an identifiable set of concepts, it is rare to find one with strong evidence regarding its diagnostic capacity with respect to individual concepts.
Methods and data sources: We present the results of multiple analyses of student performance on the CATS and the TTCI, including applications of classical test theory, item response theory, exploratory and confirmatory factor analyses, and diagnostic modeling (Pellegrino et al., 2013). The separate and collective results are germane to various sets of claims about the interpretive uses of a given CI.
For CATS, multiple indicators of strong diagnostic capacity were obtained (Jorion, et al., in preparation). For example, most of the developer-defined subscales demonstrated reasonable reliability. Exploratory factor analyses provided support for the developers’ nine concepts and confirmatory factor analyses were consistent with a modified list of the developers’ concepts. A diagnostic classification analysis showed strong support for reliable reporting of ten diagnostic “skills” derivable from the developers’ nine concepts.
The TTCI also revealed several desirable measurement properties. Evidence suggested that individual questions appeared functioned well and were aligned with the proposed conceptual structure of the instrument. We were able to generate a reasonable set of “skills” for diagnostic reporting on the TTCI and an associated Q matrix indicating which skills were required for each question. However, the numbers and types of questions were not sufficient to allow a test of the hypothesized Q matrix diagnostic structure.
Scientific and scholarly significance: This presentation describes a general analytical framework that could be adapted for conducting a validity analysis of any CI and illustrates application for two existing CIs. A key to the formative use of a CI is its capacity to provide useful and reliable diagnostic profiles of students’ conceptual proficiencies. The comparison between the diagnostic success of the CATS and uncertainty for TTCI due to the lack of numbers and balance of items to support diagnostic modeling provides a springboard for application of the Evidence-Centered Design approach described in presentation two. ECD supports developing a diagnostic framework for the TTCI useable for developing an expanded TTCI that is the subject to further validation work.

Authors

• Natalie Jorion, University of Illinois at Chicago

• Katie James, University of Illinois at Chicago

• Louis V. DiBello, University of Illinois at Chicago

• James W. Pellegrino, University of Illinois at Chicago