Quantitative Analyses of Student Performance on Concept Inventories

• In Event: Evaluating and Improving Concept Inventories as Assessment Resources in STEM Teaching and Learning

Abstract

Objectives: The objective of this presentation is to discuss a variety of structural and diagnostic analyses as a means of collecting validity evidence for the diagnostic capacity of CIs. We discuss diagnostic analyses of student performance data from the CATS and the TTCI that are informed by the conceptual foundations of each inventory as well as by the findings from qualitative studies of the two inventories as described in presentation 3.
Perspectives and Methods: Diagnostic modeling determines whether a CI is capable of reporting reliable diagnostic outcomes that would be useful for formative purposes in classrooms. New diagnostic analyses will be reported for both the CATS and the TTCI that take account of the findings from the qualitative studies discussed in presentation #3.
Sample results and conclusions: Complete results will be reported on both inventories.
Findings: We will review previously reported CATS findings as a background for new diagnostic analyses. We analyzed 1,372 cases of student performance data on CATS and a sample of 542 students for TTCI (Pellegrino et al. 2013; Jorion et al., in preparation) and reported Cronbach alpha for the total score. Exploratory and confirmatory factor analyses performed on the CATS data tended to confirm developers’ claims regarding concepts being measured with some changes (op. cit.). For example the best fitting confirmatory factor analysis was found to consist of five of the nine concepts, plus two new latent variables, each of which was the combination of a pair of the original nine concepts (see Figure 1). We also reported a diagnostic analysis of the CATS (Santiago-Roman, 2009) relative to a set of ten identified “skills” that showed strong diagnostic strength. The skills required by each item were represented by a Q matrix (op. cit., 2009).
New diagnostic analyses of the CATS and the expanded TTCI (presentation #2) will be reported that take into account findings from the qualitative analyses described in presentation #3. For example, for the CATS those findings included evidence of some unexpected skills in certain questions and the absence of some expected skills in some questions. The new diagnostic analyses will be based on a revised Q matrix that reflects the qualitative findings.
We will collect data on the new version of the TTCI from about 500 students and will report the same factor analyses and diagnostic modeling of the new TTCI. A comparison will be reported between TTCI and CATS findings.
Scientific and scholarly significance: Concept inventories have the potential to help improve teaching provide they can demonstrate a strong diagnostic capacity for providing information to instructors and students. This study describes an analytic framework for CI data that is informed by the conceptual foundations of a given CI and also takes into account qualitative findings as described in presentation #3. We believe the coordinated approach of using ECD methods to develop a CI questions and combining qualitative and quantitative analyses is broadly adaptable to any CI with a well-established conceptual foundation.

Authors

• Natalie Jorion, University of Illinois at Chicago

• Katie James, University of Illinois at Chicago

• Lianne Schroeder, University of Illinois at Chicago

• Louis V. DiBello, University of Illinois at Chicago

• James W. Pellegrino, University of Illinois at Chicago