Cognitive Interviews on New Biology Reasoning Items: Inference, Yes, Test-Taking Strategies, No

• In Event: AERA Poster Session 13
  In Poster Session: Quantitative and Qualitative Approaches to Assess Students' Cognitive Performance

Abstract

We conducted cognitive interviews on new biology reasoning items, replicating a cognitive interview study with 'Form A' items on the same construct. Thirty-seven undergraduate biology students verbalized their thinking while answering the 15 'Form B' questions, which had shown good psychometric and DIF properties. Examinees' verbalizations were coded using an existing coding scheme. Of 41 codes, only 5 showed differences between questions answered correctly versus incorrectly. The targeted inferences were significantly different, providing strong validity support. One test-taking strategy (process of elimination) and one low-level strategy (search) also showed differences. On the whole, results strongly suggest that these new items do in fact tap reasoning with newly provided information, not reading or test-taking strategies, content or vocabulary knowledge.

Authors

• Jennifer G. Cromley, University of Illinois at Urbana-Champaign

• Aygul N Dane, University of Illinois at Urbana-Champaign

• Tia M. Fechter, Northeastern Educational Research Association

• Ting Dai, University of Illinois at Chicago

• Frank Nelson, Temple University

• Elizabeth Zelenka, University of Illinois at Urbana-Champaign

• Juhi S Shah, University of Illinois at Urbana-Champaign