Qualitative Analysis of Students’ Conceptual Reasoning

• In Event: Evaluating and Improving Concept Inventories as Assessment Resources in STEM Teaching and Learning

Abstract

Objectives: The research question for this study was: How does student reasoning about heat transfer align with intended TTCI focal knowledge and identified student misconceptions? This presentation describes think aloud and interview studies of undergraduate engineering students solving TTCI heat transfer problems (see presentation #2) as a means of collecting validity evidence for developers’ claims about the nature of student reasoning elicited by individual questions and distractors. We compare TTCI findings to protocol study findings for the Concept Assessment Tool for Statics (CATS) as reported in (Denick, Santiago-Roman & Streveler, 2013).
Perspectives and theoretical framework: Guided by Minstrell’s work on facets of understanding, this research is grounded by the notion that the diagnosis of students’ conceptual understanding through questioning can help instructors design more targeted and meaningful instruction. Interviews, focus groups, or think-aloud strategies were specifically recommended as means of diagnosing student misconceptions
As described in presentation #2 an Evidence-Centered Design (ECD) approach was taken for designing new TTCI items and multiple choice options, to try to strengthen the diagnostic capability of the TTCI. The ECD design template that was developed to express the underlying focal knowledge and misconceptions intended to be assessed by a revised TTCI was used to help design protocol and follow-up questions and to inform analyses of the TTCI protocol data.
Methods, data sources, and materials: Protocol data will be collected by interviewing approximately 40 undergraduate engineering students during the summer and fall of 2013. Researchers will prompt students to explain their reasoning as they work through selected items designed for the TTCI. Students also will be asked to describe their reasons for not selecting alternate responses. An iterative questioning approach will encourage students to respond to the TTCI items initially without extensive prompting, and then again with more interviewer questioning. The method for analyzing protocol data is guided by Chi’s work on verbal analysis (ref?) in which a systematic approach is taken to coding verbal data, operationalizing evidence from the coded protocol, and identifying patterns within the data and codes for interpretation. The protocol observations and interview process and the analysis procedures will follow the same methodology that was used successfully during 2012 and 2013 with CATS (Concept Assessment Tool for Statics) data. We expect that this will limit potential pitfalls in the data collection and analysis stages of the study.
Sample results: We will report analyses of verbal evidence from the student interview transcripts to determine the extent to which students’ engagement with TTCI questions elicited both expected and unexpected conceptual or misconception thinking. In similar studies, we found evidence of both expected and unexpected conceptual and misconception thinking (Denick, Santiago-Roman & Streveler, 2013). We expect similar findings for the TTCI work. For example, as was found from the CATS protocol studies, protocol evidence may suggest that novice reasoning regarding the structure of TTCI items may differ from experts’ expectations and the focal knowledge that experts expect for a specific TTCI item may differ from the reasoning exhibited in student responses for that item.

Authors

• Dana Denick, Purdue University

• Ruth A. Streveler, Purdue University