Using Text Analysis and Clustering to Reveal Processes of Change in Engineering Knowledge and Innovation

• In Event: 72.065 - Roundtable Session 30
  In Roundtable Session: 72.065-7 - Computational Methods for Qualitative Learning Sciences Research: Affording Materiality to Text-Based Data

Abstract

Relatively little is understood about the practice of teaching for adaptability. Our prior research demonstrates that challenge-based instruction (CBI) can foster Adaptive Expertise (AE). AE is a combination of 1) traditional routine expertise as defined by content knowledge and its correct application and 2) the skills and habits to use that knowledge in new ways on novel problems. The experiments reported in this symposium investigate the hypotheses that 1) both the nature of the learning environment (adaptive or routine) and the level of expertise of the learner (adaptive or routine) are important for understanding the development of innovation and efficiency, and 2) innovation and efficiency co-develop in the CBI environment.

In Experiment 1, we examined students’ exam performance, coding for both innovation and efficiency, across a complete implementation of the CBI biotransport course. We assessed students’ level of innovation and efficiency on routine and adaptive problems at four time points at the grain size of exam questions. In Experiment 2, we interviewed eight students three times over the course of their biotransport class. Students solved the same heat transfer problem on each interview, talking aloud as they solved the problem. This design enabled the use of text analysis and clustering methods to examine how students’ use of innovation and efficiency changed over the course of the interviews in small time segments.

Both experiments demonstrated gains in innovation and efficiency, and appropriate use of them given the context of the task. Though the high level results in Experiment 2 do not show improvement on efficiency and innovation between Interviews 1 and 2, the fine-grained results do show improvement on each interview. The high level results were based only on final answers, and this difference shows the value of conducting the more detailed microgenetic learning analytics methods. The high level results appear to show that content knowledge was necessary for improvement on innovation and efficiency. Instead, the detailed results show how the GI method can improve both innovation and efficiency even before students have the required content knowledge. Students make more progress on the problem, even if they are not yet solving it correctly.

Regarding our second research question about developmental trajectories for innovation and efficiency, in Experiment 2, we have clear evidence they develop together. Students repeatedly switch from innovative to efficient clusters. Innovation could sound like what one does when starting out on a problem, while efficiency could be more about completing problems, but in this study, results demonstrate both are used throughout problem solving.

Authors

• Taylor Martin, O'Reilly Media

• Philip Janisiewicz, The University of Texas - Austin