Developing an Assessment Technology to Measure Knowledge Structures During Problem Solving

• In Event: New International Developments and Cross-Validation in Constructive Knowledge Assessment and Rerepresentation

Abstract

There is continuing emphasis on learner preconceptions and their impact on problem solving (Bransford, Brown, & Cocking, 2000; Spector, 2004). Developing assessment technologies to determine the quality of student problem conceptualizations and how they are affected by pre-conceptions and subsequent instruction is critical for systematically improving learning and instruction in complex domains (Spector, 2010). Previous efforts have involved the creation and analysis of annotated concept maps for this purpose (Ifenthaler & Seel, 2005; Pirnay-Dummer, 2007; Pirnay-Dummer, Ifenthaler & Spector, 2010; Spector & Koszalka, 2004; Taricani & Clariana, 2006). Other techniques involving probability networks and graph analysis have also been developed (see, for example, Shute et al., 2009).
Assessment of knowledge structures and how they are used to represent problem situations is the specific challenge for those seeking to developed advanced technologies to support learning in complex domains. Concept map techniques have often been used to represent a student’s knowledge structure as portrayed by propositional relations among concepts found in a body of text such as a student essay or response to a problem situation (Spector & Koszalka, 2004; Taricani & Clariana, 2006). Language plays a critical role in building and mediating individuals’ internal representations with the external world. It is assumed that using natural language enables concept maps to be more descriptive and akin to targeted mental models (Pirnay-Dummer, Ifenthaler & Spector, 2010), which helps to provide better instructional correctness and feedback suited to individual needs (Shute, 2007).
What is now needed is a way to determine which of these various techniques works best in different situations, and what modifications and new technologies might be required. This paper focuses on the first of these two concerns, and specifically on the annotations included with the annotated concept maps such as those creating using HIMATT (Highly Integrated Model-based Assessment Tools and Technogies; Pirnay-Dummer, Ifenthaler & Spector, 2010). A semantic relation analysis is used to compare different methodologies and then analyze the results in light of performance on complex problems. The analysis suggests that the elaboration of the semantic relationship network is critical for predicting problem solving performance, diagnosing learner misconceptions, and providing the basis for dynamic formative feedback.
This study is based on the belief that concept map as re-represented student’s mind can reflect certain states of student learning progress. The findings indicate that there are limitations to all of the assessment methodologies investigated, but they also show significant potential. What is missing is the ability to automatically perform a deep semantic analysis of two annotated concept maps (or two text responses to a problem situation). The current methods allow for generally reliable indicators of relative levels of learning understanding but they are not yet up to the task of providing specific and structured constructive feedback to guide systematic progress in solving complex problems.

Authors

• J. Michael Spector, University of North Texas

• Min Kyu Kim, University of Southern California