Questioning the Assessment of Mental Models: Why Written Texts and Knowledge Maps Measure Different Things

• In Event: New International Developments and Cross-Validation in Constructive Knowledge Assessment and Rerepresentation

Abstract

Over the past decades, various researchers have contributed to the further development of the theoretical foundation of mental models (e.g., Johnson-Laird, 1983; Seel, 1991) and also to their application in instruction (e.g., Anzai & Yokoyama, 1984; Seel, 2003). However, one essential question concerning the assessment of mental models is that as to which methodology should be used. Many authors consider graphical representations (e.g., concept maps, knowledge maps) to be an adequate format of externalization for analyzing complex knowledge structures (Jonassen & Cho, 2008). On the other hand, there are strong arguments indicating that natural language representations are a good basis for assessing mental models (Ifenthaler, 2010; Pirnay-Dummer & Ifenthaler, in press).
The main objective of this study is to track learners’ progression while using graphical and language-based externalization techniques and compare the strengths and weaknesses of these assessment methods. Specifically, we will compare the similarities and differences of the graphically and textually elicited knowledge structures and their semantics. Even through a close assessment and a short time between the externalization of a graphical and written re-representation of a phenomenon in question, both assessments are expected to re-represent different structural and semantic features. Accordingly, we assume that the graphical (knowledge maps) and written (written text) re-representations correlate with r ≤ .7 (Hypothesis 1).
Forty-seven students (31 female and 16 male) from a European University participated in the study. Their average age was 25.7 years (SD = 1.7). The study was realized as a design experiment. It took place in a research methods course and consisted of (1) a 60 minute introductory phase, (2) a phase of learning with five learning tasks, and (3) a phase for assessing the final task performance. The participants were asked to create a graphical representation (knowledge map) and a text-based explanation (written text) relating to their understanding of research skills. In total, five identical task explanations had to be uploaded at a specified date and time during the course. To analyze the participants’ graphical representations and written texts, we used the seven measures implemented in HIMATT (Pirnay-Dummer, Ifenthaler, & Spector, 2010).
We tested correlatives between graphical and text-based re-representations. Although all α and means of similarities are statistically significant, the overall similarities are by far not sufficient to accept the alternative hypothesis. The measures of the graphical re-representations (knowledge maps) and the measures from the text-based re-representations (written text) correlate with r ≤ .7. This means that text re-representations and graphical re-representations are attached to different constructs and therefore measure different things.
The fact that graphical notes and texts re-represent different things (even when used in the same task with the same participants) does not necessarily lead to the conclusion that either assessment is obsolete. The graphical re-representation is structurally more concise, while the text-based re-representation is better on the semantic measures. Accordingly, graphical representations are better for rebuilding a concise structure (Ifenthaler, in press). On the other hand, textual re-representations have a higher information density; they contain more information and therefore more semantic terms match.

Authors

• Dirk Ifenthaler, Open Universities Australia

• Pablo Nicolai Pirnay-Dummer, University of Freiburg