A New Measure of Conceptual Knowledge: The Concept Recall Test

• In Event: 56.063 - Problem-Based Learning in Secondary Education: Investigating Hot and Cold Learning Outcomes

Abstract

The purpose of the present study is the proposal of an alternative assessment instrument with the potential to overcome the shortcomings faced by educators when using conventional assessment instruments. Historically, classroom assessment has been defined by the use of multiple choice and open-ended formats, which are both deeply flawed in their ability to generate a valid representation of student knowledge and understanding. The shortcomings of these conventional test formats demonstrate the need for the Concept Recall Test (CRT) as a more effective assessment instrument in contemporary education. The CRT is based on the idea that knowledge, stored in long-term memory, can be represented in the form of concepts that are linked with each other. These interlinked concepts form semantic networks that are representative of a person’s knowledge. While learning, students develop over time richer and more tightly integrated semantic networks of concepts of a domain. The better integrated and the more comprehensive these semantic networks are, the more concepts will be recalled. Put simply, a student with relatively more knowledge will be able to freely recall more concepts than a student with relatively less knowledge because he or she has a more extensive semantic network. Although this assessment measure seems simplistic, it is based on a student's ability to recall concepts instead of merely recognising them.
The primary aim of this study was to establish the construct validity of the CRT as a valid tool in assessing knowledge. The reliability was established by examining the inter-rater agreement (Cohen’s kappa) between two independent raters. In addition, it was examined how generalizable the reliability evidence is by examining the inter-rater agreement across different subject domains and age groups. To establish the validity of the CRT, convergent validity evidence was generated by correlating the scores of the CRT with test scores based on conventional examinations. In addition, an experiment was conducted in which the amount of knowledge students were able to acquire was manipulated during two experimental phases. During phase 1, participants discussed a problem about ‘infectious diseases’ where it was hypothesized that participants would not gain a significant amount of new knowledge. During phase 2, group 1 undertook self-study relating to the topic of ‘infectious diseases’ whilst the control group (‘group 2’), on the topic of ‘evolution’. During this phase, it was expected (and demonstrated) that members of group 1 would gain a significant amount of new concepts, subsequently represented by a significant increase in the third CRT score. The results of the study therefore demonstrate that the CRT is a reliable and valid instrument to measure conceptual knowledge. Future areas for research should include exploring a process of automation that further enhances the ease of administration and its ability to provide a timely diagnosis of the participant learning, reducing the time taken in the scoring process. Moreover, the possibility of automation further enhances the potential desirability and utility of this measure an option that is unlikely with concept maps given their complex construction and scoring processes.

Authors

• Gavin Hays, Parramatta Catholic Education Office

• Kurt Challinor, Parramatta Marist High