Toward the Use of Concept Maps in Large-Scale Science Assessments

• In Event: Technical Issues in Performance Scoring

Abstract

Purpose
This study explored two concept map scoring methods for potential usage in large-scale assessments. The goal of this research was to understand whether scores based on all of the propositions in a concept map (a holistic approach) could be exchangeable with a more economical approach in which only the propositions deemed critical to the topic were considered.

Theoretical Perspective
Concept-mapping is a potential assessment task that taps an ability that is difficult to measure by other means: student understanding of the interrelatedness of concepts (or students’ organization of knowledge). Concept maps have been considered a complex item because of the cognitive demands placed on students and for their difficulties in their implementation (NAEP, 2009), and, potentially, in their scoring. Although computers can provide potential solutions to a variety of practical challenges posed by complex assessment exercises, some scoring issues cannot necessarily be solved technologically.

Methods
The data analyzed were part of a larger study involving six seventh-grade classrooms (see SEAL, 2003) in which students constructed two concept maps as embedded assessment: one towards the beginning and one towards the end of a science unit focusing on buoyancy. Students received a 15-minute training session, followed by a 50-minute session in which they drafted and finalized a concept map using the nine concept terms that were provided. A total of 130 student maps from occasion one and 81 from occasion two were analyzed.

Each potential proposition (i.e. a link between two concept terms, n = 36) was classified as either Critical or Non-Critical based on an analysis of the curriculum and several experts’ concept maps. All of the actual student-generated propositions (n = 868) were then compiled into a proposition inventory and subsequently scored (IRR = .87) using a 4-point rubric based on accuracy and completeness (adapted from Ruiz-Primo and Shavelson, 1996; Yin et. al, 2005).

Results
Findings indicate that the critical proposition score cannot be substituted for the total proposition score simply for the sake of economy. However, an argument can be made for the critical proposition scoring approach on the grounds that it appears to be more sensitive to instruction. The effect size for the critical scoring method across the two occasions (.56) was larger than the effect size for the scoring method that involved scoring all propositions (0.07). In addition, tests of significance across occasions indicate that the change for the total proposition score was not statistically significant (p = 0.44), whereas the change in the critical proposition scores were (p = 0.00). One explanation for this phenomenon relates to the number of accurate but non-critical links on a map. For example, students are likely to be able to come up with an accurate propositional link between the concepts “matter” and “wood” without relying on the concepts of buoyancy and density.
 
Scientific Significance
This study provides empirical evidence about concept map scoring approaches. This information is critical to decisions regarding the implementation and scoring of large-scale concept mapping assessments.

Authors

• Heidi Kroog, University of Colorado - Denver

• Maria Araceli Ruiz-Primo, University of Colorado - Denver

• Yue Yin, University of Illinois at Chicago