Assessing Students' Level of Understanding the Core Idea of Matter Using Ordered Multiple-Choice Items

• In Event: Challenges of Assessing Learning Progressions in Science: Item Design, Validation, and Measurement

Abstract

Learning progressions describe students’ progress in mastering a domain. Typically, LPs do so by defining levels that reflect an increasingly sophisticated understanding of the core ideas in a domain and the skills required to use this understanding to solve domain-specific problems (e.g. Author et al., 2009). Past work on assessing students’ understanding has mostly built on either open-ended items, scored based on rubrics reflecting different levels of understanding, or multiple-choice items, with each item assessing a different level of understanding. However, while open-ended items can provide deep insight into students’ understanding of a core idea, it takes more time to answer and score them. As a consequence, fewer items can be administered to a single student and larger resources are needed for analysis. Multiple-choice items are more efficient to administer and score, but provide less insight into students understanding. Also, multiple-choice items often yield conflicting results about students’ level of understanding when students answer similar numbers of items correctly and incorrectly on each level. Recently, ordered-multiple choice (OMC) items have been suggested as a means to overcome these limitations (e.g. Briggs, Alonzo, Schwab & Wilson, 2006). In OMC items each response option reflects a different level of understanding. Thus OMC items use the full diagnostic information available, which should yield more reliable and valid results. However, so far few studies have used OMC items to assess students’ understanding.
In this presentation we report about our efforts to develop OMC items to assess students’ understanding of the core science idea of matter. We report about a series of studies in which a) we developed an initial set of ten OMC items that were subsequently administered to N=300 students from grades 6-12 after students answered open-ended versions of these items, to examine the extent to which OMC items can yield results as reliable and valid as open-ended items, b) we developed a total of 44 OMC items to assess students understanding of four aspects of matter and administered these items to N = 1386 students from grades 6-12, c) we performed a think aloud study with N = 11 students from grades 6, 9 and 11 in order to explore the cognitive validity of OMC items.
The results suggest that OMC items can assess students’ understanding of matter as validly and reliably as open-ended items. In 78.8% of the cases, students chose the OMC response option that reflected the same level of understanding they exhibited in the open-ended item. We also found OMC items suitable for assessing different aspects of matter and to track students’ progression from elementary to high school. However, we also noted a particular variance in students’ level of understanding depending on the items’ context and – regarding cognitive validity – a particularly incoherent behavior of students from lower grades. In the presentation we will discuss issues of context and coherence in our data sets. In doing so we will also discuss the lack of statistical procedures to address these inevitable issues in designing assessments to investigate students’ understanding.

Author

• Knut Hartmut Neumann, Leibniz Institute for Science and Mathematics Education