Measuring Knowledge Integration in Science

• In Event: Exploring the Frontiers and Linking Critical Aspects of Assessments in Science

Abstract

The purpose of this paper is to report on the validation of assessments created to measure science knowledge integration ability through a large-scale four-year study (Liu, Lee, Hofstetter, & Linn, 2008).

The advancement of science education requires efficient and effective assessments. Many science assessments fall short of this goal by focusing on isolated scientific facts without capturing coherent science understanding. The knowledge integration framework, developed by Linn and colleagues (Linn & Hsi, 2000; Linn, Lee, Tinker, Husic, & Chiu, 2006), represents constructivists’ view of how science knowledge can be acquired and practiced. The KI framework was used to guide the development and scoring of the assessments used in this study.

Assessments were developed covering six science domains: middle school life, earth, and physical sciences and high school chemistry, biology and physics. Besides measuring content knowledge, the assessments also aimed to capture student ability of justifying their answers using scientific evidence. The scoring of the assessment differentiated between the different levels of knowledge integration understanding. The assessments were designed in both multiple-choice and constructed-response formats.

The assessments were administered to 18,729 students taught by 340 teachers in 21 school districts in five states. Rasch partial credit analyses (Wright & Masters, 1982) were applied to evaluate the psychometric properties of the assessment including item fit, item difficulty, test reliability, and person reliability. The software ConQuest was used to perform the analysis (Wu, Adams, Wilson & Haldane, 2007). We also examined the impact of student characteristics and test delivery mode on student science performance in knowledge integration through multiple regression analyses.

The assessment items showed satisfactory psychometric properties. Results showed that, when paired with multiple-choice items and scored with an effective scoring rubric, constructed-response items can have satisfactory reliabilities. Analyses showed that students’ English language learner status and computer use significantly impacted their science knowledge integration abilities. Students who took the assessment online also performed significantly better than students who took the paper-and-pencil version of the assessment.Implications and future directions of research are noted, including refining curriculum materials to meet the needs of diverse students and expanding the range of topics measured by knowledge integration assessments.

This study has significant implications for research on science education, especially on assessments designed to measure integrated science learning. When evaluating student performance, many factors should be considered including the assessment characteristics and student backgrounds. The study also shows that, when appropriately designed, knowledge integration assessments can be balanced between validity and reliability, authenticity and generalizability, and instructional sensitivity and technical quality.

Author

• Ou Lydia Liu, ETS