Developing Computer Model-Based Assessment of Learning Progression

• In Event: Exploring the Frontiers and Linking Critical Aspects of Assessments in Science

Abstract

The study focuses on the development of a computer model-based assessment of learning progression during a high school chemistry course. The specific research questions for this study are: (a) What is the general process for developing a computer model-based formative assessment of learning progression? and (b) What evidence is available to support the claims on the validity and reliability of the assessment?
The theoretical framework guiding this study is Johnstone’s (1993) three components of chemistry (i.e. the macroscopic, the submicroscopic, and the representational) and the neo-Piagetian theories of cognitive development (Biggs & Collis, 1982; Case, 1985; 1992). We hypothesizes that a generalized chemical reasoning drives students’ development of understanding of specific concepts in a high school chemistry course. We further conceptualize that this generalized chemical reasoning consists of two closely related dimensions: matter/energy and models.

The development of learning progression assessment follows the construct modeling approach (Wilson, 2005; 2009). Both the progress variables on matter/energy and models consist of three levels. Students’ behaviors on each of the progress variables are based on their responses to computer model-based assessment questions. That is, students first interact with a computer model related to a topic, and then answer questions related to specific aspects of the model. This approach is based on ideas of Connected Chemistry (CC) (Levy & Wilensky, 2009a, 2009b; Stieff, 2004; Stieff & Wilensky, 2003). We identified the following ten most commonly taught topics in a typical high school chemistry course: atomic structure, periodic table, state of matter, solutions, gases, stoichiometry, chemical bonding, chemical equilibrium, redox, and acids-bases. Ten sets of computer models, with one for each of the above ten topics, were developed. Each set of computer models included at least one Flash animation of a relevant chemical phenomenon, and one NetLogo model. Assessment questions are in the format of Ordered Multiple-Choice (OMC) (Briggs, Alonzo, Schwab, & Wilson, 2006). Different from ordinary multiple-choice (MC) questions, choices in an OMC question represent different levels of a progress variable.

The initial version of each test was pilot-tested with one class of students (n=15-25) during the academic year 2009-2010. Three experts with expertise in chemistry, psychometrics and science assessment reviewed our 10 initial tests. The revised tests were then field-tested in three high school chemistry classes in three different schools during the academic year 2010-2011. The three high schools represented urban, suburban, and rural schools. Multi-dimensional Rasch modeling was conducted to examine the item and test qualities.

Overall, items fit the two-dimensional Rasch partial credit model well based on fit statistics. Item category structure based on step thresholds was also reasonable. There was a statistically significant correlation between the dimension of matter/energy and model. The spread of items overall matched the spread of students’ abilities on the two dimensions. Overall person reliability for the 1- tests was around 0.5, which was moderate. Qualitative data based on interviews of select students on their reasoning of the models and assessment questions also supported the Rasch modeling results on the validity and reliability of the tests.

Authors

• Xiufeng Liu, University at Buffalo - SUNY

• Noemi Waight, University at Buffalo - SUNY

• Roberto Ma. Gregorius, Canisius College