Employing Distracter-Driven Assessments in Measuring Teacher Knowledge and Effectiveness

• In Event: Diagnostic Assessment: Recent Advances From Psychometric Modeling to Classroom Applications

Abstract

Objectives

This study examines the relationship between teacher knowledge and gains in student learning in the context of middle school physical science. The study uses distracter-driven assessment to measure teacher knowledge.

Framework

Teacher knowledge is a key prerequisite for student learning. Proxy measures of teachers’ subject matter knowledge (SMK) (degrees, courses, grades, exams, certifications, etc.), while commonly used, have been found to be poor predictors of student achievement (Mitchell, 1985; D’Agostino & Powers, 2009; Wilson, Floden, & Ferini-Mundy, 2002). A direct measurement of SMK would require that teachers take and perform well on tests tailored to a particular course’s content.

A teacher’s knowledge of common student misconceptions that make learning a concept difficult is hypothesized to be crucial to effective teaching (Ausubel, Novak & Hanesian, 1978). This knowledge is one aspect of a teacher’s pedagogical content knowledge (PCK) (Shulman, 1986). Research falls short in evaluating science teachers' knowledge of student misconceptions and the impact of that knowledge on student learning. Studies that rigorously investigate the relationship between the different kinds of teacher knowledge and student gains in understanding of science are rare (Baumert, et. al., 2010).

Methods

An assessment was previously designed to measure the National Science Education Standards in grades 5-8 (Sadler, et. al, 2009). The assessment used distracter-driven items targeting misconceptions well-documented in science education literature (Sadler, 1998; Schoon, 1988; Treagust, 1986). The current study used an instrument comprised of 20 of these items.

Teachers completed the assessment and were asked both to select the correct answer and to identify the incorrect answer most commonly chosen by students. Teachers administered the assessment to their students in a pre-test, mid-year test, and post-test. A hierarchical logistic regression model was used to evaluate the relationship between teacher knowledge and student learning.

Data

Data was analyzed from 181 seventh and eighth grade physical science teachers. These teachers administered tests to 9,556 students. A demographic description of participants and schools will be presented.

Results

Descriptive statistics of student and teacher performance will be presented, including teachers’ SMK and PCK. The complete results of hierarchical logistic regression analysis will be presented. To summarize, significant differences in student gains were associated with teacher SMK knowledge. Further, for items with prevalent misconceptions, there were significant differences in student gains associated with teacher PCK knowledge. Teachers who knew their students’ most common misconceptions were more effective than teachers who did not.

Significance

This paper contributes to the small base of research relating teacher knowledge to student gains in understanding. Knowledge of student ideas and misconceptions may be the critical component that allows teachers to construct effective lessons. This has significant effects on the development of professional development programs. An emphasis on increasing teachers’ SMK, without sufficient attention to the preconceived mental models of students and teachers, may be ineffective in ultimately improving students’ knowledge.

Further, pedagogical content knowledge of teachers has been notoriously difficult to measure. This study demonstrates an easy way to measure teachers’ awareness of mental models of students using a distracter-driven assessment.

Author

• Philip M. Sadler, Harvard University