Conceptualizing and Measuring Teaching Quality and Effectiveness

• In Event: 73.023 - Toward a Fair Evaluation of Teachers: Methodological Challenges in a Cross-National Study of Mathematics Teachers

Abstract

Objectives

The objective of this paper is to explain the conceptual underpinnings of an alternative model to assess teacher quality and the measurements we have developed to do so.

Theoretical framework

This study begins with the assumption that measuring teaching quality depends on many factors (Darling-Hammond, 2010). In our study, we define quality as the ability of the teacher to help all students be successful. We consider the ability of teachers to effectively manage four challenges as comprising what we call the Teachers’ Advanced Proficiency Index: (a) different student populations, (b) school context (i.e., rural/urban, resources, past performance, availability of instructional materials), (c) curriculum characteristics and requirements (centralized, degree of flexibility to address diverse students’ needs), and (d) teachers’ professional knowledge (mathematical knowledge for teaching), and practices.

Methods

The methods are an alternative to the traditional and limited production function commonly used in value added models to evaluate teachers (Sanders & Horn, 1994). It relies on defining the level of challenge of students by working with the teachers within a country to identify the indicators of student diversity, broadly defined. The indicators can be used to create a scale of challenge for students based on item response theory methodology, or regression methods to connect the challenge indicators to performance in classes (Author, 2009). The location of students on a challenge scale indicates the amount of challenge posed by students to teachers. This numerical scaling can be used as a difficulty index for the student. This method proposes considering students assigned to a teacher as a “test” with the challenge index as the difficulty of the test items in the test. A second important component is that students must be judged as being successful or unsuccessful relative to the desired educational outcomes. The judgments are the scores for the students as test items. These scores along with the indicators of student challenge as difficulty parameters for the students can be used in the item response theory model with students as items to estimate the location of teachers on a teacher quality scale. The measure of success or lack of success can be based on a test with standards of performance or by other judgments of student competence. From this information, each teacher can be placed on a country scale of teaching quality.

Data sources

Data sources come from assessments of teacher knowledge, analysis of curriculum materials, a teacher questionnaire, interviews, and an observation, and individual pupil data including assessments and relevant demographics.

Results

This approach has given results that are more sensitive to the school context, the curriculum teachers are expected to teach and to pupils’ diverse backgrounds and learning styles.

Scholarly significance

This approach requires only limited data collection and it connects the performance of teachers to the characteristics of students that the teachers teach. It seems much more practical for this type of international study. It promises to serve in this study as an alternative to the common value added model in use elsewhere.

Authors

• Mark D. Reckase, Michigan State University

• Maria Teresa Tatto, Michigan State University

• Michael C. Rodriguez, University of Minnesota