Conceptualizing and Measuring Opportunities to Learn and the Contexts of Teaching

• In Event: 73.023 - Toward a Fair Evaluation of Teachers: Methodological Challenges in a Cross-National Study of Mathematics Teachers

Abstract

Objectives

The design of the novice teacher questionnaire spanned several phases, including an early item tryout, a formal pilot, and final instrument design. A formal process was developed to ensure coherence and consistency in item format and presentation for all items in all the questionnaires. This paper describes our process.

Theoretical framework

This paper is primarily methodological, including the application of modern measurement theories to solve practical problems with large-scale international survey research. This includes latent-trait methods such as confirmatory factor analysis using Mplus (Muthén & Muthén, 2012) and Winsteps to employ the Rasch scaling model (Linacre, 2012). The survey items come from a variety of sources, including the Teacher Education and Development Study (TEDS-M) and the Third International Mathematics and Science Study (TIMSS) (Provasnik, et al., 2012), and focus on the key elements of mathematical knowledge for teaching (Ball, Thames, & Phelps, 2008).

Methods

In all stages of the study, item development analyses were conducted, including exploratory factor analysis, and correlations among and between items of similar and different constructs. The items functioned exceptionally well compared to prior experience with pre-existing items. Some items underwent revision based on item pilot reviews. A standard set of item-writing guidelines were adopted to assure consistency and coherence in all measurement aspects. For the pilot instrument, confirmatory factor analyses were conducted prior to scaling (Mplus), including analyses of factor structure across participating countries. Rasch analyses were conducted to assess item fit and measure quality.

Data sources

Items were solicited from the participating international research teams and many were developed to measure opportunities to learn and the contexts of teaching including opportunities to learn mathematical content and pedagogy; beliefs about teaching, learning and students; curriculum; enacted practices; school climate; pre-service teacher preparation; and teachers’ background characteristics. Some items were modeled after other international educational surveys, such as TIMSS, and TEDS-M. All items were reviewed by all participating countries and the study’s management team.

Results

The process of gathering information on the functioning of the study’s measures according to the context and outcomes of teaching internationally in some cases challenged the theory behind the method (such as whether a scale measures the same construct in every country). For example, using a confirmatory factor analysis approach to assess measurement invariance across countries was not possible because of some nuances in response patterns in some countries. The challenges and successes in building relevant measures are presented.

Scholarly significance

The development of sound measures is essential to the quality of scientific studies of teaching. Understanding the beliefs and opportunities to learn various mathematics, curricular and pedagogical topics in an international context will allow policymakers to have a research base on which to improve teacher induction. By developing valid and reliable measures this study has taken an important step in this direction.

Authors

• Michael C. Rodriguez, University of Minnesota

• Maria Teresa Tatto, Michigan State University