Challenges and Successes of Building Measures Relevant to Research on Teacher Education in International Contexts

• In Event: Measuring the Impact of Teacher Education on Learning to Teach Mathematics: The Teacher Education Study in Mathematics

Abstract

1. Objectives

The design of the TEDS-M instruments took several phases, including an early item tryout, a formal pilot, and final instrument design. A formal process was developed to ensure coherence and consistency in item format and presentation for all items in all the questionnaires and on the mathematics knowledge and pedagogical knowledge test items. This paper describes this process.

2. Perspective(s) or theoretical framework

This paper is primarily methodological, including the application of modern measurement theories to solve practical problems with large-scale international survey research. This includes latent-trait methods such as confirmatory factor analysis using Mplus (Muthén & Muthén, 2009) and Winsteps to employ the Rasch scaling model (Linacre, 2011).

3. Methods, techniques, or modes of inquiry

In all stages of the TEDS-M item development item analyses were conducted, including exploratory factor analysis, and correlations among and between items of similar and different constructs. The items functioned exceptionally well compared to prior experience with pre-existing items. Many items underwent revision based on item pilot reviews. A standard set of item-writing guidelines were adopted to assure consistency and coherence in all measurement aspects. For the final instrument, confirmatory factor analyses were conducted prior to scaling (Mplus), including analyses of factor structure across participating countries. In a small number of cases, this led to the elimination of a handful of items from the scaling procedure for some indicators of program opportunities to learn and beliefs, particularly regarding those items that functioned significantly different in some countries. Once the final scales were defined, the Rasch model was used to create score scales for study participants on the measures of opportunity to learn, beliefs, and knowledge. The Rasch analyses also provided additional scale quality information.


4. Data sources, evidence, objects, or materials

Items were solicited from the participating international research teams and many were developed to reflect the principles of mathematics knowledge for teaching. Some items were modeled after other international educational surveys, such as TIMSS, and from an earlier small-scale version of TEDS-M. Questions for opportunities to learn and beliefs were also obtained from related research instruments developed in the USA and Australia, and other countries. All items were reviewed by all participating countries and the TEDS-M management team.

5. Results and/or substantiated conclusions or warrants for arguments/point of view

The process of gathering information on the functioning of the TEDS-M measures according to the context and outcomes of teacher education internationally in some cases challenged the theory behind the method (such as whether the scale measure the same thing in every country). For example, using a confirmatory factor analysis approach to assess measurement invariance across countries was not possible because of some nuances in response patterns in some countries. The challenges and successes in building relevant measures are presented.

6. Scientific or scholarly significance of the study or work

The development of sound measures is essential to the quality of scientific studies of teacher education. By developing valid and reliable measures TEDS-M has taken an important step in this direction.

Author

• Michael C. Rodriguez, University of Minnesota