The Assessment and Reporting of the Mathematics Knowledge of Prospective Teachers from International Studies

• In Event: Measuring the Impact of Teacher Education on Learning to Teach Mathematics: The Teacher Education Study in Mathematics

Abstract

1. Objectives

This paper examines the challenges involved in developing knowledge assessments in TEDS-M, the psychometric properties of these assessments, and the procedures used to facilitate interpretation of results across countries.

2. Perspective(s) or theoretical framework

Developing assessments of knowledge for international and comparative studies is a challenging task (see Hambleton, Merenda and Spielberger, 2005; National Research Council, 2002), these include (a) the differences across countries regarding when and with what emphasis content topics are covered: even a subject matter area as well defined as mathematics shows substantial variation in timing, coverage, and emphasis of mathematical content, thus deciding what should be included on an assessment is not straightforward; (b) the breadth of material that needs to be covered in the period of time that can be obtained from the participants in a country: to solve the problem of breadth, advanced testing designs are sometimes used to give achievement estimates for a full domain. These designs require sophisticated analysis procedures to combine the results from many separate test booklets into results that apply to the full sample of participants within a country.

3. Methods, techniques, or modes of inquiry

The TEDS-M assessments for the Mathematics and Mathematics Pedagogical Content Knowledge used a balanced-incomplete-block design so that the desired content could be well covered, within a reasonable administration time. This meant that each person only took part of the full set of items. To obtain comparable estimates of performance, item response theory (IRT) was used to calibrate the items (see DeAyala, 2009 for a description of IRT methodology; and Masters, 1982, and Wu, Adams, Wilson, & Haldane, 2007 for a description of calibration procedures). The calibration results were also used to identify anchor points for the score scale to facilitate interpretation of the test results.


4. Data sources, evidence, objects, or materials

The data comes from the TEDS-M tests of knowledge administered to 15,163 future primary teachers and to 9,389 future secondary teachers in the 16 countries participating in the study.

5. Results

When assessment instruments are designed to apply to all countries in a study, the meaning of the reporting score scales must be made accessible. In the TEDS-M project, this was done with scale anchoring – giving descriptive meaning to points on the score scale. This required up-front planning for a good assessment design, a good sampling plan, and meaningful reporting.

6. Scientific or scholarly significance of the study or work

TEDS-M developed a unique strategy to interpret results of future teacher knowledge assessments. The process required the use of psychometric theory, and analysis by a panel of experts to develop descriptions of the capabilities of persons who along the assessment scale. These descriptions were produced by the collaborative work of committees of mathematicians and mathematics educators in the TEDS-M assessment development workshops. The result was “anchor point” descriptions which give tangible meaning to points on the reporting score scales. This is a unique contribution of TEDS-M to the area of measurement in teacher education.

Author

• Mark D. Reckase, Michigan State University