Developing Measures of Teacher and Student Understanding in Relation to Learning Trajectories

• In Event: 69.052 - Using Research-Based Formative Assessment to Improve Access to Mathematics in Urban Schools

Abstract

Purpose

This paper describes the design, development, and analysis of two innovative instruments: (1) a measure of teacher knowledge of student thinking in the activity of looking at and responding to student work, and (2) a measure of student performance in terms of both accuracy and sophistication of strategy for multiplicative thinking.

Perspectives

Effective implementation of the Common Core State Standards for Mathematics (CCSSM) requires teachers to have deep knowledge of their students’ conceptual understanding and an explicit awareness of the developmental trajectories that are embedded within the standards. Research on multiplicative thinking shows that students use a variety of strategies to solve problems that can be located on a developmental progression (Confrey et al, 2009; Sherin & Fuson, 2005). These strategies get progressively more sophisticated over time as students interact with different problem structures and develop stronger multiplicative reasoning (Kouba & Franklin, 1995).

Methods

To measure teacher learning in relation to learning trajectories, we used the Teacher Assessment of Student Knowledge (TASK). TASK is an on-line instrument that presents a teacher with a carefully designed set of student responses to a mathematics problem that characterize different levels of sophistication of student thinking as well as common misconceptions. Teachers are led through a series of questions that measure three key domains of learning trajectory-oriented formative assessment: (1) Analysis of Student Thinking; (2) Learning Trajectory Orientation; and (3) Instructional Decision Making.
To assess the impact on student learning, we developed a measure with three vertically equated, grade-specific forms, composed of open-ended contextual multiplication and division problems. Students are asked to show their work to allow for analysis of their strategies and errors. The assessment is closely aligned with the expectations of the CCSSM in terms of the complexity of numbers and the range of multiplicative contexts. To measure strategy sophistication, we constructed a six-level rubric. Raters were trained to use this rubric to reach reliability of over 85% agreement with an expert. After an extensive field test, Item Response Theory (IRT) methods were used to select 12 items that elicited a range of strategies appropriate to grade level.

Results and Significance

We examine score distributions from two years of the study to further investigate relationships between various dimensions of (1) teachers’ ability to analyze student work in mathematics and their instructional decision making and (2) student ability to produce correct answers and use sophisticated strategies in relation to problem difficulty and context. Using path analysis (Wright, 1934), we found that teacher’s understanding of student thinking has the largest total effect on instructional decision making but most of the relationship is mediated by learning trajectory orientation. Our analysis of student performance indicates that accuracy and sophistication are related but distinct dimensions. In addition, less sophisticated strategies are more likely to produce correct answers on easier questions, and more sophisticated strategies are more likely to produce correct answers on harder questions. We propose alternative multi-dimensional models to explore interactions between item and person parameters across dimensions.

Authors

• Caroline Brayer Ebby, University of Pennsylvania

• Philip M. Sirinides, University of Pennsylvania