Working at Scale to Recover Learning Trajectories Empirically for Grades 6–8 Math Through Ongoing Validation

• In Event: 47.063 - Validating Theories of Learning for Classroom Assessment Design: Sources of Evidence

Abstract

The Common Core State Standards for Mathematics (CCSS-M) rest on learning sciences (LS) research as expressed by learning progressions/trajectories (LPs/LTs) (Confrey, Maloney & Nguyen, 2014; CCSSO, 2010). However, standards alone cannot communicate the depth of knowledge about student learning needed to guide instruction. To address this need, Confrey (2015) built a learning map (sudds.co) for middle grades mathematics, hierarchically organized around nine big ideas, 24 relational learning clusters (RLCs), and 63 research-based LTs associated with CCSS-M. To bring the LTs to practitioners at scale, a diagnostic assessment system was designed to measure students’ progress along LTs, providing real-time data to students and teachers through user-friendly displays. Teachers embed these as classroom assessments within ongoing instruction based on their district’s curricular sequence (Wilson, 2018).
Each assessment, at the level of the RLC, comprises of 8-12 items of varying types. Items are aligned to LT-levels and designed to support rich classroom discourse. Data are returned in displays that support teachers and students in reviewing individual items, analyzing student response patterns and seeing common misconceptions. Teachers can view patterns of responses across LT-levels to decide where to intervene, and with whom. Research on the learning map includes quantitative results for specific LTs (Confrey et al., 2017; 2018) and a qualitative investigation into how teachers review the data with students and other teachers (Confrey et al., in press).
The team has developed a validation framework (Confrey, Toutkoushian, & Shah, in press) derived from Pellegrino, DiBello, & Goldman (2016) and a method to conduct ongoing validation studies of specific RLCs (Confrey & Toutkoushian, in press). A key characteristic of this work is the collaboration between experts in LS and psychometrics working in a continuous cycle and at scale with partner districts. The partners consist of three schools serving diverse populations in two states with more than 48,000 tests administered across three grades. In this paper, we analyze and summarize data patterns drawing from all 63 LTs by examining their empirical recovery from multiple vantage points. This includes obtaining quantitative evidence to determine the relationship between LT-level and item difficulty, as well as qualitative evaluations from a LS perspective about the alignment of items to the cognitive meaning of levels, and the relationships among the levels.
Comparing the results of these analyses allow for the identification of LTs that are performing as expected, theoretically and empirically. For those cases in which the data do not match the expected pattern of difficulties suggested by the LT, a deeper dive into the variation causing non-conformance, through methods such as think-aloud protocols and LS review, suggests possible paths for improvements or modifications to items, LTs, or RLCs. The results of this study illustrate the importance of undertaking this work at scale in order to improve LTs over time through systematized validation studies that are a) sensitive to variations in data from classrooms due to issues such as opportunity to learn, and b) principled enough to allow for generalizability.  

Authors

• Jere Confrey, North Carolina State University

• Emily Toutkoushian, University of North Carolina - Chapel Hill

• Meetal Shah, North Carolina State University