A Design Study of a Wireless Interactive Diagnostic System Based on a Mathematics Learning Trajectory

• In Event: Roundtable Session 62
  In Roundtable Session: New Approaches to Validity Judgments for Emergent Research Methods

Abstract

Perspective. Design studies are powerful research tools for introducing newly emerging ideas, tools, and technologies into schooling, and studying their effects on student and teacher interactions and on learning. Researchers undertake “engineering particular forms of learning, and systematically studying those forms of learning within the context defined by the means of supporting them” (Cobb et al., 2003). Design studies incorporate features that address conditions for which other forms of experimental methods are insufficient: discontinuities, domain-specific modest theories, and studies demanding iterative design. An emerging innovation, well suited for design studies is the implementation of an interactive diagnostic assessment systems (IDAS) built around explicit learning trajectories. Such systems are designed to be disruptive innovations that provide teachers and students immediate feedback on student conceptual progress in real time. Unlike other feedback systems (e.g. clickers, displays of student work), IDAS is grounded on empirical evidence of diagnostic patterns of how student learn. Using browsers on networked devices (smart phones; tablet or laptop computers), the IDAS support socially networked interactions 24/7. One such system, LPPSync (Learning Progress Profiles synchronized for networked wireless devices) is being studied with small groups of children, grades 3-5, and includes three modes of activity: administering/scoring items (including data gathered directly from the students’ actions on dynamic manipulatives), sequestered or interactive practice, and targeted interventions.

Method. The underlying cognitive model is a learning trajectory for equipartitioning, built on extensive literature syntheses and cross-grade clinical interviews. Paper and pencil assessments assessing proficiency levels across multiple task classes, were field-tested, and analyzed to validate the item difficulty in the trajectory using IRT models. A prototype of the diagnostic system was used to conduct think-aloud interviews. These studies provided partial validation of the assessments, but did not explore the use and consequences of the IDAS within ongoing instructional practice. For the current study, the team conducted a two-week design study with fifteen third graders from inner city homes working with the curricular materials and iPads comprising an IDAS for 1.5 hours per day.

Results. This paper reports on the evolution of the design team’s understanding of learning trajectories and of related IDAS systems through application of multiple methods (synthesis, clinical interview, statistical analysis of assessment field-test, design study), and how that evolution can be interpreted in the form of a “validation argument” in the sense of Kane (2006). Earlier descriptions of the role of design studies (Cobb et al, 2003; Confrey 2006) will be reinterpreted through the lenses of a) prior work via multiple methodologies, b) separate validation the underlying cognitive model and the resulting relative stability of expectations of student responses, and c) technology that can conduct diagnostic analysis on data as they are gathered during the study itself.

Significance. We will trace the evolution of those ideas in the context of building innovative technologies to strengthen teaching and learning at a time where the field needs to be “engineering [for] effectiveness” (Confrey & Maloney, 2011) in new and more agile ways.

Authors

• Jere Confrey, North Carolina State University

• Drew Corley, North Carolina State University

• Alan Maloney, North Carolina State University

• Kenny Huy Nguyen, North Carolina State University