Revisiting Common Assumptions and Practices in Vertical Scaling

Session Type: Coordinated Paper Session

Abstract

Increasingly, vertical scales are being used in practice to better understand student progress in core academic subjects over time. Yet, the construction of vertical scales often relies on practices and assumptions supported by relatively little evidence, especially given the consequences associated with uses of those scales. In this coordinate paper session, we revisit some of those assumptions and practices. The first study (Briggs) examines an assumption central to use of vertical scales—namely that they possess equal-interval properties—and provides options for psychometricians to test that assumption that are both axiomatic and pragmatic. The second (Soland & Edwards) and third (Liao & Bolt) revisit rules of thumb about how many linking items are needed in adjacent grades, including when common forms of measurement model misspecification occur. Meanwhile, the fourth paper (Student) uses moderated nonlinear factor analysis to probe assumptions underlying an incredibly common practice, namely using different grades from cross-sectional data as a proxy for how students develop longitudinally, which assumes differences across grades are not affecting item parameters. In total, this panel not only raises questions about one of the most important tools in psychometrics, but also tries to provide solutions that can be used by measurement practitioners.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• James Soland, University of Virginia

Individual Presentations

• On the Justification and Evaluation of an Interval Scale - Derek Christian Briggs, University of Colorado Boulder

• Revisiting the 20% Rule of Thumb for Linking Items in Vertical Scales - James Soland, University of Virginia; Kelly Edwards, American Board of Pediatrics

• Implications of IRT Model Misspecification on Vertical Scale Linking - Xiangyi Liao; Daniel Bolt, UNIVERSITY OF WISCONSIN, MADIS

• Validating, Calibrating and Linking Vertical Scales with Moderated Item Response Theory - Sanford R Student, University of Delaware

Discussant

• Scott Monroe, UMASS AMHERST