Is That a Mode Difference? Demystifying Mode Effect Detection and Adjustment Methods

Session Type: Coordinated Paper Session

Abstract

Mode differences are often examined through mode studies using experimental or quasi-experimental designs and, if needed, adjusted for using equating methodology. The purpose of this session is to invite researchers and practitioners into a thorough re-evaluation and discussion of commonly used mode effect detection methods and linking designs.
The first two studies presented in this session evaluate the effectiveness of various statistical methods to obtain equivalent groups without random assignment, one for multiple-choice tests and one for a writing test. The investigations are based on manipulations of operational data from paper testing so that the true mode differences are known. The third study highlights advantages and pitfalls of all three linking designs through demonstrations using simulated data. The study is motived by common misconceptions and misunderstandings and seeks to provide enhanced perspectives to researchers and practitioners. The fourth study deals with practical situations in which true mode effects are unknown. In this study, results obtained through chained linking with a random groups design and those from a direct linking through propensity score matching are compared.
Insights and information gained from this session can inform practical decisions about mode effect detection methods and mode linking designs.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• Dongmei Li, ACT

Chair

• Jeffrey Steedle, Curriculum Associates

Discussant

• Deborah Harris, University of Iowa

Individual Presentations

• Finding Optimal Solutions for Mode Effect Detections Without Random Assignment - Scott Woods, ACT, Inc.; Shalini Kapoor, ACT Education Corp; Jeffrey Steedle, Curriculum Associates

• Finding Optimal Solutions for Detecting and Adjusting Writing Mode Differences - YoungWoo Cho, ACT; Dongmei Li, ACT; Ann Arthur, ACT Education Corp

• Showcasing Advantages and Pitfalls in Mode Detection and Adjustment Methods - Dongmei Li, ACT; Ann Arthur, ACT Education Corp; Hongling Wang, ACT, INC.

• Is That a Real Mode Difference? - Shalini Kapoor, ACT Education Corp; Chen Qiu, ACT, INC.; Dongmei Li, ACT