Online Response Process Procedures: Affordances and Constraints to Collect, Analyze, and Interpret Validity Information

• In Event: Marriott Roundtable Session Two
  In Roundtable Session: Quality, Rigor, and Validity in Mixed-Methods Research (Table 3)

Abstract

The paper focuses on collecting and analyzing information about online “response process” procedures (ORPEs) e.g., data collected through Amazon MTurk. This line of validity evidence has been usually conducted with talk-aloud protocols or cognitive interviews (CIs). However, information collected from ORPEs differs from CIs in key characteristics. Further, validity evidence for context-based items needs to be examined both for the construct tapped by the item and the context characteristics relevant for a response. The paper offers an approach for using meta-questions to elicit information about the item context and the construct using OPRE. The data (341 responses collected for 12 questions) is analyzed by comparing experts’ and responders’ answer profiles to support valid inferences.

Authors

• Radhika Kapoor, Stanford University
  Presenting Author

• Maria Araceli Ruiz-Primo, Stanford University
  Non-Presenting Author

• Philip Hernandez, Stanford University
  Non-Presenting Author

• Klint Kanopka, New York University
  Presenting Author