The AVCs of Response Processing Data: The Latest What, When, and How of Think-Alouds

• In Event: 32.054 - Thinking Through Think-Alouds: Contemporary Contributions to Validation

Abstract

The latest reauthorization of the Elementary and Secondary Education Act (i.e., Every Student Succeeds Act [ESSA]) demonstrates the latest policy attempt to optimize the learning of all students. Testing specialists contribute to this endeavor by scrutinizing and enhancing student assessment; for example, with the collection of response processing data to ensure accuracy and fairness in assessment outcomes. However, although the collection of these data has increased in test development and validation efforts over the last 20 years (Leighton, 2013, in press; Shear & Zumbo, 2014), the techniques used to collect these data continue to evolve. However, few publications focus on developments for best practices (however, see Leighton, 2004, 2013). Thus, the objective of the paper is twofold – (a) to summarize the latest developments in the what, when and how of cognitive laboratories (labs) and think aloud methods, and (b) to highlight the psychometric reasons to collect response processing data in the first place, namely the affordances in adaptability, validity and credibility these data provide to educational testing and assessment efforts.
The first objective is to summarize the latest developments in the collection of response processing data using cognitive lab and think aloud methods. This section will integrate recent empirical research in educational measurement (e.g., Leighton, 2013), second language research (e.g., Bowles, 2010) and psychological research (e.g., Hsu, Babeya, Feng, Hummer, & Davison, 2014; Willis, 2015), including results from a large meta-analysis of the effects of these methods for yielding expected data (Fox, Ericsson, & Best, 2011). This section includes updated evidence-based arguments for the what, when and how of using cognitive labs and think alouds: (a) Distinguishing what kind of cognition or response processing each method – cognitive labs or think alouds – is best suited to measure, (b) Knowing when to use cognitive labs versus think alouds in collecting response processing data for a specific psychometric purpose, and (c) Understanding best practices for how to use these methods, including interview probes, sample representativeness and size depending on the objectives of data collection.
The second objective is to highlight the psychometric rationale for collecting response processing data as indicated in the Standards for Educational and Psychological Testing (2014). This section will outline the benefits, namely, the adaptability, validity and credibility or AVCs of response processing data as it pertains to developing/designing test items, generating validity arguments and fostering credibility of large-scale assessments for measuring learning and achievement. This section will include illustrations for how response processing data, collected using appropriate procedures, can be used formatively and summatively. For example, response processing data can be used formatively to inform test item adaptation (and development) for cross-cultural and special population assessment applications. Response processing data can also be used summatively to inform validity arguments, especially to provide evidence of the underlying cognitive strategies associated with correct and incorrect responses. Finally, response processing data, properly gathered and used, can be expected to lead to increases in the credibility of student assessments for measuring meaningful, higher-level thinking constructs and helping students succeed in the 21st century.

Author

• Jacqueline P. Leighton, University of Alberta