Measuring Strategic Processing With Concurrent and Task-Specific Self-Reports

• In Event: Investigating Strategies and Strategy Use: Where Do We Go From Here?

Abstract

Objective and framework
Our objective is to discuss the use and validity of concurrent and task-specific self-reports as measures of strategic processing. Given the effortful, conscious, and intentional nature of strategic processing (Afflerbach et al., 2008; Alexander et al., 1998), self-report methodologies seem applicable in gauging such thinking. We generally define self-reports as utterances or answers to prompts or questions provided by an individual about his or her cognitions and actions during learning, comprehension, or problem solving. Self-reports of strategic processing can involve on-line or off-line reporting. On-line approaches refer to measurements taken concurrent to task performance, such as having learners think aloud to create a verbal protocol. Off-line approaches refer to self-report inventories or interviews administered before or after task performance. We consider on-line measures in the form of concurrent thinking aloud and off-line measures in the form of task-specific self-reports collected immediately after task performance in this poster presentation.
Mode of inquiry
We review four different approaches to assessing strategic processing by means of self-reports, focusing on validity issues relevant to each approach.
Results
Verbal protocol analysis is a classic self-report methodology where individuals during task performance report the thoughts that are immediately accessible and reportable in language, and therefore represent the contents of working memory (Ericsson & Simon, 1993). Another approach is to record individuals’ eye movements during task performance, and afterwards display those eye movements to them to cue reports of their thinking during task performance (Salmerón et al., 2017). Eye movement cued self-reports can be considered a form of retrospective thinking aloud because individuals are responding to external cues rather than internal cognitive states. Task-specific strategy inventories include items that are tailored to a specific task context and administered shortly after task performance As such, this approach is consistent with Ericsson and Simon’s (1993) recommendation that retrospective verbal protocols be provided immediately after task performance, that is, when strategic activities are still relatively accessible for retrieval in working memory. Finally, diaries are standardized instruments that can be used to self-report or self-record strategic activities on a daily basis near the time and in the contexts they occur (Schmitz et al., 2011).
Efforts to validate these approaches have indicated that verbal protocol analysis (e.g., Magliano et al., 1999), task-specific self-report inventories (e.g., Samuelstuen & Bråten, 2007), and diary methods (e.g., Schmitz & Wiese, 2006) may yield valid data on strategic processing. However, given the few attempts to validate eye movement cued self-reports (e.g., Brand-Gruwel et al., 2017), claims about the validity of this methodology for assessing strategic processing should be considered with caution until further documentation is available.
Conclusions and Implications
A general lack of studies comparing strategy data across different methodologies makes it difficult to say whether what is captured (or missed) with respect to strategic processing is unique to particular methodologies. Especially, the different concurrent and task-specific methodologies we reviewed have seldom been compared with one another. Further research including more than one of these measures is therefore needed to probe their concurrent validity.

Authors

• Ivar Braten, University of Oslo

• Joseph Magliano, Georgia State University

• Ladislao Salmerón, University of Valencia