The Think-Aloud Method in the Process-Related Validation of an Assessment of Academic Student Learning Outcomes

• In Event: Valid Assessment of Learning Outcomes in Higher Education: International Perspectives and Methodological Challenges

Abstract

For many years, validation practices have been criticized for not generating comprehensive explanatory results that would allow test users to draw causal inferences from test scores. Hence, recent validity concepts highlight the need to draw valid inferences from the broadest possible basis of logical and empirical evidence (Kane, 2013; Zumbo, 2009). This means test developers should not only rely on quantitative validation practices based on the final test or item scores, but also analyze the processes that generate these scores using mixed-method approaches (AERA, APA & NCME, 2004; Borsboom, Mellenbergh & van Heerden, 2004; Leighton & Gierl, 2007; Zumbo, 2009). In higher education research, attention is increasingly being given to process-related validation and the think-aloud method for analyzing response processes in assessments. The use of think-aloud in educational settings has been controversial because it can lead to missing or substantially biased results if test takers lack language proficiency, for example, if they are non-native speakers (Wilson, 2005). Unlike school-age children, most university students have sufficient language skills to make think-aloud an attractive method for analyzing the content of mental processes (Gorin, 2007). Analyses are often conducted ad hoc, based on qualitative data only and lacking connection to other validity evidence, for example, from quantitative field studies or online expert interviews. This can lead to erroneous evaluations of test score interpretations and jeopardize validity. In this paper, I explore the potential and limitations of the think-aloud method for valid assessment of student competencies in higher education. I evaluate the method on theoretical and methodological levels based on validity evidence generated through other methods (e.g., see the Standards of Educational and Psychological Testing, AERA, AEA, & NCME, 2014) and place it the core of a socio-cognitive approach (Mislevy & Duran, 2014). In this approach, the student is seen as being part of and interacting with a comprehensive environment in different situations, in which the items represent different situations. Thinking aloud is also a mentally socio-cognitive act, in which the item responses can be regarded as results of student–item interaction and therefore can be quantified and clustered into students and items. This evaluative and exploratory study provides a methodological foundation for the think-aloud method for higher education using a multi-level model (Zumbo, 2009).
To introduce this approach, we will first show data and analysis results from think-aloud interviews on the levels of single responses (level 1) and of students and items (level 2). Second, I will present and critically discuss an integrative approach to combining these results with findings one-parametric item response models and with feedback from expert interviews. Doing this, the mental processes can be assigned as covariates to the responses on level 1 and the other types of validity evidence are included as covariates on level 2, because they describe the situations by assigning meaning to features of test takers or items. This innovative approach will be illustrated with an empirical example from an assessment of university students’ economic competence.

Authors

• Sebastian Brueckner, Johannes Gutenberg University of Mainz

• James W. Pellegrino, University of Illinois at Chicago