Innovative Assessments of Student Attitudes and Beliefs in PISA (Programme for International Student Assessment) 2012: Validity Evidence for an Alternative Scoring Approach for Likert-Type Items Based on Anchoring Vignettes

• In Event: Comparison of 67 Countries in Achievement and Noncognitive Factors: Highlights of PISA (Programme for International Student Assessment) 2012

Abstract

This presentation focuses on innovations in the assessment of student attitudes and beliefs with the PISA 2012 background questionnaires. Indexes capturing noncognitive student factors, such as mathematics interest, mathematics self-concept, or sense of belonging to school, have been used in PISA since its first cycle in 2000. Many of these indexes are based on self-report items with the vaguely defined response categories “strongly agree” to “strongly disagree”. A robust finding across all previous PISA cycles is that inconsistencies in the directionality of relationships between some background constructs measured with Likert scales (e.g., mathematics interest) and achievement outcomes on the individual versus aggregated level are found. These inconsistencies might be a result of systematic differences between countries with regards to how students interpret the agreement response scale. Based on PISA 2012 student questionnaire data, this paper presents validity evidence for an alternative scoring model for Likert-type items based on anchoring vignettes (King & Wand, 2004). Anchoring vignettes are short descriptions of hypothetical individuals. Ratings of these vignettes in terms of a student’s level of agreement with a statement about the hypothetical individual described provide a basis for an alternative scoring model: numerical values for student responses are not assigned based on the concrete response option chosen (e.g., the value 4 for “strongly agree” and 3 for “agree”) but based on the self-report answer relative to the personal standard captured by the rating of three vignettes. The approach has been used for making cross-country comparisons in various fields of research but PISA 2012 is the first educational large-scale assessment to use the technique. PISA 2012 data shows that there are substantial differences in students’ ratings of the vignettes both within and across countries. Differences in these ratings can be attributed to differences in the interpretation of the rating scale or general differences in preferred response behavior as the underlying levels in the vignettes were held constant across respondents. Results show that any comparisons of absolute values on questionnaire indexes between countries are likely to be biased by these differences in response behavior and that differences in average values on questionnaire indexes between countries cannot be interpreted as different values on the respective construct captured by this index. On the other hand, with the new scoring approach, relationships with students’ plausible values in mathematics, reading, science, and problem solving are highly consistent within and across countries and stronger in line with theoretical assumptions about true construct relationships. Anchoring enhances especially the international comparisons by providing an alternative metric for comparisons that is invariant across countries. Effects on three different level (student, school, country) are distinguished. The paper discusses implications of these findings for the assessment of noncognitive constructs in K-12, higher education and workforce populations in general and provides a rationale why these findings should change how we think about measuring noncognitive constructs, especially in cross-national and multi-cultural populations.

Authors

• Jonas Bertling, Educational Testing Service

• Patrick Charles Kyllonen