Opportunity to Learn: Measuring Concept and Item-Type Familiarity With Signal Detection Theory

• In Event: Comparison of 67 Countries in Achievement and Noncognitive Factors: Highlights of PISA (Programme for International Student Assessment) 2012

Abstract

There was interest in measuring opportunity to learn (OTL) in PISA 2012 as a possible explanatory variable for individual and country level achievement. Textbook and curriculum surveys were deemed impractical, and so several methods were attempted to measure OTL involving presenting students with example mathematics items, or topic concepts, and asking them about their prior exposure and familiarity with these items or concepts. The overclaiming technique (OCT; Paulhus, Harms, Bruce & Lysy, 2003; see also Zimmerman, Broder, Shaughnessy, & Underwood, 1977) is a method that can be used to estimate respondents’ concept familiarity and to adjust that estimate based on one’s tendency to overstate what one knows. It does this by presenting students with example mathematics items, or topic concepts, and asking them about their prior exposure and familiarity with these items or concepts. For example, in the PISA survey booklets, students were presented with a list of mathematics concepts, such as logarithm, rational number, and divisor, and asked how familiar they were with those concepts on a six-point scale (from “never heard of it” to “quite familiar with it”). Mixed in with the concepts were several foils (e.g., subjunctive scaling). Two scores were computed from students’ responses. One was a simple mean familiarity scores on the 6-point scale for concepts, and the other was an adjusted concept familiar score that subtracted out the mean familiarity score of the foils. (In the PISA 2012 Field Trial, on average students indicated a 3.6 familiarity rating for the mathematics concepts and a 2.3 familiarity rating for the foils.) On the PISA 2012 Field Trial the average within-country correlations between the concept familiarity ratings and achievement were r = .45 and r = .44 for the unadjusted and adjusted ratings, respectively, that is, they were essentially the same. However, the differences were striking at the between-country level. The country-level correlation between the unadjusted topic familiarity score and achievement was r = .17, and with adjustment that correlation increased to r = .58. Also, the foil score from this topic familiarity test could be used to adjust different OTL measures based on familiarity with various mathematics items, and with PISA items. This can be interpreted to indicate that a general response style factor may have contributed to response patterns on related measures. In this study I will report findings from the Main Study and show findings from a signal detection theory analysis (where hits = reported familiarity with concepts, misses = reported non-familiarity with concepts; false alarms = reported familiarity with foils; correction rejection = reported non-familiarity with foils), based on the 6-point rating scale for concepts and foils (and relaxing the concept-foil equal-variance assumption, Mickes, Wixted, & Wais, 2007). An interpretation of the findings is that concept familiarity is correlated with achievement at the within-country level, and at the between-country level, but that country-level response-style differences have to be accounted for to be able to compare countries on concept familiarity.

Authors

• Patrick Charles Kyllonen

• Jonas Bertling, Educational Testing Service