Using Public Opinion to Inform the Validation of Test Scores

• In Event: Exploring Unintended or Inappropriate Interpretations, Uses, and Consequences of Test Scores

Abstract

Companies routinely ask for customer feedback — whether they produce laptops, movies, or provide banking services — public opinion matters. Surveys, polls and focus groups can help reveal perceptions about the product, identify areas for improvement and reduce future problems. However, the application of these methodologies for test improvement is limited, especially in education (Stricker, Wilder & Bridgeman, 2006). This is unfortunate given that standardized tests are the subject of a heated public debate (Phelps, 2003). Public opinion can affect the test's content and structure, its administration and scoring, and the interpretation and usage of test scores. Regardless of whether a test is psychometrically valid or not, public pressure can influence policy-makers to consider the test invalid for its intended purpose.
This paper discusses why and how test developers should consider what people think about the test. We argue that it is worth exploring the disparities between the intended purposes of the test and what people think the test actually measures - namely, the "face validity" of the test. Although face validity is considered a subjective judgment which is not indicative of the test scores' predictive or construct validities (Anastasi, 1988), we believe that it might offer a unique contribution to the development of a sound interpretive argument. The disparity between the intentions of the test developers and people's perceptions of the test can lead to various misinterpretations and misuses of test scores. Test developers must not only explain how test scores should be interpreted and used (Kane, 2006), but also why certain alternative interpretations and uses are inappropriate and may lead to undesirable consequences.
To support our claims, we present findings from a study about perceptions of the Psychometric Entrance Test (PET) to Israeli institutes of higher education. Following the methodology outlined by Nevo (1985), we conducted surveys, interviews and focus groups with future, present and past PET examinees, faculty and admissions officers from institutes of higher education, politicians and other public figures who deal with educational issues. We compared their views and perspectives with those of the PET developers and other measurement experts. Apart from background information, the data we collected included multiple-choice and open-ended questions about: (a) the relevance of the PET to its intended use, (b) the extent to which the PET measures abilities important for success in higher education, (c) the extent to which people agree with various claims for and against the PET, (d) ways that the PET scores are used that differ from their intended use, and (e) preferences among tools and methods for selecting students to higher education. The paper discusses the study findings, their implications, and how they were used to support proposed changes in the structure, content, administration and score interpretation of the PET.

Author

• Tzur M. Karelitz, National Institute for Testing & Evaluation