A Valid Decision Based on Multiple Tests: Validation of Competence Assessment Programs

• In Event: Roundtable Session 117
  In Roundtable Session: The Whole Is Greater Than the Sum of Its Parts: Research on Aggregated Test Scores

Abstract

In educational practice it is common to use the results of several tests for a single decision. Validity research is however focused around the validity of individual tests. The purpose of this paper is to propose a framework that guides integral validation efforts for a combination of tests.
According to Kane (2006), validity is about the interpretations assigned to test scores rather than the scores themselves. Interpreting a test score concerns explaining the meaning of a score and making the implications of the scores explicit. In education these implications are often high stake decisions about individual students. The process of evaluating the appropriateness of these decisions is called validation. To guide this process Kane (2006) proposed the argument-based approach to validation. Within this approach two arguments are distinguished: an interpretive argument to specify the inferences made within the test and a validity argument to evaluate the interpretive argument. This approach is exemplified for several single tests such as certification testing (Kane, 2004), language testing (Llosa, 2008), and a driving performance assessment (Wools , Eggen, Sanders, 2010). In this paper we will use the argument-based approach to validate a decision based on a combination of tests: a competence assessment program (CAP) for social workers. From this example, we will formulate a more general framework for the validation of multiple tests.
The elements of the CAP that is validated are administered on several occasions during a 4-year track for social workers in higher education. Within the CAP three key competences are distinguished: designing and executing treatment plans, working in a professional environment, and becoming a better professional. All key competences are divided in several components and are tested using different assessment forms, such as performance assessments, interviews and knowledge tests. After four years, the results of all tests are combined into one decision about students.
The paper shows that the main difference in using the argument-based approach to validation for a CAP instead of a single test lies within the content of the interpretive argument. This is because the major claim of a CAP is that the combination of tests compensates for the down sides of one single test. Therefore, it is necessary to provide evidence that justifies this particular claim. And to do so, we need to evidence the individual components as well as the integrated part that they form together. The paper concludes with a discussion of lessons learned from this example for the validation of aggregated tests in general.

Author

• Saskia Wools, Cito