State Supported Local Assessment Systems: Serving Multiple Coherent Purposes

• In Event: Roundtable Session 4
  In Roundtable Session: Validating Local Assessments

Abstract

An often repeated set of axioms in educational measurement is that validity evaluations must be tied to the purposes and uses of the assessment, and that assessments cannot serve multiple purposes well. However, well-implemented local assessment systems may offer the possibility of serving several purposes well if the assessment and related accountability systems are coherently designed. Of course, the validity of each use, as well as of the program as a whole, must be substantiated. This paper draws from experiences in two states where local school districts were required, often working in partnership with the state, to design and implement local assessment systems to certify students’ graduation proficiency. A primary purpose is the obvious student-level certification and the system must be validated for this important use. The local assessment system also can serve important consequential purposes such as improving assessment literacy among local district personnel, and because of the intended close tie with the actual curriculum, the assessment system can better contribute to improvements in instruction, curriculum, and learning than a “drop from the sky” state graduation exam. Importantly, these “secondary” purposes can be fulfilled quite well if attended to explicitly in the design of the policies and practices associated with the system.

This paper presents a detailed analysis of how the designs of the various local assessment systems influence the likelihood of the system serving both its primary and secondary purposes. The paper attends to key tradeoffs (e.g., reliability-validity) that must be considered when state and district leaders create the policies and procedures that shape the local assessment system. For example, states have varied in their approaches for reviewing the technical quality of local assessment systems. Some have used a “peer review” system whereby educators and leaders review—after significant training—the technical quality of systems other than their own. Other states have employed an “expert validation committee” approach whereby all local assessment systems are review by a common set of experts. The peer review approach gives up some technical authority (but not necessarily quality) while enhancing assessment literacy development. The expert panel approach has more technical authority and if designed correctly may support assessment literacy levels. The choice of approach must be tied to the prioritization of the various purposes in the coherent design.

Both of the approaches described above yield technical analyses and results that can be used as part of the evaluation of the validity of the local assessments. The paper describes the opportunities and challenges of the implications of the various design and implementation decisions on evaluating the validity of the local assessment systems. Further, this paper argues that states employing such local assessments need to extend their validity evaluations beyond simply evaluating the technical quality of the local assessments (not that it is simple) and evaluate the impact of the policies and programs on the additional purposes that they system is intended to serve.

Author

• Scott F. Marion, National Center for the Improvement of Educational Assessment, Inc.