Using Assessments in an Evaluation: A Mission-Critical Balancing Act

• In Event: Program Evaluation in a Large Urban District: A Nuts-and-Bolts Perspective

Abstract

Objectives or purposes: The Zoology One study includes the collection of large amounts of diverse student outcome data from 2,400 kindergarten students. We are administering assessments in reading, writing, science, math, and reading motivation, using a combination of standardized assessments, adapted existing measures, and newly created instruments. This paper will:
• Explain the team’s decision-making process for whether and when to use existing assessments; modify assessments; or develop assessments. This includes the factors our team considered for assessing young children in busy schools;
• Discuss the realities of using standardized assessments that are often not as “standardized” as they seem;
• Share our experience navigating the challenges of assessment development within a fast-moving study and given the constraints of the district and schools.

Perspective(s) or theoretical framework: Published reports from program evaluations generally address the choice of assessment instruments only in passing. Assessments are identified, and there may be a brief discussion of the rationale for particular decisions. In contrast, this paper addresses the nuanced and complex issues our team has encountered with making these decisions. And, once assessments are selected, important decisions must be made and critical challenges addressed with regard to administration and other issues. Adapting or creating instruments in the context of program evaluation introduces more challenges still. All of these topics will be of interest to program evaluators, and will be addressed in this paper.

Methods, techniques, or modes of inquiry: Student assessments were selected using the perspective of the student and teacher. The research team intentionally selected assessments that provided reliable and valid responses from kindergarten students, but also were a low-burden on student time and energy. Our processes for weighing these and other factors will be details in the paper.

Data sources, evidence, objects, or materials: This paper will detail our timeline for assessment in the first year of the project and will document our decisions about and use of the following assessments:
• Woodcock Reading Mastery Test
• Math and Writing subtests of the Kaufman Test of Educational Achievement, Third Edition (KTEA-3)
• Science Learning Assessment (Samarapungavan, Patrick, & Matzicoupolis, 2006)
• Sentence Writing Assessment (Coker & Ritchie, 2012)
• Kindergarten Reading Motivation Scale (Castillo & Gray, 2017)

Results and/or substantiated conclusions or warrants for arguments/point of view: In the context of our first year of evaluation, our small team administered a range of standardized assessments, worked with partners to adapt other assessments, and developed a new measure of motivation to read. Each of these assessments is critical to our project, and we carefully weighed a range of factors in the selection and actual use of each.

Scientific or scholarly significance of the study or work: This paper seeks to serve as a reference or to others who intend to use or develop academic assessments for young students as part of an evaluation. Our experience in this area will provide a much-needed perspective on the role and use of assessments in the context of large evaluations.

Authors

• Tesla DuBois, University of Pennsylvania

• Anushka Patel, University of Pennsylvania

• Wendy Castillo, Overdeck Family Foundation