Data Quality From the Ground Up

• In Event: Program Evaluation in a Large Urban District: A Nuts-and-Bolts Perspective

Abstract

Objectives or Purposes: This presentation highlights the specific processes and procedures implemented assure data quality for the evaluation of Zoology One. The primary aim is to outline data collection and management systems our team used, and to addressed the problems we encountered, and solutions we developed, in the course of the first year of the study.

Theoretical Framework: There is a great deal of research regarding the role of data in program evaluation. However, generally speaking, the evaluation literature focuses on research methods; little attention is paid to the day-to-day problem-solving required in order to ensure high-quality results as these methods are deployed. Furthermore, these theoretical frameworks are discussed under the pretenses of an idealized experimental environment, often not accounting for the many areas of variability that exist in a study conducted in actual schools. There is therefore a need for new literature focused specifically on the development of systems for ensuring data quality from the earliest stages of planning and organizing data collection, through the process of addressing potential barriers, and ensuring strong data management practices—given the realities of other project activites--post-collection.

Methods, Techniques or Modes of Inquiry: This paper will discuss our team’s efforts to maintain constant communication with school and district staff regarding data collection visits; tracking of materials to and from school sites; creative processes for on-site deidentification; recruiting, hiring, and training teams of individuals to administer assessments; creating data inputting platforms that are specific to each assessment and allow for simultaneous data cleaning; timeline management in order to meet data collection deadlines; and navigating continual challenges in each of these and other areas. This presentation will identify how each of these items can be systemized, from planning steps to logistics of implementation, to create and preserve a smoothly operating data collection process.

Results and/or Substantiated Conclusions or Warrants for Arguments/Point of View: The initial year of this study successfully collected data from 12 schools and over 1100 students. Students were deidentified and tracked throughout the year, including noting each point of contact, or missed point of contact, the assessment team had with each student. Additionally, in tracking each student individually, the assessment team was able to prioritize the assessment of students in the stable sample. Nearly all students in the stable sample were assessed both in the beginning, and at the end, of the year. All data collection was completed within the timeline parameters. Statistical analyses periodically performed throughout the data collection period showed that there was no statistically significant variation in the data collected based on assessor, ensuring high data quality.

Scientific or Scholarly Significance of the work: The strategies outlined in this presentation illustrate the areas that benefit from both a systematic approach and a flexible, problem-solving stance. Moving beyond theoretical methodology, ‘tried and true’ examples of precise planning, coordination and implementation of data collection processes are shared for others to replicate, adapt, or draw insight from.

Authors

• Anushka Patel, University of Pennsylvania

• Tesla DuBois, University of Pennsylvania

• Philip M. Sirinides, University of Pennsylvania