Data Properties Matter: How Data Characteristics Shape Teachers’ Responses

• In Event: From Data and Evidence to Decisions for School and District Improvement: Synthesizing Findings and Extending Theory

Abstract

Today’s educators are inundated with different forms of data, with the expectation that they will use them to make decisions for their practice (Author, 2012; Mandinach, 2012). However, few studies have carefully examined the ways in which properties of the data affect users’ responses to data (Author, 2012). This paper helps fill this gap by shedding light on the ways in which different data characteristics shape teachers’ perceptions of data and their instructional responses to them.

The paper draws on a one-year, comparative case study (Merriam, 1998; Ragin & Becker, 1992) of six low-income, high needs middle schools in four districts. Data sources include interviews with district leaders (n=13) and school administrators, coaches, and case study teachers (n=83); focus groups (n=6) with non-case study teachers; observations (n=16); survey responses (response rate >90% for case study teachers, coaches, and PLC leads). All transcripts and documents were coded using NVivo. The conceptual framework for this paper is developed from prior research on data-driven decision-making (Author, 2006; Mandinach & Jackson, 2012). In this model, teachers as “data users” start by collecting or accessing raw data that is organized, filtered, and analyzed to become information. Information is synthesized and combined with teacher understanding/expertise to become actionable knowledge. This knowledge can then be used to support different types of responses, which yield a set of instructional outcomes.

Survey and interview results indicate that several key data characteristics were associated with teacher perceptions of usefulness. Teachers reported that the “soft,” qualitative data (e.g., student essays) provided them with more in-depth understanding of a student’s knowledge, skills, and reasoning, compared to the “hard,” numeric data collected through the district’s interim assessments. Another key dimension was the teacher’s role in creating the assessment and in generating the data: teachers were more apt to trust and pay attention to results from internally developed assessments then those from assessments developed by “external” stakeholders such as the district or commercial vendors. Other properties that shaped teachers’ opinions of data included the specific properties of the assessment results (e.g., the degree of alignment between the data and the standards) and timeliness of the results.

Analyzing teachers’ self-reported response to data, we found that superficial instructional responses (e.g., reteaching material without changing pedagogy) were more often associated with quantitative, externally generated data. Deeper level changes in instructional delivery often derived from an analysis of independently created assessments, student work, and observation feedback. Finally, school and district discourse, culture, and policy around the types of data also shaped how teachers responded to them. For instance, in one school system, administrators were required to provide teachers with time on a weekly or daily basis to develop their grade-level common assessments and analyze the resulting data.

These findings suggest that one cannot fully understand the dynamics of data use in schools or districts without a consideration of the characteristics of the data itself. The results and discussion provide implications for accountability policy and instructional practice, as well as future research.

Authors

• Caitlin Farrell, University of California - Berkeley

• Julie A. Marsh, University of Southern California