What Are You Driving At? How School Leaders Fill Gaps in Data and Evidence When Making School Improvement Decisions

• In Event: From Data and Evidence to Decisions for School and District Improvement: Synthesizing Findings and Extending Theory

Abstract

Recently, data-driven decision-making (DDDM) policies have moved beyond encouraging teachers to use data in their classrooms to emphasize leaders using data and evidence to guide school improvement decisions such as developing annual improvement plans or selecting professional development (Data Quality Campaign, 2011; Means, Padilla, DeBarger, & Bakia, 2009). Principals report that they try to use data when making decisions but that they encounter barriers, e.g., lack of adequate and timely data or limited analytic skills (Anderson, Leithwood, & Strauss, 2010; Author, 2006). Despite these challenges, school leaders continue to make decisions. Research has yet to probe what leaders do when they encounter inadequate or old data or other gaps in data and evidence. How do school leaders fill gaps in data and evidence when making school improvement decisions? How do their responses influence their decisions?

This paper begins to address these questions with an in-depth qualitative study of the leadership teams in two middle schools. I developed a conceptual framework drawing on theories of organizational learning and decision making from an institutional perspective (Argyris, 1976; Argyris & Schon, 1996; March, 1991, 1994; March & Olsen, 2008; March & Shapira, 1987). The framework highlights the multiple ways in which decision makers gather and use a broad range of data and evidence, and how the resulting decisions call for different degrees of changes in practice. The data predominantly come from observations of weekly leadership team meetings in two middle schools in a large, urban district. I selected this district not only for its DDDM policies, but because its leaders could also access multiple forms of data including state/local test scores, staff and community satisfaction surveys, and classroom observation data. I supplemented the meeting observations (73 hours) with 31 interviews and reviews of related documents.

I found that leaders in these two schools used multiple forms of data and evidence when making school improvement decisions, but that student test data typically emphasized by DDDM policies played a relatively small role. Instead, leaders relied most heavily on data on their schools’ operations—implementation data—and evidence from potential programs or strategies—intervention data.

Despite having access to several forms of data, leaders faced data and evidence gaps for every decision. In response, leaders actively searched for additional data by formally and informally surveying colleagues and researching potential programs. If these external searches fell short, they used their own experiences as data. When leaders relied most heavily on their own personal experiences to fill gaps, the resulting decisions called for adjustments to current practice. When leaders filled gaps with a balance of data and evidence gathered from external sources and personal experiences, the resulting decisions called for fundamental changes in practice.

Existing data and evidence use research highlights how personal experience can function as a lens through which decision makers make sense of data. These findings highlight the how personal experience can function as actual data and how they can influence decision outcomes.

Author

• Lydia Rainey, University of Washington