More Than Test Scores: Designing Accountability Systems That Include Noncognitive Factors

• In Event: 53.060 - Rethinking Accountability: Early Research on California's CORE Waiver Districts

Abstract

Purpose

In recent years, policymakers have shifted away from test-based accountability towards systems that consider multiple dimensions of school performance, including students’ “non-cognitive factors,” beliefs, dispositions, attitudes, traits and behaviors that predict a broad range of important outcomes in school and life (Almlund, Duckworth, Heckman, & Kautz, 2011; Blad, 2015; West et al., 2016). Though researchers debate whether these constructs should be used in accountability (e.g., Duckworth & Yeager, 2015), some policymakers have initiated efforts to use these measures to evaluate schools. As states develop new multiple-measure accountability systems under ESSA, it is useful for researchers to examine how policymakers design and adopt these models. This qualitative study examines the case of the California CORE Districts, an early adopter of non-cognitive factors in an accountability system, through the following questions:
1. What influenced CORE leaders’ adoption of non-cognitive factors in its School Quality Improvement System (SQIS)?
a. What political factors affected the adoption of the SQIS?
b. How did cognitive paradigms and normative ideas shape the adoption process, and to what extent did CORE leaders strategically frame these policy changes?
c. What role did research play in the design and adoption of the SQIS?

Theoretical Framework

Our theoretical framework draws from two theories of policy change: Kingdon’s (2011) multiple streams framework and Campbell’s (2002) politics of ideas. In the multiple streams model, Kingdon argues that policies are adopted through the interaction between three distinct process streams: first, the definition and recognition of social problems; second, the development of policy alternatives; and third, political forces. Campbell (2002) explains that policymakers’ choices are constrained by their assumptions about cause and effect relationships as well as their beliefs about values and identities. In addition, policymakers may intentionally frame a particular alternative within the context of their preferred paradigms; thus, policymakers both shape and are limited by ideas.

Method & Data Sources

Using an instrumental case study design (Creswell, 2007; Stake, 2005), we examined documents (e.g., agendas, minutes, presentation slides, the NCLB waiver application) and conducted interviews with key CORE staff (n=4) and administrators from CORE Districts (n=18). We analyzed documents and interview transcripts through multiple rounds of coding and analytic memos (Maxwell, 2013; Saldaña, 2013).

Results

In developing the SQIS, CORE leaders were influenced by a general anti-testing sentiment, reflecting public frustration with NCLB, and supported by politically powerful teachers’ unions and parent organizations. The policy alternative of non-cognitive factors emerged through popular media sources on grit and character (e.g., Duckworth, 2013; Tough, 2012). Though policymakers hoped to move away from NCLB-style accountability, their assumptions about accountability systems limited their approach and led them to reframe non-cognitive factors as objective, school-based outcomes.

Scholarly Significance

This case study yields insights about the policy adoption process and about how policymakers engage with research. More specifically, this study sheds light on how policymakers conceptualize non-cognitive factors, and how this conceptualization might affect the new accountability systems that states are considering under ESSA.

Authors

• Taylor N. Allbright, University of Southern California

• Julie A. Marsh, University of Southern California

• Heather Hough, Stanford University