Design to Engage: Features of the CALL (Comprehensive Assessment of Leadership for Learning) Formative Feedback System That Promote Leadership for Learning in Middle and High Schools

• In Event: Roundtable Session 18.
  In Roundtable Session: The Testing, Development, and Implementation of an Online Instructional Leadership Formative Assessment Tool

Abstract

This paper reports the results of a design study conducted in Spring, 2011 to inform the development of formative feedback from the Comprehensive Assessment of Leadership for Learning (CALL). The study uses research on the principles of effective feedback (DeNisi and Kluger, 2000; Wimer, 2002), school leader and professional development (Darling-Hammond et al, 2007; Goldring et al, 2009; Gusky, 2005; Smylie et al, 2005) and adult learning (Knowles, 1980; Merriam & Caffarella, 1999; Wood & Thompson, 1980), and an analysis of features of formative feedback systems with purposes similar to the CALL system to inform the development of a conceptual framework of research-based principals of effective feedback design. Based on this review, CALL researchers developed feedback templates using survey responses from six middle and high schools, and conducted cognitive interviews with six principals and 18 teachers regarding their experience with the CALL survey and draft feedback, to answer the following research questions:
• In what ways does a 360 degree formative assessment of distributed instructional leadership in middle and high schools affect teacher and administrator reflection and direct school improvement?
• What elements of feedback design facilitate principal instructional leadership and promote practitioner reflection for school improvement?
The interview data were analyzed to identify themes including opportunities for feedback perceived as important to principals and teachers, and forms of feedback design perceived to be most useful and informative to support a change process.
Opportunities for feedback included:
• The experience of taking the survey, which communicated a clear theory of action to survey respondents. Teachers reported that the survey promoted informal conversations about professional practice among teachers; created expectations for future action in the school; and responding to the survey was perceived as a means of communication with the principal about the school’s strengths and weaknesses.
• Analysis of survey results. The survey is designed as a standards-based rather than norm-referenced survey, and principals were more interested in data communicating levels of practice (what does a “3” mean in terms of practice?) rather than comparative data (how did my school do compared to other schools?).
Principals also identified specific design features that promote effective communication of results and support mobilization for improvement, including:
• The transparency of the survey, which communicates effective research-based practices to survey participants and gives the school a sense of areas it may need to work on;
• Presentations of survey results that promote clear identification of current and desired practices;
• Item-by-item guidance on steps the school could take to improve practice for that particular item, connected to a larger vision or theory of action for effective leadership practice.
• The survey’s focus on distributed leadership rather than the leadership of a particular individual.
The paper has important implications for the design of formative assessment and feedback systems for middle and high schools. This is particularly important given expanding opportunities for automated, on-line feedback that provide limited access to external consultants who can help school leaders interpret data and motivate school change.

Authors

• Carolyn Kelley, University of Wisconsin

• Steven M. Kimball, University of Wisconsin - Madison

• Matthew A. Clifford, American Institutes for Research

• Seann Mason Dikkers, Ohio University - Athens