Statistical and Social Validation of Leadership Scales

• In Event: 72.014 - Feedback and Coaching to Promote Professional Reflection and Learning of School Leaders

Abstract

Objectives
This paper reports on results from the piloting of the ICT-based inventory in seven country contexts, i.e. Australia, Cyprus, Denmark, Norway, Spain, Sweden, and Switzerland with a particular focus on the statistical as well as the social validation of the 29 leadership scales used in the inventory. Besides testing psychometric measures, it is argued that social validation in terms of meeting participants’ expectations and the perceived usefulness related to using the feedback for reflection on strengths and weaknesses, and possible areas for improvement is also crucial in order to improve the inventory.

Perspectives
The scales developed and used as part of the inventory draw on school leadership research conducted over the last decades and aim to measure constructs related to key competencies required of school leaders. The inventory formats include questionnaire scales, cognitive tests, and implicit test formats. The scales comprise 18 general school leadership competencies and nine tasks specific school leader competencies.

Methods
First, classical test theory was used to test for homogeneity, item difficulty etc. Second, further analytic techniques to assess the quality of the scales include Rasch analysis which makes it possible to investigate unidimensionality, irrelevant variances (misfits) and representation of scales for the whole dataset as well as how the scales work within each country context (DIF-analysis). Third, methods to investigate social validation includes a quantitative survey which was sent to participants in their own language towards the end of the project aiming to map and compare key features of their learning experience across countries, including their perceptions of the inventory and the feedback provided, as well as the impact of these activities on their reflection and leadership behavior.

Data sources
Altogether, the datasets include responses from 998 school leaders who all took part in the international piloting of the inventory. Further data sources include transcriptions of interviews with participants.

Findings
The results of the Rasch analysis show overall good power of fit for most of the scales. For some scales the Person Separation Index is too low due to problems with skewness. For problematic scales, it is suggested to include more items in each scale, preferable items which are more difficult to rate highly. Related to social validation, the participants state that their feedback report partly confirmed what they already knew but at the same time the feedback gave them ideas, especially related to the relationship between scales they scored high and low on, which they reflected on and discussed in coaching sessions. Two thirds regard the self-assessment exercise as helpful for further professional development.

Scholarly or scientific significance
Using Rasch analysis implies a stricter way of testing the scales than is normally applied for such inventories. However, the results show improvement areas which would not have been easily discovered without this model. Especially, the validation of the scales in a couple of countries indicates some challenges related to item formulations and culture contexts. The paper provides examples of more precise actions to be taken as a result of this type of analysis.

Authors

• Stephan Gerhard Huber, University of Teacher Education Zug

• Rolf Olsen

• Alexandra Petridou, University of Manchester

• Marius Schwander, University of Teacher Education Zug

• Christian Brandmo, University of Oslo

• Jonas Melker Höög, Umea University