Survey design in education policy research: Innovative tools, tips and lessons learned

Group Submission Type: Pre-conference Workshop

Description of Session

Within the social sciences, surveys are among the most common methods of data collection. Although at a first glance, designing a survey questionnaire can be seen as a relatively easy endeavour, in practice a number of choices must be taken in order to limit non-response, ask valid and accurate questions within the constraints that characterise every research project. Some aspects can be particularly challenging: how to ask teachers and principals in an effective way about complex concepts of a subjective nature, such as attitudes or motivations? How to look for truthful answers when questions on sensitive topics are asked (salary bonuses, students’ selection, teaching to the test)? How can comparability of data be pursued in different national and sub-national contexts? Apart from offering an introduction to questionnaire design, this workshop offers its participants the opportunity to learn and discuss about ways to face these particular challenges as well as the chance to analysing and sharing experiences with survey implementation in the context of on-going comparative and international education projects.

Proposal

Names of Workshops Leaders/Organizers:
Clara Fontdevila, Antonina Levatino, Lluis Parcerisa, Cecilia Peraza Sanginés, Emily Richardson, Andreu Termes and Antoni Verger.

Workshop Rationale:
Surveys are fundamental tools of data collection and are increasingly used in educational research. Their implementation is, however, not exempt from challenges that researcher has to overcome within the time and cost constraints characterising every research project. Right choices valid for all do not exist, rather the design strictly depends to the specific research questions, goals and population under study.
This workshop aims to provide its participants an overview of the theoretical basics and empirical evidence related to questionnaire design with particular regard to the cognitive process of survey responding, question wording and response formats. Additionally, it seeks to offer its participants some methodological tools to overcome some of the most challenging aspects of questionnaire design: the design of effective questions to capture complex concepts of a subjective nature, the planning of ways to ask about sensitive topics trying to avoid social desirability biases and the gathering of comparable data in cross-national surveys.
The workshop is therefore directed to those who wants to learn more about survey research, those who plan to conduct a survey for the first time but also to those who would like to share and discuss experiences with this method of data collection and associated challenges. No previous experience in survey research is thus needed, even though some practical experience in the implementation of surveys could be valuable.
In order to reach the aforementioned goals, the workshop will be articulated in three parts lasting two hours each. In the first two hours, the workshop will start with an initial presentation of the strengths and weaknesses of surveys in comparison to other research designs. This will be followed by the provision of an overview of the key elements of questionnaire design as highlighted by existing literature. We will especially clarify why operationalizing concepts into survey questions can be intricate, which aspects may affect responses and which factors could cause a respondent to do not provide the real answer requested by the researcher or provoke different respondents to understand differently the same question. Afterwards, an interactive dynamic will follow where the participants will be divided into smaller working groups each dealing with a particular ‘challenge. The facilitators will accompany the working groups by offering the participants support in understanding the problems related with the faced problem. They will also provide useful notions and examples that will help the participants to identify meaningful and viable strategies to overcome the assigned challenge. After this exercise, in the second part, the workshop will engage participants to share the work done in the assigned groups by presenting the others the problems faced and the strategies identified to overcome them. In the third part, the workshop will continue by presenting the participants concrete examples of challenges faced by pieces of survey research. In particular, the design of a cross-national survey in the framework of the research project "REFORMED - Reforming Schools Globally: A Multiscalar Analysis of Autonomy and Accountability Policies in the Education Sector" and the implementation of the "Teachers Motivation WG" project will be presented and become the objects of evaluation and discussion. The facilitators will also encourage participants to share their experiences with questionnaire design and/or their doubts with survey implementation in order to give and receive useful feedback for their personal research.

Duration and Size: The proposed duration of this workshop is six hours (double section).
The maximum number of participants is 40.

Special Requests: The room should be equipped with a projector.

Sub Unit

• Pre-Conference Workshops

Workshop Organizers

• Clara Fontdevila, Universitat Autònoma de Barcelona

• Antonina Levatino, INED & Universitat Autònoma de Barcelona

• Lluís Parcerisa, Universitat Autònoma de Barcelona

• Cecilia Peraza, Universidad Nacional Autónoma de México

• Emily Richardson, Save the Children

• Andreu Termes, Universitat Autònoma de Barcelona

• Antoni Verger, Universitat Autònoma de Barcelona