Implementing Quality Criteria in Designing and Conducting a Sequential Mixed-Methods Study

• In Event: Ensuring Quality in Mixed-Methods Research in Higher Education: From Theory to Praxis

Abstract

Ensuring quality in mixed methods studies can be especially challenging because of the intended mixing of qualitative and quantitative data to produce credible meta-inferences. The quality of the respective quantitative and qualitative findings directly affects the quality of the inferences drawn from the whole study. Mixed methods researchers are faced with the need to address validity checks in all quantitative and qualitative strands of their studies to enhance legitimacy of the final results. Special consideration should be given to types of mixed methods designs. Different chronology of quantitative and qualitative data collection and analysis procedures, concurrent vs. sequential, and the order of the quantitative and qualitative strands in the study, quantitative or qualitative first, call for special strategies aimed at minimizing threats to validity caused by mixed methods design features. The objective of this presentation is to discuss how quality criteria were implemented in designing and conducting a sequential design mixed methods study of student engagement in learning applied research methods online, and the lessons learned from this process. The purpose of this study was to explore from students’ perspective how they engage in learning applied research methods in computer-mediated environment using quantitative survey and qualitative interviews with graduate students who successfully completed one or more online research methods courses from a research one university in the south-east. The study consisted of two sequential phases, a quantitative, survey phase followed by a qualitative, individual interview phase. The quantitative phase was guided by the following research question: What teaching and learning strategies do students perceive effective in learning applied research methods online? One hundred and nineteen students responded to the online survey (84.5% response rate). In the follow-up qualitative phase 13 students were interviewed to elaborate on the quantitative results. Additionally, the qualitative phase sought to understand how students value and apply knowledge of research methods acquired in online courses. To secure the quality of the study meta-inferences generated from the sequential employment of quantitative and qualitative methods, reliability and validity of the survey data was assessed using regular psychometric procedures, and the credibility of the interview data was evaluated through the use of several qualitative data verification strategies. Specific legitimation issues related to a mixed methods sequential design were addressed by elaborate selection of the sample for the qualitative phase and by using the results from ANOVA and regression analysis tests to guide the qualitative data collection. The participants for interviews were systematically selected from those who responded to the survey using a statistical procedure that helped identify consistency of students’ responses on four dependent variables tested in the first phase: two measures for perceived learning effectiveness, one for perceived acquired knowledge and one for knowledge application. A maximal variation procedure was used to select participants who varied on demographic characteristics when more than one case in a certain category was available. The interviews were carefully designed to provide understanding of the statistical results from the first phase and connect to the individual responses of the interviewees on the survey.

Author

• Nataliya V. Ivankova, The University of Alabama - Birmingham