Assessing Study Abroad: A Quantitative Analysis Design

• In Event: Constructing identities through studying abroad

Proposal

As the participation in study abroad increases rapidly, a need to supply evidence of learning outcome through more rigorous study abroad program assessment and deeper research is growing. Although the history of assessment in higher education is quite long, the field of study abroad is still a latecomer to assessment. Despite the variety of the study abroad assessment domains, little work has been done in the literature concerning the impact of study abroad on degree attainment. Study abroad, as one of the high-impact activities, aims to enhance students’ learning and success (Kuh, 2008). To address the gap in the literature and find out how study abroad affects bachelor degree attainment, this study discusses the current methodological and design challenges within the field of study abroad, identifies factors in the literature that influence both bachelor degree attainment and the selection into study abroad, and then proposes a quantitative analysis design that gauges the impact of study abroad on bachelor degree attainment.
To address the gap in the literature and find out how study abroad affects bachelor degree attainment, the research design is to investigate the following research questions:
• What is the impact of participation in study abroad on bachelor degree attainment?
• How do institution type and institutional selectivity differentiate the impact of participation in study abroad on degree attainment?
I anchor this research design to an existing data set—Educational Longitudinal Study of 2002 (ELS:2002), which surveyed a nationally representative sample of young people as they progressed from tenth grade and twelfth grade to schooling beyond high school and to the world of work. The base year data were collected during the 2001-2002 academic year when students were 10th graders. Follow-up surveys of the sampled students were administered in 2004 (1st follow-up) when they were in 12th grade, 2006 (2nd follow-up) and 2012 (3rd follow-up). The responses to the 3rd follow-up surveys provide a rich source of data for research on college access, choice, activities and degree completion, where I drew my outcome variable – bachelor degree attainment, and the main independent variable – participation in study abroad. I also focused on the first follow-up survey because it allowed me to draw some variables of students’ previous academic achievements, gender, race and socioeconomic status before entering colleges. The variable selection process is theory driven, but limited to the availability of the dataset. This dataset is not initially design for the purpose of study abroad research, the limitation of which cause potential threats to internal validity. I have separate discussions on this issue in the end.
Much of the existing study abroad research has been undermined by methodological and design shortcomings, such as selection bias, ignorance of institutional level data, and generalizability. In order to achieve an unbiased estimate of the impact of study abroad on bachelor degree attainment, I first use PSM to balance differences between study abroad participants and non-study abroad participants based on the observed individual level variables at the baseline—prior to study abroad, including race, gender, socioeconomic status (SES), high school GPA and Pell Grant I. Additionally, due to the clustering nature of the dataset, the relationship between selection to study abroad and student level characteristics may differ across institutions or interact with institution level variables, such as institution type, and institution selectivity. Clustering in this study design is central feature. So, I use full multilevel modeling with both fixed and random effects to do PSM.
The following analysis takes a multilevel logistic regression modeling approach called Hierarchical Generalized Linear Modeling (HGLM) due to the following two reasons. First, the outcome is categorical—either attaining a degree or not attaining a degree, therefore, the expected values result from probability distributions other than normal distribution. Second, the dataset is clustered—the relationship between study abroad and bachelor degree attainment, including may differ across institutions or interact with institutional level variables, such as institution type, and institution selectivity. The first step of analysis is to assure that the extent to which the odds of attaining a bachelor degree varies from one institution to another (Sommet & Morselli, 2017). I will use a null model to estimate the proportion of variance in the chance of attaining a bachelor degree rather than not attaining it that lies between institutions. Once within and across institutions variance has been portioned, multilevel modeling is applied to examine whether there is significant variance at the student level intercept and slopes. Based on the literature and available student level variables in the data set, I include race, SES, gender, Pell II, delayed employment, number of high impact activities attended in college, and educational expectations collected when the students were in high schools as confounding variables, in addition to the main independent variable—study abroad. In the final step, I will use another random regression model to assess whether the significant variance in intercept and slopes are related to institutional characteristics, including both institution type and institutional selectivity.
Even though I use PSM to account for the self-selection of students to study abroad, the study itself by no means implies that it is free of threats to its validity. Threats to internal and external validity from this design will be discussed in the paper. Yet another weakness in this study design is failure to account for study abroad program characteristics. ESL dataset is not initially designed for the purpose of study abroad research, which limits the depth of the analysis.
The significance of the study is to propose a quantitative research design analyzing any large-scale dataset in order to assess the impact of study abroad on student success outcomes, which can be bachelor degree attainment, as discussed in this paper, or any other outcomes, such as graduation rates, retention rates, etc. As discussed previously, research in the field of study abroad assessment is facing three main challenges: selection bias, ignorance of institutional characteristics, and generalizability, this research design is able to reduce selection bias, and be generalized into a broader population.

Author

• JIE DAI, University of Kentucky