Valid Assessment of Competence Change in Higher Education

• In Event: Valid Assessment of Learning Outcomes in Higher Education: International Perspectives and Methodological Challenges

Abstract

Longitudinal research in higher education faces several challenges. First, data usually is collected on volunteers, who have little incentive to participate (repeatedly), and sometimes miss measurement sessions or drop out of the panel completely. Second, large numbers of college or university students are difficult to reach (repeatedly) and therefore samples tend to be small. Third, unlike in primary and secondary school, where students usually are the same age, have had the same amount of schooling, and have similar prior knowledge, students in higher education do not belong to fixed classes and therefore they may attend courses with students who are in different semesters and have different prior knowledge and competences. Consequently, conducting studies in higher education may mean having a combined cohort-longitudinal design (Hof et al., 1977), which requires an appropriate statistical model. Fourth, if the difficulties and discriminations of the test items measuring the same competence construct show item parameter drift (Bock et al., 1988) over time, responses may reflect genuine growth in competence or may simply be the result of invariant item parameters (Millsap, 2010). To handle item parameter drift such items should be removed before analyzing growth; however, doing so would result in removing information even if there are items with differential item functioning.
Appropriate methods of analyzing competence growth in higher education are needed to deal with the aforementioned challenges and thereby obtain generalizable and valid results. If the data is affected by missing cases due to panel dropouts or by measurement invariant items, and if the measured constructs are not directly observable as students’ competences, traditional methods such as multilevel models or latent growth curve models of analyzing longitudinal data cannot be employed to answer questions on growth. However, each of these two methods has its strengths, which leads to the idea of combining them in a multivariate multilevel structural equation/IRT model for repeated measures.
In this paper, a newly developed multivariate multilevel IRT model for repeated measures is presented. The model is designed to address an educational research question given genuine data containing the problems described above. Estimation of the model is conducted within a Bayesian framework with a Markov Chain Monte Charlo (MCMC) algorithm. In this model, dependencies between students’ observations due to repeated measures are considered not, as usual, by clustering observations within persons, but rather by clustering observations in semesters. By doing this, missing values from single students at one or more measurement occasions present no challenge (as long as they are at least missing at random), because growth in competence is measured as growth of semester cohorts. This means that focus is on growth patterns of different semester cohorts without conditioning on unreliable individual growth trajectories based on few measurements. The model was tested with simulated data, and finally the project’s real data results are discussed. The real data contained observations of 3,120 students on four occasions, and were assessed to represent the growth in competence of several cohorts.

Authors

• Jean-Paul Fox, University of Twente

• Susanne Schmidt, Johannes Gutenberg-Universität Mainz