The Effect of Conditioning Model Choice: A Simulation Study

• In Event: Future Directions of Educational Measurement in International Large-Scale Assessments

Abstract

The Programme for International Student Assessment (PISA) is an important international study that compares mathematics, science and reading skills of 15-year-olds across countries and has long become an established tool of soft governance for educational policies (Meyer, 2014). In order to provide a valid basis for educational policy-making, the underlying statistical methodology must be appropriate for comparisons across time and countries. Our simulation study enables us to address multiple questions around the computation of plausible values and the conditioning model such as:
Does it matter whether every student answers questions in all domains?
Can the addition of single variables to the conditioning model lead to substantial changes?
What are the effects when variables are entered directly into the conditioning model in contrast to when principal components are extracted first?

The statistical methodology is a core element of PISA, but it faces a lot of controversies. For a start, it is a complex, multi-step procedure, which has been criticized for being opaque (Goldstein, 2017) and not well understood outside the psychometric community. This holds especially true for conditioning where background variables are used in addition to the test responses to adjust the student achievement distributions.

We simulate our own data based on characteristics of PISA 2012.

In order to enable the high applicability, we use both (data) structure and methodology from PISA 2012 as a guideline. We extract the correlations between cognitive achievements and background questionnaire data for eight countries. Based on these correlations, we simulate both cognitive items and background questionnaires. Afterwards, we use this data to compute plausible values. Thereby, we vary different aspects, such as the conditioning variables or the used observations. As a result, we end up with different sets of plausible values based on different conditioning model choices. One big advantage of simulated data is that we know the ‘true values’, which allows us to validly compare these different plausible values in terms of bias.

While there has been a lot of work around methodology in international large-scale assessments, little work has been done around the conditioning models there. Those then mostly have concentrated on a simplified setting (e.g. only two background variables considered) or focused on a specific aspect. This is the first paper to facilitate multiple comparisons around conditioning model choice in a realistic setting.

Our simulations yield multiple interesting results which can and should be taken into consideration when designing an international large-scale assessment or evaluating one. First, results in domains were all students answered at least one question are less biased then domains were the values for some students are ‘imputed’, because they were not assigned a single question there. Second, single variables in the conditioning model can significantly change the results. Careful sensitivity and robustness check for the variable choice should be conducted. Last, at least it does not matter whether the variables are entered directly into the model or principal components are computed beforehand. Those effects are negligible.

Authors

• Laura Zieger, University College London

• John Jerrim, University College London