Advanced methods in cross national and cultural comparisons using process data

Group Submission Type: Formal Panel Session

Proposal

International large scale assessments (ILSAs) offer important information on the development of cognitive skills and the consequences of differences in the distribution of these skills across subgroups (von Davier, Gonzalez, Kirsch, Yamamoto, 2013). The ILSAs such as the Programme for International Student Assessment (PISA), the Programme for International Assessment of Adult Competencies (PIAAC), and Trends in International Mathematics and Science Study (TIMSS) are usually regarded as high-quality metrics to provide an honest view of comparisons across countries, inform education policy and practice in the participating countries, and more broadly to improve teaching and learning in general as aspirations to promote the best possible talent development. Cultural diversity is an interesting and unavoidable topic in the ILSAs. To explore how culture, language, and social economic status influence the ways people think and solve problems not only provides deeper understanding for respondents’ behavioral patterns, but also support the improvement in test development to derive a more valid scale (WestEd, 2010).

Increasingly use of computer-based testing in ILSAs provide possibilities to capture more data than paper-based tests. These computer-based assessments (CBAs) allow us to collect not only the response data but also the time-stamped response process data in log files. The process data, a new data source for research, contains rich information that cannot be obtained from the response products. Leveraging the process-based information has great promises for assessment, ranged from validating or improving the validity of the test scores to furnishing evidence of new constructs that are not covered by the test scores.

Cultural differences may be associated with how respondents interact with the items, and hence process data collected from culturally diverse samples may show such differences (He &von Davier, 2016). This effect may be even more salient than item-score by population interactions, so the distribution of action sequence patterns may differ noticeably from country to country. This leads to new research questions and future studies: for instance, whether test takers from different cultural backgrounds or different countries adopt the same solution to a complex task, and whether the differences in strategies have an association with test takers’ responses in tradition (correct/incorrect) item-scores or in responses to questionnaire items given along with the test. Therefore, we believe that process data will play an important role in providing a new angle to explore the cultural diversity.

This panel session introduces cutting edge research in developing advanced methods in using process data and timing information to explore differences across language and culture groups. Specifically, we will focus on two objectives: (1) for understanding how individuals from different cultural and language groups engage with measurement instruments and questionnaires, and (2) to explore what we learn from process data that is not possible to observe from observed data. We collected four exciting researches that utilize the response process data to better understand test taker's performance and behaviors in different domains, hence to support improvement in ILSAs.

The first paper stresses the importance of measurement comparability in ILSAs and introduces the advanced psychometric modeling to detect differential response functioning by process data besides responses data. Deep cultural awareness would inform creation, dissemination and evaluation. This study will focus on two aspects: a) identify differences in process data that may violate measurement comparability, and b) develop methodologies for examining measurement comparability using process data. An empirical study regarding differential response function on PISA 2015 across languages, cultures, and item types on science assessment will be shared.

The second paper discusses the findings and lessons in the Bright Futures survey (2017), which is a multi-country survey of students enrolled in tertiary education (undergraduate and master’s courses) in China and the UK. This presentation will exhibit a) substantive differences in response patterns to sensitive questions by Chinese students in China and the UK, b) total and item non-response rates in both groups, c) time invested in completing the questionnaire in both settings, and d) differences in the quality of answers given using a trap question.

The third paper presents an advanced method in identifying sequence patterns in test takers’ problem solving behaviors by different performance groups with a focus on problem solving items in technology-rich environments in PIAAC. Background variables are of importance to further exam consistency of test takers’ behavior patterns across items and countries in large scale assessment. The primary objectives of this study is to focus on investigating the effects in adding background variables as additional features in identifying malleable factors associated with test takers problem-solving proficiency, and exploring the relationship between sequence patterns and cultural background.

The last paper investigates whether including timing data in models for item parameter estimation offers any advantage in accuracy over currently used methods, and further whether the inclusion of timing data in the latent regression improves precision about achievement distributions. This study adopts simulation study and an empirical study using PISA 2015 data to show the promise of using process data to potentially improve overall and subpopulation achievement estimates in international assessments, such as PISA and related studies.

Through this session, we share four studies as examples of the latest progress in analyzing response process data. We hope that these presentations can inspire more creative work in this booming field.

Sub Unit

• SIG: Large-Scale Cross-National Studies in Education

Chair

• Kadriye Ercikan, Educational Testing Service

Individual Presentations

• Use of Response Process Data in Large-Scale Assessments for Cross Cultural Comparisons - Kadriye Ercikan, Educational Testing Service; Hongwen Guo, Educational Testing Service; Qiwei Britt He, Educational Testing Service

• Cross Cultural Implications for Survey Research: lessons from the Bright Futures Project - Dorothee Schneider, University of Essex; Mónica Méndez, CIS; Hector Cebolla Boado, UNED, Madrid; Yasemin Soysal, University of Essex

• Cross National Comparisons in Problem Solving Sequence Patterns Using Process Data - Qiwei Britt He, Educational Testing Service; Hok Kan Ling, Columbia University; Jingchen Liu, Columbia University; Zhiliang Ying, Columbia University

• Improving International Achievement Estimation with Timing Data - David Rutkowski, Indiana University; Leslie Ann Rutkowski, Indiana University; Yuan-Ling Liaw, University of Oslo

• Analyzing heterogeneity in students' reporting behavior across cultures: results from PISA 2015, anchoring vignettes data on motivation for learning - Hana Vonkova, Charles University in Prague; Ondrej Papajoanu, Charles University in Prague; Katerina Kralova, Charles University in Prague; Arash Shahriari-Rad, Charles University in Prague; Gema Zamarro, University of Arkansas

Discussant

• Paulina Korsnakova, IEA