The Challenge of International Comparison: Seven Variants of the Validity–Comparability Compromise

• In Event: 75.022 - Dilemmas, Tensions, and Trade-Offs: Some Methodological Conundrums in Education Research

Abstract

Objectives
International comparisons of curricula or classroom practice are critically contingent on how the research design addresses the dual priorities of validity and comparability. These priorities are frequently in tension and researchers undertaking international comparative research must find a satisfactory balance between these competing obligations. Empirical examples drawn from recent research (Author, 2001, 2003, 2006, 2010) illustrate seven “dilemmas” facing researchers undertaking international comparisons of either curricula or classroom practice and strategies for their accommodation.
Theoretical Framework
Perhaps only the drive to categorize is more fundamental than our inclination to compare (cf. Lakoff, 1987). Indeed, the two activities are intrinsically entwined. In this paper, commensurability is interpreted as the right to compare (cf. Stengers, 2011). And this right to compare cannot be assumed, but is contingent on our capacity to legitimize both the act of comparison and the categories through which this act is performed.
Data Sources
The presentation draws on data from: (i) The Learner’s Perspective Study – a 16-country comparison of the practices and products of well-taught mathematics classrooms, and (ii) The Alignment Project – investigating curriculum alignment as a dynamic process of selective interpretation in Melbourne, Beijing, Helsinki, and Tel Aviv. Both studies combine video data with document analysis and interviews with key stakeholders in each setting.
Warrants for Arguments
In both of these studies, it was necessary to address seven dilemmas related to the validity-comparability compromise. Each dilemma can be illustrated by reference to particular key constructs.
Dilemma 1: Cultural-specificity of cross-cultural codes
Use of culturally-specific categories for cross-cultural coding (e.g. participation, mathematics)
Dilemma 2: Inclusive vs Distinctive
Use of inclusive categories to maximize applicability across cultures, thereby sacrificing distinctive and potentially explanatory detail (e.g. lesson structure).
Dilemma 3: Evaluative Criteria
Use of culturally-specific criteria for cross-cultural evaluation (e.g. student-student interaction)
Dilemma 4: Form vs Function
Confusion between form and function, where an activity coded on the basis of common form is employed in differently situated classrooms to serve quite different functions (e.g. “between desks instruction”)
Dilemma 5: Linguistic Preclusion
Misrepresentation resulting from linguistic preclusion encrypted in discourse conventions (e.g. Japanese classrooms underplaying intellectual ownership)
Dilemma 6: Omission
Misrepresentation by omission, where the authoring culture of the researcher lacks an appropriate term or construct for the activity being observed (e.g. pudian [Chinese] or učitelská ozvěna [Czech])
Dilemma 7: Disconnection
Misrepresentation by disconnection, where activities that derive their meaning from their interconnectedness in the context being studied are separated for independent analysis (e.g. obuchenie [teaching/learning]).
Scholarly Significance
The pursuit of commensurability in international comparative research by imposing general classificatory frameworks can misrepresent valued performances, school knowledge, and classroom practice as these are actually conceived by each community and sacrifice validity in the interest of comparability. The “validity-comparability compromise” has significant implications for international comparative research. Partnerships with those being compared can minimize misrepresentation, but the necessity of the compromise is inescapable. The interpretation and application of international comparative research will be critically contingent on researchers’ capacity to address those “dilemmas” pertinent to their particular design.

Author

• David J. Clarke, University of Melbourne