Do We Really Care About That?

• In Event: Media Comparisons: To Do or Not to Do, That is the Question

Abstract

The logic of experimental inquiry is seductively simple: hold everything constant, vary one factor, and any difference can be causally attributed to that factor (i.e. ceteris paribus or ‘all else being equal’). In practice, however, education provides particularly unwelcoming grounds to meet these simple conditions; classrooms are messy, technology and pedagogy hopelessly entangled. New technologies bring distinctive representational and interactive affordances (e.g. virtual reality’s immersion or LLM chatbots’ dialogic interaction style), which inevitably reshape instruction (Kozma, 1994). To control them away in a media comparison is to amputate precisely what makes the medium educationally interesting.

Researchers basically have two options. Path A exploits the technology’s affordances, comparing a rich, authentic implementation of Novel Technology against business‑as‑usual teaching. The resulting effect is hopelessly confounded: is learning better because of the medium, the new activities it enabled, or both? Path B strives for a “fair” test by matching instructional features across conditions; that is, same content, tasks, feedback, pacing, until only the delivery channel differs between conditions. Clark’s meta‑analyses showed that under such tight control any medium advantage evaporates (Clark, 1983, 1994). The closer we get to perfect matching, the more our treatment degenerates into a caricature of the technology, and the research question itself becomes trivial.

Consider evaluating Socratic dialogue with ChatGPT. Our attempts in equivalent control might suggest peer‑to‑peer Socratic dialogue. Yet face‑to‑face talk carries rich social cues, so we switch to text chat. But peers may write more personalised messages –another confound–, so we script their utterances. But students may suspect bias toward or against AI, so perhaps we should conceal the partner’s identity altogether. At each iteration we either dilute the AI’s real affordances or allow new differences to creep in. Where does legitimate control end and overengineering begin? There is no principled stopping rule. The logical endpoint of perfect matching is a comparison of near-indistinguishable activities.

The implication is not that rigorous design is futile. Instead, I suggest that the unit of analysis should shift from “medium‑A versus medium‑B” to the affordance‑method bundle. Causal claims should target clearly articulated instructional functions (e.g., adaptive feedback, distributed rehearsal, collaborative knowledge building) and examine how different technologies instantiate those functions under authentic constraints. This would direct our efforts toward understanding how unique affordances, in concert with pedagogy, shape learning processes and outcomes, which is what we really care about.

Author

• Joshua Weidlich, Heidelberg University of Education