A Methodological Framework for AI-Assisted Image Stimulus Design

• In Event: Implications of AI for Qualitative, Quantitative, and Computational Sociological Research

Abstract

A Methodological Framework for AI-Assisted Image Stimulus Design

Surveys and survey experiments have long relied on text-based stimuli to measure and manipulate constructs of interest. Yet text has well-known limitations: it requires respondents to mentally construct visual representations, introduces interpretive ambiguity, and often lacks the ecological validity of stimuli people encounter in everyday life. Images can address these shortcomings, conveying richer contextual cues and reducing reliance on individual imagination, and recent advances in AI image generation have made visual stimuli far more accessible to researchers.

Realizing these advantages, however, requires solving two challenges. The first applies to any use of image stimuli: visual operationalization. Not all theoretical constructs translate easily into images; some attributes are concrete and straightforward to depict, while others resist visual representation. We introduce a generability index that quantifies how faithfully an attribute is realized, and evaluate two creation approaches, end-to-end generation and retrieval from existing image databases, each under recommended best practices.

The second challenge is specific to survey experiments: precise control so that treatment/control images only differ by intended treatment variations. Even when a feature is easily visualized, AI models introduce subtle unintended variation each time an image is generated, quietly altering non-focal features across conditions in ways that are difficult to detect. We define confounding metrics to measure this variation and find that both generation and retrieval systematically fail the control requirement. We propose a two-step workflow in which a base image is first generated or retrieved, then edited to enforce the focal feature while holding context fixed, and show it substantially reduces confounding.

We illustrate the framework with two case studies: one on perception of urban neighborhoods using street view images, and one on perceived Asianness and educational outcomes. This work contributes to survey methodology, experimental design, and computational social science.

Authors

• Han Zhang, Brown University

• Yishi Yin, Brown University