PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Is Not Perceiving: When Synthetic Consumers Can and Cannot Pretest Visual Marketing

Yi-Lin Tsai, Yung-Hsiu, Lai

arXiv:2609.25677v1cs.AIcs.CYecon.GNq-fin.EC

TL;DR

Visual pretesting with synthetic consumers is attractive, but evidence that models perceive consumer meaning from visual cues remains limited. Across six canonical experiments and configurations varying model generation and input format, the study finds frequent human-effect mismatches, partial steering with in-context evidence, and persistent variance shrinkage.

  • Problem

    Whether synthetic consumers perceive the consumer meaning of visual cues, rather than merely seeing their features, remains largely untested despite their use for visual pretesting.

    Source: §1 · p. 2

  • Method

    The study tests six canonical visual marketing experiments across model generations and input formats, then supplies conceptual or empirical evidence through in-context learning.

    Source: §2.3 · p. 5, §4 · p. 11

  • Results

    None of four configurations reproduced more than two of six human effects; evidence-based prompting can steer average responses toward human benchmarks, but response variance remains substantially shrunken.

    Source: §4 · p. 11, §6 · p. 26

  • Takeaways & Limitations

    The Calibrate, Intervene, Deploy protocol delineates when calibrated agents can support decisions, while human panels remain necessary to reveal consumer differences.

    Source: §6.2 · p. 27

    The benchmarks predate both models’ training cutoffs, so prior exposure cannot be ruled out; empirical-prompt alignment demonstrates steerability, not spontaneous agreement.

    Source: §6.3 · p. 30

Abstract

from arXiv · show

Marketers now deploy generative AI agents as synthetic consumers to pretest visual assets such as logos, packaging, and advertising at a fraction of human-panel cost. However, this procedure assumes that a model seeing a visual cue can also perceive its consumer meaning, which is largely untested. We stress-test the assumption using six canonical visual marketing experiments, varying the two levers managers control: model generation (GPT-4o-mini vs. GPT-5.4-mini) and input format (plain text vs. JSON). Every resulting configuration passed the manipulation checks; however, none of the configurations reproduced more than two of the six human effects, and the remainder were nonsignificant. The one exception was a significant reversal of the human pattern. Providing conceptual or empirical evidence through in-context learning steers average responses toward the human effect. Yet steering has a limit: even when it succeeds, a configuration reproduces less than half of the natural spread of human responses and so understates consumer heterogeneity. We integrate these results into an AI governance protocol (Calibrate, Intervene, Deploy) that delineates when synthetic consumers can responsibly screen creatives and when human panels remain necessary.

1. Introduction

Synthetic consumers promise inexpensive visual pretesting, but whether they perceive consumer meaning rather than merely extract visual features remains largely untested. Across six benchmark experiments, models often miss human effects; evidence-based steering improves average alignment but does not recover human response diversity, motivating a governance protocol that preserves a role for human panels.

1. Introduction

  • Synthetic consumers can extract visual features while missing the implicit consumer meaning those features carry, a gap that makes unvalidated pretests a governance risk.The authors describe seeing as literal feature extraction and perceiving as mapping features to latent consumer meaning.Source: §1 · pp. 2–3
  • The study tests six canonical visual marketing experiments across two model generations and two input formats, then evaluates conceptual and empirical in-context evidence as steering.Human panels were matched to the original studies, and rationale topic modeling was used to distinguish seeing from perceiving.Source: §1 · p. 3
  • All four configurations passed every visual manipulation check, yet none reproduced more than two of six benchmark effects, with failures traced to missing or competing semantic associations.Source: §1 · p. 3
  • Evidence-based steering moves average responses toward human effects, especially for the newer model generation, but no configuration restores half of human response spread.Varying persona demographics also does not restore the natural heterogeneity, limiting calibrated agents to average-effect questions rather than segmentation or targeting.Source: §1 · p. 3
  • The Calibrate, Intervene, Deploy protocol delineates decisions calibrated agents can support while reserving consumer-difference insights for human panels.The authors frame the protocol as a control mechanism for responsible generative AI use.Source: §1 · p. 3

2. Theoretical background

Prior synthetic-consumer research has focused largely on text, leaving open whether models reproduce the semantic pathways behind visual judgments. The paper frames this gap as a distinction between feature extraction and learned consumer associations, and tests prompt-based evidence as a steering mechanism while also assessing response heterogeneity.

2.1. Synthetic consumers as simulators of human behavior

  • Prior silicon-sampling research relies predominantly on textual inputs; this study extends the paradigm to visual stimuli and pairs mediation analysis with topic modeling to test semantic pathways.The closest image-stimulus exception is Yoo et al. (2025), whose work is described as an informal demonstration rather than a test of visual perception.Source: §2.1 · pp. 3–4

2.2. Technical priors versus psychological priors in visual perception

  • Technical priors are baseline tendencies to extract physical features, whereas psychological priors map those features onto latent consumer meanings.A model may pass visual recognition checks while missing a human association, or express a competing association that differs from the benchmark.Source: §2.2 · p. 5

2.3. In-context evidence as a steering mechanism

  • In-context learning adds conceptual or empirical evidence to prompts without changing model weights, and the study tests steering across model generations and input formats.Beyond mean alignment, it evaluates whether prompting restores the natural heterogeneity of human responses.Source: §2.3 · pp. 5–6

3. Methodology

The methodology reruns six canonical visual-marketing studies with synthetic respondents, collecting numerical ratings and textual rationales to assess statistical alignment and semantic associations. It compares the published studies as operational benchmarks, then uses a four-stage workflow to generate and analyze zero-shot baselines.

3.1. Overview of the studies

  • Six studies were selected using publication, stimulus availability, perceptual focus, and citation criteria, with published results treated as operational benchmarks rather than absolute ground truth.Table 2 summarizes the benchmark studies.Source: §3.1 · p. 6, Table 2 · p. 7

3.2. Research design using AI participants

  • A four-stage workflow—Design, Data Collection, Analysis, and Diagnosis—generates the zero-shot baselines for RQ1.Source: §3.2 · p. 7, Figure 1 · p. 8

3.3. Design and data collection

  • The design compares GPT-4o-mini and GPT-5.4-mini in plain-text and JSON formats to test whether alignment varies by model generation or prompt structure.Temperature was set to 1.0, top_p remained at 1, and separate API calls were used for each participant to support response variation and sample independence.Source: §3.3 · pp. 8–9
  • Each AI participant receives a study-matched persona and visual stimulus, provides zero-shot ratings and rationales, and then completes a manipulation check.The process is repeated to match the original study sample size, with prior conversation history carried forward across steps.Source: §3.3 · p. 9

3.4. Analysis and diagnosis

  • Analyses replicate the original statistical tests and mediator models, while textual rationales diagnose feature identification and semantic associations.Rationales are treated as observable artifacts rather than faithful accounts of model reasoning; short-text analysis uses Sentence-BERT and BERTopic.Source: §3.4 · pp. 10–11

4. Results

Across six visual-marketing studies, all configurations passed manipulation checks, but none reproduced more than two human effects; one produced a significant reversal. Individual results varied by study and model, while robustness checks found stable repeat runs and no systematic treatment heterogeneity across assigned personas.

4. Results

  • All configurations passed visual manipulation checks, yet none reproduced more than two of six human effects; 4o-mini (TXT) also significantly reversed the human comfort finding.Rationales pointed to recognized visual features but missing or competing semantic associations.Source: §4 · pp. 11–12, Table 3 · p. 12

4.1.1. Study 1: Logo descriptiveness and brand authenticity

  • 4o-mini reproduced the human descriptiveness–authenticity effect, whereas 5.4-mini showed no significant main effect.4o-mini linked authenticity to professionalism rather than the human pathway of ease of processing.Source: §4.1.1 · p. 12, Figure 3 · p. 13

4.1.2. Study 2: Color saturation and temporal distance

  • Neither model reproduced humans’ association between lower image saturation and greater temporal distance.Both models treated saturation as an aesthetic property without connecting it to temporal distance.Source: §4.1.2 · p. 13, Figure 4 · p. 14

4.1.3. Study 3: Logo shape and product attributes

  • On comfort, 4o-mini reversed the human logo-shape effect, while 5.4-mini reproduced it; neither model reproduced the durability effect.5.4-mini’s comfort pathway aligned with softness, whereas 4o-mini invoked support.Source: §4.1.3 · p. 14, Figure 5 · p. 15, Figure 6 · p. 15

4.1.4. Study 4: Logo asymmetry and perceived excitement

  • Neither model reproduced the human association between logo asymmetry and excitement; 5.4-mini’s estimate was directionally reversed.Neither model met the benchmark’s arousal-pathway criterion at zero-shot.Source: §4.1.4 · pp. 15–16, Figure 7 · p. 16

4.1.5. Study 5: Color saturation and size estimation

  • 5.4-mini reproduced humans’ association between greater color saturation and larger estimated size, but 4o-mini’s directionally consistent result missed significance.For 5.4-mini, attention did not mediate the size effect, despite significant upstream links from saturation through arousal to attention.Source: §4.1.5 · p. 16, Figure 8 · p. 17

4.1.6. Study 6: Logo stability and perceived utility

  • Neither model reproduced the human pattern in which unstable logos raise utility for safety-oriented but not neutral products.4o-mini showed an opposite crossover pattern; 5.4-mini’s effect appeared in the neutral category instead.Source: §4.1.6 · pp. 17–18, Figure 9 · p. 18

4.2. Robustness of findings

  • Results remained stable across repeated runs, temperature settings, and image resolutions; carryover was not significant, and assigned personas showed no systematic treatment heterogeneity.Between-subjects experiments were repeated five times and within-subjects experiments twice; the Study 4 age-related excitement difference was the only reported persona exception.Source: §4.2 · pp. 18–19

5. Mitigating perceptual misalignment through in-context learning

The section tests whether conceptual or empirical in-context evidence can steer synthetic consumers toward human judgments and whether effects depend on model generation or input format. Steering often improves average alignment, but semantic recovery and human-like response variability remain limited.

5. Mitigating perceptual misalignment through in-context learning

  • The intervention section frames in-context learning as contextual evidence that shifts models toward a human-like psychological prior.The authors describe this as implicit Bayesian inference and distinguish the psychological prior from a model’s technical default or competing psychological associations.Source: §5 · p. 19

5.1. Testing the effectiveness of prompts with conceptual vs. empirical evidence

  • Conceptual prompts supplied theoretical links without statistics, whereas empirical prompts added benchmark findings and explicit input-output mappings.Empirical prompts therefore test an upper bound on steering rather than spontaneous reproduction; prompts were added during persona instantiation before the visual stimulus.Source: §5.1 · pp. 19–20

5.2. Mean-level alignment after intervention

  • Both conceptual and empirical evidence generally moved model conclusions toward human benchmarks, with empirical evidence producing the most consistent gains.Study 2’s conceptual prompts turned all four zero-shot null effects into significant aligned effects; Study 1’s empirical prompt also recovered a positive effect, but its effect sizes remained about half the human benchmark.Source: §5.2 · pp. 20–21, Table 5 · p. 22

5.3. The model generation matters more than the input format

  • Under empirical evidence, GPT-5.4-mini reproduced the human effect in six of seven cells, versus three for GPT-4o-mini, with identical success rates across formats within each model.The newer model’s recovery was stable across tested JSON styles, while the older model’s performance declined with more granular tagging.Source: §5.3 · p. 23

5.4. Reproduction of the underlying semantic associations

  • GPT-5.4-mini recovered three of five tested semantic associations with empirical evidence, while GPT-4o-mini never recovered more than one.Association recovery required a reproduced main effect, a significant indirect effect, and mediator-related keywords among the dependent variable’s three largest topics; GPT-5.4-mini’s mediator keyword share also rose with evidence.Source: §5.4 · pp. 23–24, Table 6 · p. 24

5.5. Variance shrinkage persists in every configuration

  • All twelve configuration-by-intervention combinations had SD Ratios below 1.0, and every zero-shot value was below .50.Empirical evidence moved each configuration closer to the human mean, but not significantly within any setup; variability increased for GPT-5.4-mini and decreased for GPT-4o-mini in the reported comparisons.Source: Figure 10 · p. 25, §5.5 · p. 26

6. General discussion

The discussion distinguishes accurate visual recognition from perceiving consumer meaning: evidence-based prompting can steer average judgments toward human benchmarks, but does not restore response heterogeneity. It translates these findings into a governance protocol and identifies boundaries on the evidence.

6.1. Theoretical implications

  • Visual manipulation checks can pass while consumer meaning is missed: failures trace to missing or competing semantic associations, not recognition errors.Recognizing a cue, reproducing a human outcome, and reproducing the semantic association underlying the human pathway are distinct tasks.Source: §6.1 · pp. 26–27
  • Evidence-based prompting can steer models toward human benchmarks, but no configuration restores even half the natural spread of human responses.The authors therefore see calibrated synthetic consumers as suitable for mean-level comparisons, while heterogeneity requires human panels.Source: §6 · p. 26, §6.1 · p. 27

6.2. Managerial implications

  • The protocol calibrates agents against historical human benchmarks, intervenes when calibration fails, and returns unresolved misalignment to a human panel.Calibration requires reproducing the benchmark at its original sample size, in both direction and significance; re-testing is advised in each deployment context.Source: §6.2 · p. 27, Figure 11 · p. 28

6.2.1. A governance protocol case study

  • In the running-shoe logo case, GPT-5.4-mini (TXT) failed the zero-shot benchmark, while empirical evidence restored a significant effect roughly half the human benchmark.On the manager’s own logos, the empirical prompt produced a directional signal, whereas conceptual evidence overshot the effect size more than fourfold; without a human baseline, the output was not treated as a human effect size.Source: §6.2.1 · pp. 29–30

6.3. Limitations and future research

  • The evidence is limited to generic demographic personas, two low-cost models from one vendor, and numerical ratings of static images.The authors also note that alignment is benchmarked against published findings rather than absolute ground truth, and that their behavioral evaluation does not unpack internal mechanisms.Source: §6.3 · p. 30

Use of generative AI

The manuscript reports AI use for copyediting and stylistic refinement, while the appendices list study preregistrations and describe study designs, variables, and visual stimuli.

Use of generative AI

  • The authors used Claude for copyediting and stylistic refinement, then reviewed and revised the manuscript and retained responsibility for the final article.The declaration distinguishes manuscript preparation from using LLMs as experimental subjects in the study methodology.Source: Use of generative AI · p. 30

Web Appendix A: Pre-registration for studies

  • The preregistration appendix lists registrations for the studies and robustness checks, including no-intervention, conceptual-evidence, and empirical-evidence prompts.It also lists checks using different temperatures, stimulus resolutions, and AI-generated stimuli.Source: Web Appendix A: Pre-registration for… · p. 36

Web Appendix B: Description of studies and visual stimuli

Main Questions/Hypotheses

The six benchmark studies test whether synthetic consumers reproduce established visual-marketing effects and their proposed psychological pathways. They cover logo descriptiveness, color saturation, shape, symmetry, and stability.

Main Questions/Hypotheses

  • More descriptive logos were expected to elicit stronger brand-authenticity impressions, with ease of processing tested as a mediator.The design varied logo descriptiveness and brand scenario, using the original stimuli and target sample size.Source: Main Questions/Hypotheses · p. 38
  • Distant-future imagination was expected to produce choices of less-saturated images than near-future imagination.Participants chose among low-, medium-, and high-saturation images across three scenarios.Source: Main Questions/Hypotheses · p. 39
  • Circular logos were hypothesized to increase perceived comfort, while angular logos were expected to increase perceived durability, through softness or hardness associations.Participants rated sofa comfort, durability, and cushion softness after viewing one logo shape.Source: Main Questions/Hypotheses · p. 40
  • Asymmetrical logos were hypothesized to seem more exciting than symmetrical logos, mediated by logo-evoked arousal.The study used four logo-pair replicates and measured excitement and arousal.Source: Main Questions/Hypotheses · p. 41
  • Highly saturated objects were hypothesized to attract attention and receive larger size estimates; arousal and attention were also measured as mediators.The study compared high- and low-saturation images of a red laptop.Source: Main Questions/Hypotheses · p. 42
  • Unstable logos were expected to increase perceived utility for safety-oriented products, with the effect attenuated or absent for neutral products.The study varied logo stability and product category and measured utility across six products.Source: Main Questions/Hypotheses · p. 43

Web Appendix C: Heterogeneity analysis by personas

Assigned age and gender personas had little consistent relationship with treatment responses across the studies.

Web Appendix C: Heterogeneity analysis by personas

  • Age and gender effects and their treatment interactions were almost all nonsignificant, with no consistent pattern across configurations; treatment effects were therefore essentially homogeneous across assigned demographics.The exception was a small negative age effect on excitement in all four configurations in Study 4.Source: Web Appendix C: Heterogeneity analysis… · p. 45

C.1 Demographic composition

Table C1 presents the demographic distributions for each study. The samples are about 50% female by design, and distributions are pooled across three interventions using newly drawn personas from the same sampling rule.

C.1 Demographic composition

  • Table C1 reports the demographic distribution of each study.Source: Table C1 · p. 45
  • The samples are about 50% female by design.Source: §C.1 · p. 45
  • Distributions pool responses across three interventions, each drawing new personas from the same sampling rule.Source: §C.1 · p. 45

C.2 Demographic effects and treatment-effect heterogeneity

This appendix reports demographic coefficients for the six studies and describes the natural-language topic pipeline and evidence prompts used in the analyses. The demographic models estimate age and gender associations and their interactions with study treatments.

C.2 Demographic effects and treatment-effect heterogeneity

  • Tables C2–C7 report combined-model zero-shot coefficients for age, treatment-by-age, gender, and treatment-by-gender; Studies 2 and 6 include participant random intercepts.The coefficients represent age main effects, treatment × age, gender main effects, and treatment × gender.Source: §C.2 · pp. 46, 48, Table C2 · p. 46, Table C3 · p. 46

D.1 Natural language pipeline

  • The natural-language pipeline embeds rationales by contextual meaning, clusters them with BERTopic, then uses keywords and representative documents to support GPT-assisted topic labels.The clustering uses UMAP and HDBSCAN; the minimum topic size is 5 and the neighbor count is 15, with hyperparameter robustness checks.Source: §D.1 · p. 49

D.2 Example of prompt for GPT-assisted topics

  • The example GPT-assisted topic prompt asks for one concise topic name and description from representative evaluation comments, with a fallback when no good label is found.The prompt requires a dictionary containing a topic name and description.Source: §D.2 · pp. 49–50
  • HDBSCAN identifies dense document regions as topics and treats sparse points as outliers, while c-TF-IDF highlights words frequent within a topic but less common across topics.Source: §D.2 · p. 49

Prompt with Conceptual Evidence

E.2 Detailed results by study after prompt interventions

The appendix reports study-level outcomes under no intervention and conceptual or empirical evidence prompts across four model-format configurations. It also describes the case-study deployment and the JSON tagging scheme for empirical evidence.

E.2 Detailed results by study after prompt interventions

  • The study tables report intervention-specific main-effect directions and p-values, mediation verdicts where hypothesized, and effect sizes across four configurations; the case-study table covers GPT-5.4-mini TXT only.The appendix conventions mark directions as consistent or inconsistent with the human finding and identify significant p-values in bold.Source: §E.2 · p. 51, Table E1 · p. 52, Table E2 · p. 52
  • In Study 1, GPT-4o-mini configurations show significant human-consistent effects under all three intervention conditions, while GPT-5.4-mini effects are significant only with empirical evidence.The reported GPT-4o-mini directions are positive with p < .001; GPT-5.4-mini TXT and JSON show significant positive directions under empirical evidence (p = .003 and p = .013).Source: §E.2 · p. 52
  • Study 2 reports selected color saturation by temporal distance, while Study 3 reports comfort and durability separately for circular versus angular logos.Source: §E.2 · pp. 52–53
  • The cited tables identify Study 4 as logo asymmetry and brand excitement (asymmetrical versus symmetrical), Study 5 as color saturation and estimated product size (high versus low saturation), and Study 6 as logo stability and utility.Study 6 rows report the logo-stability simple effect within the safety-oriented product category, comparing unstable and stable logo conditions.Source: Table E4 · p. 53, §E.2 · pp. 53–54, Table E5 · p. 54
  • The case study tests two AI-generated running-shoe logos with GPT-5.4-mini TXT, using 160 observations per intervention and no human baseline for those stimuli.Its direction marker is benchmarked against Study 1’s human finding that more descriptive logos are more authentic.Source: Table E7 · p. 55, §E.2 · p. 55
  • The empirical-evidence JSON schema separates method and results, preserves each source sentence verbatim, and uses references when one sentence serves multiple tags.Source: Table E8 · p. 56, §E.2 · p. 56

Web Appendix F: Robustness of findings

The appendix checks robustness of the Study 1 replication across temperature settings and stimulus resolutions, and describes a test for rating carryover across consecutive API calls.

Web Appendix F: Robustness of findings

  • Across temperatures 0.5, 1.0, and 1.5, the Study 1 authenticity effect remains positive and significant for GPT-4o-mini TXT.The reported Dir(p) values are + (p = .002), + (p < .001), and + (p = .001), respectively.Source: Web Appendix F: Robustness of findings · p. 57
  • At 256, 512, and 1024 pixels, both the descriptiveness manipulation check and the authenticity main effect are positive and significant.The table reports positive directions with p < .001 at each resolution for both measures.Source: Web Appendix F: Robustness of findings · p. 57
  • The carryover analysis regresses ratings on condition, the preceding participant’s rating, and replicate fixed effects, with standard errors clustered by replicate.The appendix defines a carryover coefficient indistinguishable from zero as indicating no carryover between consecutive API calls.Source: Web Appendix F: Robustness of findings · p. 58, Table F4 · p. 58

Related papers

Loading 2609.25677v1…