PaperlayerView exact revision on arXiv

Source-linked AI summary

Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models

Sumin Hong, Katsumi Ibaraki, Renee Shi, David Chiang, Toby Jia-Jun Li

arXiv:2609.36475v1cs.CLcs.AIcs.CV

TL;DR

Prior human–VLM comparisons often used different stimuli or tasks, leaving their alignment under matched conditions unclear. This study compares choices and gaze on shared stimuli; some larger models align with human choices, but their attention remains less gaze-aligned than a center-bias baseline.

  • Problem

    Prior studies often compared humans and VLMs using different stimuli or tasks, leaving direct matched evidence on choice and attention alignment limited.

    Source: §1 · p. 2

  • Method

    The authors compare human and VLM choices and eye-tracking on identical pseudo-word–image stimuli and test fine-tuning small models with human choices and gaze.

    Source: §1 · p. 2, §5.1 · p. 7

  • Results

    Some larger VLMs align with human pseudo-word choices, but their saliency trails center bias; choice fine-tuning transfers to unseen stimuli without closing the spatial gap.

    Source: §1 · p. 2, §7 · pp. 9–10

  • Takeaways & Limitations

    Choice agreement, or even closer gaze alignment, is insufficient evidence of human-aligned cross-modal processing, so both choices and visual attention must be evaluated.

    Source: §1 · p. 2, §7 · p. 10

    Because +Gaze+Side partly re-supplies the choice label through gaze, a choice gain in that condition would not isolate gaze information beyond the label.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

Cross-modal associations are systematic pairings of features across modalities, such as the association of 'bouba' with round shapes and 'kiki' with sharp shapes. Prior work has compared humans and vision-language models (VLMs) on such associations, but often using different stimuli or tasks between humans and models. Here, we ask whether VLMs align with humans not only in choices, but also in where they look when making those choices. We study both VLMs and humans (N = 53), presenting them with the same stimuli, a pseudo-word and two images, and record participants' choices and eye movements, which we release. We find choice alignment in a few larger VLMs, but their saliency matches human gaze less closely than a center-bias baseline, a fixed Gaussian at the center of each image. Fine-tuning small VLMs on human choices brings their choice alignment to the level of a human majority-vote reference on unseen words and images, yet their attention still matches human gaze less closely than this baseline. Training model attention on human gaze raises attention-gaze correlation without improving choice alignment, and a single average gaze map per image position raises it by a similar amount. Matching human choices, or even human gaze patterns, is therefore not sufficient evidence of human-aligned cross-modal processing.

1 Introduction

The bouba–kiki effect offers a test of whether vision-language models form human-like sound–shape associations. This study compares choices and gaze on matched stimuli, finding that choice agreement can coexist with different visual attention.

1 Introduction

  • Humans commonly pair bouba with rounded shapes and kiki with spiky shapes, a sound–shape correspondence replicated across languages and observed in infants and pre-literate children.The authors use this established association as a testbed for human cross-modal mapping.Source: §1 · p. 1
  • Choice agreement alone cannot distinguish attention to the relevant shape from attention to other stimulus properties that covary with roundness.The authors therefore compare model saliency with human gaze on the same trials.Source: §1 · pp. 1–2
  • Prior human–VLM comparisons often used different stimuli, tasks, or analysis levels, leaving matched choice and attention alignment unresolved.The paper also notes conflicting findings about whether VLMs show cross-modal associations.Source: §1 · p. 2
  • The study compares humans and VLMs on identical stimuli using choices and eye-tracking from 53 participants, and releases the data.The stimuli pair a pseudo-word with two images.Source: §1 · p. 2, Figure 1 · p. 2
  • Some larger VLMs align with human choices, but their saliency matches human gaze less closely than a fixed center-bias baseline.Fine-tuning small VLMs on human choices reaches human-level choice agreement on unseen words and images, while gaze training raises attention–gaze correlation without improving choice agreement.Source: §1 · p. 2

2 Related work

Human sound symbolism is a well-established cross-modal association, while VLM findings have been mixed. Prior studies have often lacked a direct human–model comparison under matched stimuli and tasks.

2.1 Cross-modal associations in humans

  • The bouba–kiki effect pairs bouba with round shapes and kiki with sharp ones, alongside reported cross-modal associations involving magnitude and texture.The sound–shape association has been documented across studies of phonological properties and visual shape.Source: §2.1 · p. 2

2.2 Cross-modal associations in VLMs

  • VLM studies report mixed results, and typically compare humans and models on different stimuli or tasks or only at the output level.Prior work includes conflicting CLIP findings and studies extending to multimodal LLMs and audio–visual models.Source: §2.2 · p. 3

3 Dataset and human study

The study pairs pseudo-words with round–sharp images and records choices and gaze under a shared human–VLM task. Human responses establish reliable sound–shape associations, and the held-out split tests generalization to unseen words and image pairs.

3 Dataset and human study

  • The dataset combines canonical labels, English adjectives, and constructed pseudo-words with round–sharp image pairs, and records human choices and gaze for comparison and supervision.Labels and images have intended categories based on prior human studies rather than objective ground truth.Source: §3 · p. 3

3.1 Stimuli

  • The 162 constructed pseudo-words combine consonants and vowels associated with the same intended shape category, and are presented in English orthography.Participants read labels aloud, whereas models receive only written forms; the study does not separate orthographic from phonological contributions.Source: §3.1 · p. 3
  • The 184 image pairs include prior examples, line-based minimal pairs, and Gemini-generated abstract pairs, with orientations and assignments randomized.The randomized assignments prevent labels from being tied to a particular image pair or side, except for canonical labels.Source: §3.1 · p. 3

3.2 Human study procedure

  • After cleaning, 53 participants contributed 8,208 trials; constructed pseudo-words elicited 73.98% intended-category agreement, above chance.Gaze was recorded with an eye tracker, and cleaned trials required valid choices and sufficient in-image fixations.Source: §3.2 · p. 4
  • On 80.58% of constructed pseudo-word trials, the image receiving more gaze samples was the one chosen; the authors describe this as an association, not causation.The first fixation agreed with the choice at chance.Source: §3.2 · p. 4

3.3 Splits and behavioral references

  • Fine-tuning and evaluation use constructed pseudo-words split by word and image pair, so held-out test stimuli are unseen during training.Participants appear across splits, but participant identity is not a model input; the evaluation tests generalization to new stimuli, not new people.Source: §3.3 · p. 4
  • On Test, intended-category and leave-one-out majority-choice references agree with participants on 72.82% and 72.86% of trials, respectively.These are reference points, not ceilings; the best attainable agreement cannot be estimated because most word–pair–orientation conditions were seen by only one participant.Source: §3.3 · p. 4

4 Cross-modal associations in pretrained VLMs

The evaluation compares VLM choices and saliency with human judgments and gaze on matched trials. Some larger models show pseudo-word choice alignment, but every evaluated model's saliency falls below its center-bias reference.

4.1 Evaluation protocol

  • Embedding models choose by image–text similarity, while generative models answer the forced-choice prompt; spatial alignment compares each model map with same-trial gaze on each image.The spatial measure is computed within each image, so it does not measure how attention is split between the two images.Source: §4.1 · p. 5

4.2 Zero-shot results

  • Gemma4-26B achieved the highest pseudo-word ICA (0.62), followed by Qwen3-VL-30B (0.57); SigLIP-so400m alone showed a clear embedding-model signal (ICA = 0.57).Most generative models were near ceiling on adjectives, while pseudo-word alignment varied with model size.Source: §4.2 · p. 5
  • Neutral wording preserved positive correlations with human pseudo-word round-choice rates for Gemma4-26B and Qwen3-VL-30B, although individual decisions changed.The paper reports that removing sound-symbolic framing did not remove the graded human-like pattern.Source: §4.2 · p. 5
  • Every evaluated model's spatial ρ fell below its center-bias baseline; CLIP ViT-B/32 reached ρ = .588 versus center .743.The authors note that differing saliency methods across model families prevent separating model-level from readout-level differences.Source: Table 2 · p. 6, §4.2 · p. 6

5 Fine-tuning on human choices and gaze

This section tests whether choice alignment learned from participants generalizes to unseen words and image pairs, and whether adding gaze supervision brings model attention closer to human gaze. Choice alignment approached the human majority-vote reference, but gaze targets did not yield an additional choice gain.

5.1 Training conditions

  • Choice-only uses participant choices; +Gaze adds trial-specific gaze; +AvgGaze uses a fixed position-specific Train mean; +Gaze+Side is a robustness condition with coupled design changes.The +Gaze+Side condition changes normalization and image weighting together and lacks a matched constant-target control.Source: §5.1 · p. 7

5.2 Training objective

  • The per-trial loss adds λ-weighted gaze KL to answer-token cross-entropy; separately normalized images leave relative A/B attention mass unsupervised.Choice-only uses λ=0; gaze-supervised conditions use λ=1, with no λ sweep.Source: §5.2 · p. 7

5.3 Alignment measures

  • Test measures cover choice agreement and NLL, within-image gaze ρ and ΔKL, and between-image JS and side-agreement gaps, with paired clustered intervals and Holm correction.Positive ΔKL means attention is closer to human gaze than the fixed training-mean map.Source: §5.3 · p. 7

5.4 Results

  • Training on 1,932 observed human choices raised human-choice agreement from 52.91/48.54/50.00% under Zero-shot to 72.72/72.91/71.94% under Choice-only, similar to the 72.86% human majority-vote reference.Source: §5.4 · p. 8
  • Adding trial-specific gaze changed human-choice agreement by +0.10/−0.10/+1.17 pp, with intervals containing zero; +Gaze+Side likewise showed no detectable choice gain.All nine NLL contrasts among Choice-only, +Gaze, and +AvgGaze had intervals containing zero after Holm correction.Source: Table 3 · p. 8, §5.4 · p. 8
  • Choice-only gaze correlations (.308/.065/.062) stayed below the center-bias baseline (.637/.637/.603), while +AvgGaze nearly matched +Gaze correlations.The +Gaze−+AvgGaze ΔKL difference was an estimated reduction of .074–.081 bits, meeting the adjusted threshold for Gemma4-E4B and Qwen3-VL-8B but not Molmo2-8B.Source: §5.4 · p. 9
  • Choice–attention side-agreement gaps were +43.12/+9.59/+0.51 pp despite similar choice scores, indicating different associations between attended side and answer.A positive gap indicates association, not that attention causes choice.Source: Table 3 · p. 8, §5.4 · p. 9

6 Discussion

Pseudo-word choice alignment appears mainly in some larger VLMs, but their visual attention remains less aligned with human gaze; fine-tuning or gaze supervision does not close both gaps. The authors qualify these findings by the limited fine-tuning setup and by correlational, layer-dependent attention measures.

6 Discussion

  • Fine-tuning raises human-choice agreement by about 20 percentage points, yet gaze correlation remains below the center-bias baseline.Adding gaze as an attention target raises attention–gaze correlation to .67–.73 without detectable change in choice agreement or NLL; an average gaze map gives a similar gain.Source: §6 · p. 9
  • Pseudo-word alignment emerges mainly in a subset of larger VLMs, while adjective matching is widespread; Molmo2-8B scores 0.94 adjective ICA and 0.53 pseudo-word ICA.Gemma4-26B and Qwen3-VL-30B show partial pseudo-word success, which the authors suggest may reflect training-data regularities linking written form, phonology, and visual features.Source: §6 · p. 9
  • Gemma4-26B’s pseudo-word choices correlate with human judgments at r≈.58, while its attention correlates with gaze at .27.The authors suggest holistic image representations may support choices even when local shape-informative features receiving human fixations are not attended.Source: §6 · p. 9
  • Fine-tuning was tested with three models, one optimization seed, one gaze-loss weight, and the original prompt; attention and Grad-CAM are layer-dependent correlational readouts, not causal explanations.Written labels confound phonology and orthography; participants were English-speaking members of one institution and were shared across splits.Source: §6 · p. 9

7 Conclusion

The conclusion reports a matched human–VLM comparison of sound–shape choices and spatial attention. Choice alignment can improve without spatial alignment, so choice agreement alone does not establish human-aligned cross-modal processing.

7 Conclusion

  • Fine-tuning closes the choice gap on unseen words and images without closing the spatial gap.The conclusion reports that pseudo-word alignment appears mainly in some generative VLMs, whose saliency still diverges from human gaze.Source: §7 · p. 9
  • The study compares human and VLM sound–shape choices and spatial alignment using the same stimuli.The conclusion contrasts widespread matching of English adjectives with pseudo-word alignment mainly in a subset of generative VLMs.Source: §7 · p. 9
  • Choice agreement alone is insufficient evidence of human-aligned cross-modal processing; both choices and visual attention must be evaluated.Source: §7 · p. 10

AI use statement

The authors disclose using generative AI to generate synthetic datasets and polish writing. They state that it was not used for the listed theoretical, methodological, analytical, or implementation tasks.

AI use statement

  • Generative AI tools were used to generate synthetic datasets and polish writing for readability.Source: AI use statement · p. 10
  • The authors state that generative AI was not used to develop theoretical models, conceptual frameworks, or mathematical claims and proofs.Source: AI use statement · p. 10
  • The authors also report no AI use for methodology or experiments, implementation, translation, data cleaning, qualitative analysis, or result interpretation.They take responsibility for the final content, including AI-assisted text, claims, or artifacts.Source: AI use statement · p. 10

Ethics statement

The work reports adherence to the ICLR Code of Ethics and institutional IRB approval. Participants consented to release of de-identified choice and gaze data, while audio and video recordings will not be released.

Ethics statement

  • The work states that it adheres to the ICLR Code of Ethics and received institutional IRB approval.Source: Ethics statement · p. 10
  • Participants gave informed consent, including consent to release de-identified choice and gaze data.Source: Ethics statement · p. 10
  • Audio and video recordings were used only to verify responses and will not be released; released data contain no names, audio, video, or other personal identifying information.Source: Ethics statement · p. 10

Reproducibility statement

This appendix documents the stimuli and human-study procedures, reports how gaze relates to choices, and describes the released data and statistical analyses.

Reproducibility statement

A Stimuli

  • AH is included among round-associated vowels following prior human studies, although it is not rounded in every English dialect.Source: A Stimuli · p. 13

E.2 Methodology

  • Participants read each word aloud, then verbally chose A or B for the following image pair; word order was randomized within blocks.Source: §E.2 · p. 15, §E.3 · p. 16

E.3 Stimuli

  • The study paired 166 pseudo-words with image pairs, added 20 adjectives, and presented each image pair in both left–right orientations.The pseudo-word set comprised four canonical and 162 constructed words; image pairs included 75 line-drawing, 92 generated, and 17 prior-work pairs.Source: §E.3 · p. 16

E.4 Human Gaze–Choice Coupling

  • Accumulated gaze matched participants’ choices on 77–80% of trials, while the first in-image fixation agreed at chance level.The authors note that this association does not establish causal direction.Source: §E.4 · p. 17

E.5 Participants and cleaning

  • The cleaned dataset contains 8,208 trials from 53 participants after requiring a valid choice, complete image presentation, and sufficient gaze coverage and fixations.It includes 7,141 constructed pseudo-word, 910 adjective, and 157 canonical trials.Source: §E.5 · p. 17

G Full Statistics for Table 1

  • The appendix reports participant-cluster bootstrap intervals conditional on the observed stimuli and checkpoint, which do not measure generalization to new labels or training-seed variation.Source: G Full Statistics for Table 1 · p. 19

G.1 Neutral-prompt robustness and paired changes

The authors compare original and neutral prompt formulations for two large VLMs. Positive pseudo-word choice alignment persists under the neutral formulation, although individual decisions often change and the comparison does not isolate a single prompt phrase.

G.1 Neutral-prompt robustness and paired changes

  • The neutral prompt removes label type and sound-symbolic framing while retaining the participant-style choice instruction and JSON requirement.Source: Figure 10 · p. 20, §G.1 · p. 20
  • This compares two full prompt formulations, not an isolated phrase, and the neutral runs provide choices but no spatial-alignment comparison.Source: §G.1 · p. 20
  • Neutral pseudo-word Pearson correlations are .534 for Gemma4-26B and .535 for Qwen3-VL-30B; the paired correlation-change intervals include zero, but do not establish equivalence.Source: §G.1 · pp. 20–21
  • Neutral adjective ICA is .945 for both models, compared with original-prompt values of .970 for Gemma4-26B and .937 for Qwen3-VL-30B.Source: §G.1 · p. 20
  • Paired contrasts use 10,000 participant bootstrap resamples and nominal 95% percentile intervals conditional on the observed labels, images, and checkpoints, without multiple-comparison adjustment.Source: §G.1 · pp. 20–21
  • Gemma4-26B’s pseudo-word ICA falls by 3.99 pp (95% CI [−5.29, −2.74]); its actual-response agreement changes from 57.30% to 57.74% (+0.43 pp, [−0.96, 1.84]).Source: §G.1 · p. 21
  • Qwen3-VL-30B’s actual-response agreement changes from 55.73% to 57.72% (+1.99 pp, [0.76, 3.22]), while its ICA point estimate is unchanged.Source: §G.1 · p. 21
  • Original and neutral prompts yield different choices on 33.23% of Gemma4-26B’s and 20.55% of Qwen3-VL-30B’s unique pseudo-word inputs.Unchanged aggregate Qwen3-VL-30B ICA therefore does not imply unchanged individual decisions.Source: §G.1 · p. 21

H.2 Trial-wise native-grid spatial evaluation

Trial-wise spatial evaluation compares model attention with participant gaze on native image-token regions, alongside center and gaze-based references. Across the reported analyses, matching gaze patterns does not by itself establish human-aligned choices or cross-modal processing.

H.2 Trial-wise native-grid spatial evaluation

  • Maps are compared with gaze on the same trial using native image-token regions; inputs are resized to 512 × 512, and boundary-crossing tokens are excluded.Generative readouts average decision-token attention over selected full-attention layers; embedding models use model-specific Grad-CAM or attention-gradient maps.Source: §H.2 · pp. 22–23
  • .340/.637 (Gemma4-E4B), .265/.637 (Gemma4-26B), .022/.637 (Qwen3-VL-8B), −.003/.637 (Qwen3-VL-30B), −.074/.601 (Molmo2-4B), and .004/.601 (Molmo2-8B) were model/Center means, with every model below Center.Each mean uses 7,141 pseudo-word trials; model and Center denominators both include 14,070 valid image-level correlations.Source: §H.2 · p. 25

I Probability-based Analysis

  • Probability-based top-choice analysis differs from human forced choice and is reported only as ICA, with label-side bias potentially distorting round–sharp accuracy.A tendency toward sharp labels can inflate sharp-label accuracy while reducing round-label accuracy, even when overall accuracy is near chance.Source: I Probability-based Analysis · p. 26

J Choice agreement and majority-vote reference

  • On Test, the human majority-vote reference predicts each participant’s response from other participants’ responses to the same word; ties receive expected agreement .5, across the same 1,030 test trials.Source: J Choice agreement and majority-vote… · p. 26

K.1 Training details

  • Fine-tuning uses LoRA adapters with frozen vision components, and selects checkpoints by validation agreement with human choices rather than gaze fit or test performance.The gaze loss uses participant-specific or average gaze targets; when gaze and choice coincide, +Gaze+Side can re-supply the choice label through the gaze channel.Source: §K.1 · pp. 27, 29

K.2 Reference maps and uncertainty

  • Reference maps compare attention with a central spatial prior and a population-mean gaze pattern; bootstrap intervals are conditional on selected checkpoints and omit retraining uncertainty.The reference maps are empirical comparisons, not performance ceilings.Source: §K.2 · p. 29

K.3 Trial-specific versus average-gaze supervision

  • .733/.729/.673 under +Gaze and .733/.731/.679 under +AvgGaze are close to the population-mean reference, while similar correlations do not establish statistical equivalence.The fixed average target reproduces high correlation without trial-specific gaze targets; metric comparisons are not uniform across models.Source: §K.3 · pp. 29–30

K.4 Allocation between images

  • +Gaze+Side reduces between-image JS versus Choice-only in all three models, while choice–attention side agreement can reflect shared side preferences rather than trial-specific alignment.For Gemma4-E4B, +Gaze and +AvgGaze have similar within-image correlation (.733) but different choice–attention side agreement (92.14% versus 53.20%).Source: §K.4 · p. 30

L Position-swap evaluation

The position-swap evaluation tests whether model choices persist when image panels exchange positions, while distinguishing that consistency from evidence of a word-dependent association. Choice-only conditions show high swap consistency and position-balanced ICA, but these measures do not rule out every shortcut.

L Position-swap evaluation

  • Each trial exchanges image panels without mirroring them, keeps the word and A/B labels fixed, and matches predictions across layouts under a shared input, prompt, and decoding protocol.The same response-acceptance rule is applied across all 17 conditions and both layouts.Source: L Position-swap evaluation · p. 31
  • Swap consistency measures selecting the same shape across layouts, while position-balanced ICA averages intended-category agreement across layouts with the intended side reversed after exchange.An always-A or always-B strategy scores 0% swap consistency; independent uniform random answers have an expected 50%.Source: L Position-swap evaluation · p. 31, Table 26 · p. 32
  • High swap consistency does not establish a word-dependent association, and averaging layouts does not eliminate every shortcut; larger zero-shot references do not isolate parameter-count effects.Intervals are individual, unadjusted 95% percentile intervals conditional on fixed checkpoints and exclude training-seed and checkpoint-selection uncertainty.Source: L Position-swap evaluation · p. 32
  • Position-balanced human-reference agreement uses each participant’s original shape choice in both layouts, not a new human response to the exchanged stimulus; human-gaze alignment is not evaluated there.Table 27 also reports model B-choice rates before and after exchanging images.Source: L Position-swap evaluation · p. 32, Table 27 · p. 33
  • Choice-only swap consistency reaches 90.10/96.89/88.45% (vs. 30.97/8.25/19.51% under Zero-shot), while position-balanced ICA reaches 92.72/94.56/91.50%.The two larger zero-shot models reach swap consistency of 57.38/54.56% and position-balanced ICA of 61.99/60.10%, respectively.Source: L Position-swap evaluation · p. 32, Table 26 · p. 32
Loading 2609.36475v1…