PaperlayerView exact revision on arXiv

Source-linked AI summary

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs

Alexa R. Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer, Martin Wattenberg, Been Kim, Yuxuan Li

arXiv:2610.36449v1cs.AIcs.CLcs.CV

TL;DR

This paper asks whether VLMs communicate visually, rather than merely reason about visual content. Across studies of symbol-driven behavior and model-to-model visual communication, it finds implicit steering, creative visual communication, and gaps between visual production, interpretation, and human conventions.

  • Problem

    Whether VLMs share humans’ ability to communicate through visual content remains unclear, despite their extensive exposure to visual culture and strong visual reasoning.

    Source: §1 · p. 1

  • Method

    The paper tests symbol-driven behavior and self-reports, model generation and interpretation of visualizations for concepts and stories, and alignment with human interpretations.

    Source: §1 · p. 1

  • Results

    Simple symbols steer unrelated behavior without faithful verbalization, while models creatively communicate concepts and stories but show generation–interpretation gaps and only partial alignment with human interpretations.

    Source: §1 · pp. 1–2, §4.2 · p. 9

  • Takeaways & Limitations

    Visual communication is a rich, underexplored dimension of VLM behavior, with visual meaning that can diverge from human conventions.

    Source: §6 · p. 10

    The tasks are synthetic, real-world manifestations remain open, and the human study is exploratory and small-scale.

    Source: §6 · p. 11

Abstract

from arXiv · show

Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.

1. Introduction

The paper asks whether VLMs can communicate visually, distinguishing visual reasoning from communicating through visual content. Across three investigations, it finds that visual symbols can steer unrelated behavior without faithful self-reports, and that generated visualizations can convey meaning while remaining misaligned with model and human interpretations.

1. Introduction

  • Simple visual symbols can strongly steer unrelated behavior, but their associations are model-specific and are not faithfully verbalized.Models also generate creative visualizations for abstract concepts and stories, but do not always interpret them as intended.Source: §1 · p. 1
  • Models’ visual meaning spaces are only partially aligned with human interpretations; effective or emergent visualizations are often less legible to people.The paper frames these findings as evidence of a rich, underexplored dimension of VLM behavior.Source: §1 · p. 2

2. Related Work

Prior work documents both VLMs’ growing visual capabilities and persistent gaps in perception, visual reasoning, and human-like visual meaning. This paper connects those areas to study visual communication between models, including how simple symbols affect behavior and how models explain visual cues.

Visual understanding and image generation

  • VLMs show strong benchmark performance and image-generation ability, yet still fail at some simple perceptual tasks, differ from human color perception, and can favor prior knowledge over visual evidence.Related work describes frontier VLMs as increasingly matching or surpassing human baselines on complex scene understanding (Yue et al., 2024).Source: §2 · p. 2

Visual reasoning

Visual meaning and communication

VLM robustness to adversarial and distractor images; Faithfulness and self-knowledge

3. Visual steering

The section tests whether simple visual symbols systematically steer behavior, whether that effect depends on image features, whether it generalizes across tasks, and whether models recognize their own visual influences.

3.1. Simple visual symbols steer models in surprising ways

  • A red heart raised recursion by Δ = +1.0 for Claude Opus 4.7 and GPT-5.4, while a yellow checkmark yielded 90% recursion (Δ = +0.5) for Gemini 3.1 Pro.Baseline recursion was 0 for Claude Opus 4.7 and GPT-5.4, and 0.4 for Gemini 3.1 Pro; steering could match or approach direct recursion instructions. Across 288 symbols, steering profiles differed by model, and no single shape or color steered uniformly.Source: §3.1 · p. 4

3.2. Is visual steering pixel-level or concept-level?

  • Captions weakly tracked image steering, while noise and naturalistic images produced little systematic steering; disrupting shape structure or upright orientation diminished the effect.Image–caption steering correlations were −0.05 for Claude Opus 4.7, 0.31 for Gemini 3.1 Pro, and 0.17 for GPT-5.4. Controls were near zero for Claude Opus 4.7 and GPT-5.4, and approximately −0.1 for Gemini 3.1 Pro on noise and −0.05 on ImageNet.Source: §3.2 · pp. 4–5

3.3. Does visual steering generalize to other tasks?

  • Visual steering extended across five additional recursion-versus-iteration problems, other algorithmic and data-structure choices, writing styles, and binary preferences.Associations included hearts steering some models toward recursion or verbose prose, and pink shapes and hearts steering all three models toward poetic rather than professional writing.Source: §3.3 · pp. 5–6

3.4. Are models aware of how visual symbols steer them?

  • With silent image injections, maximum recursion steering reached Δ = +1.00 for Claude Opus 4.7, +0.60 for Gemini 3.1 Pro, and +0.50 for GPT-5.4, often without image mentions in traces.Silent-injection steering symbols often differed from those effective under explicit instruction (r ≤ 0.18). For Gemini 3.1 Pro, 71.5% of traces dismissed a steering symbol as irrelevant; across nine additional coding tasks, 64.1% did so. GPT-5.4 often attributed influence when symbols did not steer.Source: §3.4 · pp. 6–7
  • Direct symbol interpretations only weakly correlated with empirical steering, with correlations of 0.22 for Claude Opus 4.7, 0.29 for Gemini 3.1 Pro, and 0.31 for GPT-5.4.The analysis compared models’ direct recursion-versus-iteration interpretations with their observed effects on binary-tree traversal.Source: §3.4 · pp. 6–7

4. Model-to-model visual communication

The sender–receiver games test whether models can communicate coding strategies, concepts, and stories through visual symbols or free-form drawings. Models can converge on useful visualizations, but the symbols they choose may not match those that most strongly steer behavior.

4. Model-to-model visual communication

  • The sender–receiver setup lets a sender refine a visualization based on receiver behavior, and tests both a symbol library and free-form PIL drawings.For coding-strategy steering, senders see the receiver’s baseline and post-visualization strategy distributions and may refine up to 20 turns.Source: §4 · p. 7, §4.1 · p. 7

4.1. What visualizations do models construct to steer coding behavior?

  • Senders’ recursion symbols often differed from the strongest steering symbols, although feedback found effective symbols in an average of 1.3–14.0 turns across models.Claude Opus 4.7 tended to send intuitive spirals, infinity signs, and arrows, while hearts and triangles were among its strongest steering symbols; free-form drawing required fewer turns than symbols.Source: §4.1 · pp. 7–8, Figure 2 · p. 8

4.2. What visualizations communicate abstract concepts and stories?

  • 97.8% of free-form drawing experiments eventually conveyed the correct concept or story, versus 47.8% zero-shot success with visual primitives.Free-form drawings succeeded zero-shot in 91.2% of experiments; 30.7% of primitive-symbol experiments exhausted the five-turn limit without success.Source: §4.2 · p. 9
  • Successful visualizations are creative and varied, though some intended meanings remain opaque to people.The concept and story games ask a receiver to select the intended target from four options; senders may refine visualizations for up to five turns.Source: §4.2 · pp. 8–9

5. Alignment with human interpretation

Human judgments often diverge from models’ visual steering associations. Model-refined symbols become less legible to people, while free-form drawings remain highly interpretable.

5. Alignment with human interpretation

  • Only about 36% of human judgments matched top symbols’ empirical steering direction, below the 50% chance baseline.Humans associated hearts with iteration more often than recursion; sender-chosen symbols were more legible, at about 67% accuracy.Source: §5 · pp. 9–10
  • Human accuracy on symbol visualizations fell from 66.7% to 44.4% for recursion versus iteration, 56.7% to 40.0% for concepts, and 77.8% to 55.6% for stories.These figures compare first-turn visualizations with refined final-turn visualizations.Source: §5 · p. 10
  • Free-form drawings remained highly legible across turns, with about 79–91% accuracy on concepts and 100% on short stories.The human study included seven participants.Source: §5 · pp. 9–10

6. Discussion and Conclusion

The discussion finds that VLMs can be steered by visual symbols and communicate through visualizations, but their visual meanings are not reliably reported, interpreted as intended, or aligned with human interpretation. The authors identify unresolved mechanisms and scope limits, and caution that visual-input influence and output legibility should not yet be assumed in deployed agents.

6. Discussion and Conclusion

  • Visual steering and visual communication are dissociated from faithful reporting and interpretation, while some model associations diverge from human interpretation.The authors argue that visual-input influence should be transparent and outputs legible to users, neither of which can yet be assumed.Source: §6 · p. 10
  • The mechanism behind visual steering remains an open question; semantic-space proximity is one proposed explanation, and image-to-text objectives may leave associative pathways poorly aligned with generation.The authors suggest measuring representational distances and studying internet co-occurrence, while noting that closing the gap may require visually grounded communicative training.Source: §6 · p. 10
  • Deliberate visual collusion appears unlikely for now because associations are weakly shared across models and senders struggle to access their own, but unintentional steering may affect screenshot and GUI agents.Trace-based oversight may miss visual influence because it is often unverbalized; naturalistic agent workflows remain a future-work target.Source: §6 · pp. 10–11
  • Less capable models in each family tend to be more sensitive to Gaussian-noise images or naturalistic photos, while frontier models largely ignore those inputs.The authors hypothesize that capability may shift steering from indiscriminate sensitivity toward more structured, symbol-specific influence.Source: §6 · p. 11
  • The study uses synthetic tasks, leaving real-world manifestations open, and its small exploratory human study cannot establish robust human–model alignment.The authors call for larger participant pools and more systematic investigation.Source: §6 · p. 11

A.1. Model evaluation details

The appendix documents model evaluation, coding-task prompts, and a 288-symbol library, then presents full steering heatmaps across tasks and baselines.

A.1. Model evaluation details

  • Models are evaluated through provider APIs without system prompts, generally using high reasoning effort and common-practice settings.The appendix lists model-specific API parameters and includes Claude Sonnet 4.5, Gemma 4 31B-IT, and GPT-4o in some experiments.Source: §A.1 · p. 16

A.2. Gemini 3.5 Flash judge validation

  • Gemini 3.5 Flash’s coding-strategy classifications matched two other model judges on 180 samples and author labels on 20 samples, with 100% agreement in both checks.The authors labeled the latter sample without access to the automated judge’s decisions.Source: §A.2 · p. 16

B.1. Task and prompt details

  • In visual-steering experiments, prompts began with a 400×400-pixel visual-symbol image, followed by a task instruction and brief problem statement, and explicitly directed models to look at the image.5The symbol dataset contains 288 symbols spanning 12 colors and 24 shapes, rendered on a plain white background.Source: §B.1 · p. 17
  • Candidate strategy order is balanced across 20 generations, and its average steering differences are small across the three models.Arrow shapes have the largest order-related steering-value differences and may sometimes be read as choosing the left or right option.Source: §B.1 · p. 17

B.2.1. Full steering heatmaps

  • The full heatmaps cover all 288 symbols for inorder traversal and aggregate additional baseline-iterative and baseline-recursive coding problems.The aggregate recursion heatmaps cover 4, 3, and 4 problems for Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4; iteration heatmaps cover 2, 3, and 2, respectively.Source: §B.2.1 · p. 18, Figure 5 · p. 19, Figure 6 · p. 20

B.2.2. Additional steering baselines

The appendix compares image steering with text instructions, captions, and generic-image controls, finding that steering is not reproduced by captions or arbitrary images.

B.2.2. Additional steering baselines

  • Direct text instructions steer all models toward the requested strategy 100% of the time, setting a ceiling for other steering effects.Text controls repeat strategy words to approximately match image-token counts, and silent text injection mirrors the silent image condition.Source: §B.2.2 · p. 22
  • Image and matching-caption steering have low correlations: r = −0.05 for Claude Opus 4.7, r = 0.31 for Gemini 3.1 Pro, and r = 0.17 for GPT-5.4.Comparisons use paired symbols from the 288-symbol library, with 190 Gemini pairs after excluding API errors.Source: §B.2.2 · pp. 22–23
  • Gemini 3.1 Pro has the highest image-caption correlation, which the authors suggest may reflect somewhat more visual thinking from text descriptions, while noting further work is needed.The figure presents top steering captions and compares caption effects with image effects.Source: §B.2.2 · p. 23, Figure 8 · p. 23
  • Five synthetic Gaussian-noise images and 20 ImageNet animal images serve as controls for generic visual-token steering.The noise images match symbol-image dimensions; the ImageNet images span animal classes and varied visual compositions.Source: §B.2.2 · p. 23, Figure 9 · p. 23, Figure 10 · p. 24

B.2.3. Visual perturbation experiments

The appendix tests how image structure, orientation, position, size, and prompt placement affect steering, and describes additional tasks used to probe generalization.

B.2.3. Visual perturbation experiments

  • Disrupting shape geometry through pixel shuffling sharply diminishes recursion steering for Claude Opus 4.7 and Gemini 3.1 Pro, while Gemini appears more sensitive to color and low-level noise.The experiment progressively shuffles pixels while preserving colored-pixel proportions across five symbols.Source: §B.2.3 · p. 24, Figure 11 · p. 25
  • Steering is highly orientation-sensitive, usually peaking near canonical orientations but not aligning perfectly with them, especially for Gemini 3.1 Pro and GPT-5.4.Three symbols were rotated from 0° to 315° in 45° increments.Source: §B.2.3 · pp. 24, 26, Figure 12 · p. 26
  • Position and size effects vary by model: Claude Opus 4.7 favors centered and large symbols, GPT-5.4 shows a top-to-bottom gradient, and Gemini’s mean rates stay relatively stable despite symbol-level shifts.For Claude, centered symbols average P(recursion) = 0.83 and large symbols 0.79; small and medium symbols at bottom right suppress steering to 0.00 and 0.38.Source: §B.2.3 · p. 24, Figure 13 · p. 26, Figure 14 · p. 27

B.2.4. Visual steering prompt robustness

  • Steering profiles largely persist across prompt rephrasings that retain explicit choice language, but neutralizing that language nearly eliminates steering for Claude Opus 4.7 and GPT-5.4 with strong iterative priors.Placing images in the middle suppresses steering, while placing them at the end changes which symbols steer most strongly; the authors offer attention as one possible explanation.Source: §B.2.4 · pp. 28–29, Figure 15 · p. 29

B.3.1. Task descriptions

  • Additional probes include recursion versus iteration on six coding problems, DFS versus BFS on four graph or tree problems, two data-structure choices, writing-style choices, and binary preferences.The preference task includes ordinary pairs and random character-string pairs; style contrasts include professional versus poetic and terse versus verbose.Source: §B.3.1 · pp. 30–32, Table 2 · p. 30

B.3.2. Additional task results

Across additional tasks, visual symbols steer coding strategies, writing styles, and binary preferences. Steering profiles show more shared structure in shapes than colors, while less capable models are more sensitive to noise and ImageNet images.

B.3.2. Additional task results

  • Across baseline-iterative coding problems, visual symbols strongly steered all three models toward recursion, while Gaussian noise induced little to no steering.Gemini 3.1 Pro’s branching arrows and GPT-5.4’s hearts remained among their top steering symbols across problems.Source: §B.3.2 · p. 32, Figure 16 · p. 33, Figure 17 · p. 34
  • Circular shapes consistently steered models toward ring-buffer implementations rather than their default linked-list baseline across three coding problems.GPT-5.4 is omitted because it showed no steering on any of the three problems.Source: §B.3.2 · p. 32, Figure 18 · p. 35
  • In writing tasks, warm and cool colors steered Claude Opus 4.7 toward optimistic and melancholic prose, while expansive and straight shapes steered Gemini 3.1 Pro toward verbose and terse prose, respectively.For poetic versus professional prose, fluid shapes steered toward poetic writing and structured icons toward professional writing.Source: §B.3.2 · p. 36, Figure 19 · p. 36, Figure 20 · p. 37
  • Visual symbols steered all three models on matched semantic preferences, whereas length-6 random strings left Claude Opus 4.7 and Gemini 3.1 Pro unsteered and shifted only three GPT-5.4 pairs.Some semantic pairs also shifted strongly under Gaussian noise, indicating generic visual perturbation rather than a purely semantic signal in those cases.Source: §B.3.2 · pp. 36, 38, Figure 21 · p. 38

B.4. Is visual meaning universal across models?

  • Across 288 symbols, 13 of 15 model pairs had |r| ≤ 0.16; shape profiles correlated up to r = 0.75, while no color pair had significant positive correlation (p ≥ 0.28).Shared shape profiles were anchored by recursion triggers such as hearts and triangles; color sensitivity was the main source of cross-model variance.Source: §B.4 · p. 38

B.5. Less capable models tend to be more sensitive to random noise and ImageNet images

  • Claude Sonnet 4.5 and Gemma 4 31B-IT were 100% recursive at baseline, and the smaller models were generally more sensitive to noise and ImageNet images than frontier models.Gemma 4 31B-IT was the exception to the increased sensitivity comparison.Source: §B.5 · p. 39

B.6. Silent injection results

Silent visual-symbol injection steers differently from explicitly instructing models to inspect the image. Across the 288-symbol library, explicit and silent steering were only weakly correlated.

B.6. Silent injection results

  • Silent injection and explicit image instruction produced different steering effects on inorder binary tree traversal.The comparison covers the 288-symbol library.Source: §B.6 · p. 39
  • Across 288 symbols, explicit-versus-silent steering correlations were r = 0.04 for Claude Opus 4.7, r = 0.14 for Gemini 3.1 Pro, and r = 0.18 for GPT-5.4.The reported p-values were 0.48, 0.021, and 0.0024, respectively.Source: §B.6 · p. 39
  • The figure compares top-three shapes and steering deltas under silent injection with explicit-versus-implicit steering across models.The comparison plots explicit steering on the y-axis and silent-injection steering on the x-axis.Source: §B.6 · p. 39

B.7. Model symbol interpretation methodology and results

The paper compares models’ direct interpretations of visual symbols with their empirical steering effects on recursion. These measures only weakly align, and models sometimes offer unusual explanations for the symbols’ influence.

B.7. Model symbol interpretation methodology and results

  • Cross-model visual steering correlations were generally higher for shapes than for colors.Figure 23 reports Pearson correlations for inorder binary tree traversal.Source: Figure 23 · p. 40
  • The interpretation prompt asked models to choose recursion or iteration from an image and explain their answer; answers were extracted from tagged responses.The method queried interpretations for the 288 visual symbols.Source: §B.7 · pp. 39–40
  • Interpretation proportions only weakly tracked empirical recursion steering: r = 0.22 for Claude Opus 4.7, r = 0.29 for Gemini 3.1 Pro, and r = 0.31 for GPT-5.4.The analysis covered 288 symbols for Claude Opus 4.7 and GPT-5.4, and 171 for Gemini 3.1 Pro after API errors were excluded.Source: §B.7 · pp. 40–41
  • Claude Opus 4.7 and GPT-5.4 often linked hearts to recursion, while Gemini 3.1 Pro connected an exclamation-point warning sign to factorials and recursion.The paper provides examples of interpretations that both match and mismatch empirical steering.Source: §B.7 · p. 41, Table 5 · p. 41, Table 6 · p. 42

B.8. Thinking trace faithfulness: methodology and extended results

The appendix classifies thinking traces by whether they attribute influence to an image or disregard it, then reports trace and communication-game results. The game pairs sender and receiver models to test visual communication across coding, concept, and story tasks.

B.8. Thinking trace faithfulness: methodology and extended results

  • Trace classification identifies visual-stimulus mentions, categorizes them as attribution or disregard, and treats a symbol as steering at an absolute delta of at least 0.10.When both categories appear, an explicit adoption phrase or the final stance in the trace resolves the classification.Source: §B.8 · p. 44
  • Under silent injection, GPT-5.4 omitted image mentions in 100% of traces; Gemini 3.1 Pro omitted them in 35.5% of steering traces and 16.9% of non-steering traces.With explicit image instruction, image mentions were omitted in 32.7% of GPT-5.4 steering traces and 9.6% of non-steering traces; Gemini’s corresponding rates were 3.5% and 2.8%.Source: §B.8 · p. 44, Table 8 · p. 45

C.1. Communication game setup

  • The communication game pairs a sender that generates a visualization with a receiver that identifies a target concept or story, or uses a coding strategy on a problem.Receiver samples use four answer options, and sender scores are proportions correct across 10 independent samples.Source: §C.1 · p. 46
  • Games ran for up to 20 turns for code steering and five turns for concept and story communication, ending early when receiver accuracy reached 100%.The free-form channel allowed Python PIL drawings without text; the symbol channel used one of 288 base shape-color combinations with limited spatial transformations.Source: §C.1 · p. 46

C.2. Example concepts and stories

  • The concept and story communication experiments used 30 abstract concept sets and 20 three-sentence stories generated with Gemini 3.1 Pro.The concept sets included coding and more abstract concepts; stories were specified not to include color references.Source: §C.2 · pp. 46–47

C.3. Detailed convergence statistics for concept and story communication

  • Table 13 breaks down concept and story communication convergence by model and channel; convergence means 100% correct receiver identification across 10 samples.The table presents detailed convergence rates and the generation-interpretation disparity.Source: §C.3 · p. 47, Table 13 · p. 48

C.4. Symbol clouds (sender choices vs. actual steering symbols)

Models’ chosen symbols for communicating recursion often differ from the symbols that actually steer them. The section also presents examples from visual communication tasks and describes how a human survey evaluated interpretations of those visualizations.

C.4. Symbol clouds (sender choices vs. actual steering symbols)

  • Models often choose recursion symbols that differ from those with actual steering effects, as the symbol clouds compare choice frequency with steering strength.Boxed numeric annotations show steering effects toward recursion; symbols without annotations have no actual steering effect.Source: §C.4 · p. 48, Figure 26 · p. 49, Figure 27 · p. 49

C.5. Additional visualization examples

  • Examples show recursion-versus-iteration, abstract-concept, and short-story visualizations across communication turns and models.The story examples use free-form and symbol-library visualizations, and models appear to arrive at largely similar story layouts.Source: §C.5 · p. 51, Figure 29 · p. 51, Figure 30 · p. 52

D. Human study details

  • The human study received internal ethics approval, and participants gave informed consent.Source: D. Human study details · p. 54
  • The human survey had 48 items in three fixed-order blocks covering recursion versus iteration, concept matching, and story matching.Participants selected among two choices for recursion-versus-iteration and four options for concept and story items.Source: D. Human study details · p. 54
  • Survey options and item order were shuffled, and concept and story blocks included first- and final-turn visualizations.Participants evaluated either free-form PIL or symbol visualizations, across all three models; they were not told the comparison or model names.Source: D. Human study details · p. 54

Related papers

Loading 2610.36449v1…