PaperlayerView exact revision on arXiv

Source-linked AI summary

(How) Do MLLMs Report Bistable Images Like Humans?

Ryota Takatsuki, Tomoki Doi, Amane Watahiki, Anil K. Seth, Hitomi Yanaka

arXiv:2609.13254v1cs.CVcs.AI

TL;DR

Bistable images support incompatible interpretations, but how MLLMs handle such ambiguity remains poorly understood. The paper tests LLaVA reports on duck–rabbit and synthetic images, then analyzes their internal computation. Visual and linguistic manipulations bias reports while responses remain predominantly exclusive.

  • Problem

    How MLLMs handle ambiguous visual inputs remains poorly understood, including whether they report bistable images in human-consistent ways.

    Source: §1 · p. 1

  • Method

    The authors test LLaVA reports on the duck–rabbit and synthetic Visual Anagrams, manipulating visual cues and linguistic prompts while analyzing report mechanisms.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    Visual and linguistic manipulations systematically bias reported interpretations across the duck–rabbit and Visual Anagrams, while responses remain predominantly exclusive.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Bistable images provide a compact testbed for studying how ambiguous visual evidence is transformed into language.

    Source: §5 · p. 9

    The experiments do not address temporal switching dynamics, and their primary scope is static object-category ambiguity.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Bistable images such as the duck-rabbit are classic stimuli in which one image supports multiple mutually incompatible interpretations, typically reported one at a time in humans. We ask whether multimodal large language models (MLLMs) show similar report behavior and what internal computations support it. Using the LLaVA family, we study two tractable dimensions: modulability, whether reports can be biased by bottom-up visual cues and top-down linguistic priors, and exclusivity, whether responses commit to a single interpretation. We test both on the canonical duck-rabbit and on synthetic Visual Anagrams to mitigate memorization confounds. Behaviorally, both visual and linguistic manipulations systematically shift reports in human-consistent ways, while responses remain predominantly exclusive. Mechanistically, these effects arise from competing image-token representations, distinct pathways for bottom-up and top-down modulation, and a link between exclusive reporting and object-count encoding. Code and data are available at https://github.com/rtakatsky/mllm-bistable-images.

1 Introduction

The paper asks whether MLLMs report ambiguous images in human-consistent ways and what computations support those reports. Across five LLaVA variants, it studies report bias and commitment on canonical and synthetic bistable images.

1 Introduction

  • Understanding how MLLMs transform ambiguous visual evidence into report matters for reliable systems and for comparing computational processing across systems.The comparison does not assume that MLLMs and biological brains share implementation.Source: §1 · p. 1
  • The study operationalizes modulability as systematic bias from visual or linguistic manipulations and exclusivity as commitment to a single interpretation.It tests the canonical duck–rabbit and synthetic Visual Anagrams to mitigate memorization concerns.Source: §1 · pp. 1–2
  • Across five LLaVA variants, human-like bistable report behavior emerges in identifiable computational forms, with reports remaining predominantly exclusive.The framework examines competition between image-token representations, distinct pathways for bottom-up and top-down modulation, and object-count encoding.Source: §1 · p. 2

2 Related Work

Prior work describes human bistable reports as sensitive to both visual and contextual influences, while MLLM studies probe ambiguous figures and visual illusions. This paper’s related work situates its stimulus-driven, mechanistic focus among those benchmarks and analysis methods.

2.1 Bistable perception in humans

  • Human bistable reports can reflect bottom-up visual factors and top-down language, expectations, or context, with partly independent, approximately additive contributions.The paper focuses on modulability and exclusivity as tractable dimensions for ambiguous figures.Source: §2.1 · p. 2

2.2 Bistable perception and relevant phenomena in MLLMs

  • Panagopoulou et al. (2024) found limited effects from most image manipulations but substantial sensitivity to prompt variations; Visual Anagrams reduce likely memorization of canonical examples.The study also compares visual cueing with prompt priors on a single bistable stimulus.Source: §2.2 · pp. 2–3

2.3 Interpreting visual-to-text processing in MLLMs

  • Attention ablations can test which image information causally influences reports, while probing can localize visual information across layers.Unlike work on conflicts between vision and external or in-image text, this study examines competing interpretations within one image.Source: §2.3 · p. 3

3 Behavioral Analyses

Across five LLaVA-family models, the analyses test visual and linguistic modulation and whether ambiguous stimuli elicit exclusive reports, using Harper’s duck–rabbit and synthetic Visual Anagrams.

3.1 General Setup

  • The study uses Harper’s duck–rabbit, 50 generated Visual Anagrams, and matched two-animal and single-animal controls, examining five LLaVA-family models in depth.The standard query asks the model to list every animal; next-token probabilities probe modulation, beam search probes exclusivity, and an object-count query probes single-object construal.Source: §3.1 · pp. 3–4

3.2.1 Bottom-up Modulation

  • Rotation shifts interpretation preferences in Harper’s image and most generated stimuli; red-circle placement near each animal’s distinctive feature shifts its corresponding output.For 37–44 of 50 Visual Anagrams, depending on the model, interpretation reverses at a particular angle; controls show no comparable reversal. Rabbit-related output increases near the mouth, and duck-related output near the beak.Source: Figure 2 · p. 4, §3.2.1 · pp. 4–5

3.2.2 Top-down Modulation

  • Prefix and semantic cues bias reports toward the intended animal, with prefix cues stronger, while the same cues leave that interpretation essentially absent for matched controls.The authors describe linguistic context as biasing competition between interpretations supported by an ambiguous image, rather than inducing a label whenever it is named.Source: Figure 4 · p. 5, §3.2.2 · p. 5

3.3 Exclusivity

  • All five LLaVA-family models answer “1” for Harper’s image and boundary-angle Visual Anagrams, but “2” for matched two-animal controls; this correspondence does not establish causation.The reported ranges are 0.88–0.99 for “1” on ambiguous stimuli and 0.87–0.99 for “2” on controls.Source: §3.3 · pp. 5–6
  • Ambiguous Visual Anagrams receive multiple-animal continuations for only ∼10% of beam mass near the ambiguity boundary, versus ≥98% for matched two-animal controls.The authors report the same predominantly exclusive pattern for ambiguous images in comparison with human bistable perception.Source: §3.3 · p. 5, Figure 5 · p. 6
  • With an interpretation forced in the prefix, single-interpretation continuations outweigh continuations that would list both, even when the forced interpretation opposes the model’s initial preference.The rabbit-prefixed condition behaves similarly to the bird-prefixed condition.Source: §3.3 · p. 5, Figure 5 · p. 6
  • For Raven–Bear figure–ground ambiguity, weaker single-object judgments co-occur with 43–51% of beam mass naming both interpretations, while models with strong single-object judgments remain fully exclusive.This is a boundary condition: the authors note that figure–ground images can more readily support a two-entity construal than duck–rabbit ambiguity.Source: §3.3 · p. 6

3.4 Robustness Across Prompts and Models

  • Qualitative modulation and exclusivity persist across prompts and non-LLaVA models; attribution plots show both interpretation logits rising later, with image-token attention strongest in middle-to-late layers.The authors also report that an explicit two-animal hint can reduce exclusivity in a model-dependent manner, so it is a default construal rather than a hard output constraint.Source: §3.4 · p. 6, Figure 6 · p. 6

4 Mechanistic Analyses

Mechanistic analyses trace bistable-image reports to competing image-token evidence and distinguish bottom-up visual modulation from top-down linguistic modulation. They also identify a shared image-token contribution to object-count judgments and exclusive reporting.

4.1 Neutral Setup

  • Bird and rabbit logits rise sharply from around layer 20; image-token attention contributes strongly in layers 20–26, with heads attending to interpretation-linked regions.Zero-ablation of dominant image embeddings drives the corresponding output-token probability nearly to zero.Source: §4.1 · p. 7, Figure 7 · p. 7

4.2.1 Bottom-up Modulation

  • Across rotation and red-circle conditions, the final logit difference broadly tracks the balance of bird- and rabbit-dominant image patches.This supports bottom-up modulation through changes in image-side evidence available to downstream language-model processing.Source: §4.2.1 · p. 8, Figure 9 · p. 8

4.2.2 Top-down Modulation

  • Top-down cue information moves from prefix to final token mainly over layers 15–20, with early-layer MLPs making especially strong indirect contributions.The observed transfer runs counter to the initial query-mediated hypothesis.Source: §4.2.2 · p. 8
  • Query-side patching recovers only about 3% of the top-down effect, while query-side effects of bottom-up modulation are negligible.This indicates that neither modulation type is primarily driven by changes in text-to-image attention queries.Source: §4.2.2 · p. 8, Figure 11 · p. 9

4.3 How Does Exclusivity Occur?

  • A small subset of image tokens has large positive effects on both object-count and exclusivity contrasts, supporting a shared causal substrate.These tokens are largely distinct from bird- or rabbit-dominant patches, and most number-dominant patches have little individual effect on either measure.Source: §4.3 · p. 8

5 Conclusion

The study finds that LLaVA models show human-like modulability and predominantly exclusive reports on duck–rabbit and synthetic Visual Anagram stimuli. Mechanistic analyses link these patterns to competing image-token representations, distinct modulation pathways, and object-count encoding.

5 Conclusion

  • Bottom-up visual manipulations and top-down linguistic cues systematically bias reports across duck–rabbit and synthetic Visual Anagram stimuli, while responses remain predominantly exclusive.Mechanistic findings link the report patterns to competing image-token representations, distinct modulation pathways, and object-count encoding.Source: §5 · p. 9
  • The authors propose bistable images as a compact testbed for studying how ambiguous visual evidence is transformed into language.Synthetic Visual Anagrams are included because famous bistable images may be memorized by web-scale models.Source: §5 · p. 9

Limitations

The paper’s scope is report behavior and supporting computational mechanisms, not the full range of human bistable perception or perceptual experience. Its experiments use fixed images and reports, focus primarily on object-category ambiguity, and include specific synthetic-stimulus and red-circle procedures.

Limitations

  • The analysis covers report-level modulability and exclusivity, but bistable images do not exhaust human bistable perception or perception more generally.Similarities in report behavior should not be taken as evidence of corresponding perceptual experience or consciousness.Source: Limitations · pp. 9–10
  • The fixed-image, single-report setup does not capture spontaneous perceptual alternations during continued viewing.A closer model-side analogue would require active acquisition of new visual evidence, such as shifting fixation, cropping, or zooming.Source: Limitations · p. 10
  • Experiments primarily concern static object-category ambiguity; other forms of bistability may require different operationalizations.The Raven–Bear figure–ground analysis is an initial boundary case: modulability generalizes, but exclusivity is weaker and covaries with whether the model construes one or two objects.Source: Limitations · p. 10

A.1.1 Visual Anagrams and matched controls

  • The synthetic Visual Anagram set contains 50 candidates accepted by an external VLM judge for yielding duck-class and rabbit-class responses at their respective views.Candidates were generated from different random seeds, pre-rotated, and tested five times per view; the accepted set was drawn from 100 candidates.Source: §A.1.1 · p. 14
  • Each accepted ambiguous stimulus has a style-matched two-animal control and derived duck-only and rabbit-only controls, all of which passed the judge check.The single-animal controls were made by zeroing the other animal’s alpha channel on the same canvas.Source: §A.1.1 · p. 14

A.1.2 Red-circle manipulation

  • The red-circle manipulation sweeps a fixed-radius circle over Harper’s duck–rabbit image on a regular grid and records interpretation-token logits at each position.The image is resized to 336 × 336; the circle has a 24-pixel radius and moves with 48-pixel stride.Source: §A.1.2 · p. 14

A.2 Evaluation details

The appendix specifies how model-specific Visual Anagram decision boundaries are fitted and how animal mentions in continuations are classified. It also reports the lexicon’s agreement with judge labels.

A.2.1 Decision-boundary fitting and per-model inclusion

  • For each model–stimulus pair, a logistic fit to duck- versus rabbit-first-token probabilities defines the decision boundary at 50%; rejected fits are excluded from aligned analyses.Fits are rejected under specified usability, probability-range, slope, or threshold-side-count criteria; Table 1 reports included stimulus counts per model.Source: §A.2.1 · p. 14, Table 1 · p. 15

A.2.2 Classifying reports by the animals they name

  • Aggregate exclusivity labels continuations as single, multiple, or none according to whether they name one, two or more, or no animal.The analysis uses eight-beam search and an LLM judge, averaging each class’s returned-beam probability mass across stimuli.Source: §A.2.2 · p. 15
  • The 103-noun lexicon reproduces the judge’s labels for 97.4% of 207k texts.The lexicon was distilled from 133k generations and matches animal nouns on word boundaries.Source: §A.2.2 · p. 15

B Additional Behavioral Results

Across LLaVA and non-LLaVA models, image manipulations and linguistic cues shift reports, while beam continuations usually favor one interpretation. The strength and profile of these effects vary across models and cue types.

B.1.1 Rotation

  • Rotation shifts duck–rabbit and Visual Anagram reports across LLaVA models, but the reversal’s sharpness and location vary by model.Near 0° on Harper’s version, all models prefer duck-related tokens.Source: §B.1.1 · p. 15, Figure 15 · p. 16, Figure 16 · p. 16

B.1.2 Red Circle

  • Red-circle cues produce a similar spatial dissociation across LLaVA models, with rabbit-related effects near the mouth-like region and duck-related effects near the beak-like region.Across non-LLaVA models, red circles also modulate outputs, but the same spatial pattern does not hold in every architecture.Source: §B.1.2 · p. 16, Figure 17 · p. 17, §B.2.2 · p. 18

B.1.3 Top-down cues

  • Prefix cues consistently shift LLaVA reports toward the intended interpretation, while semantic cues have weaker, more model-dependent effects.Across non-LLaVA models, prefix cues also show stronger effects than semantic cues.Source: §B.1.3 · p. 16, Figure 18 · p. 17, Figure 19 · p. 17

B.1.4 Exclusivity under beam search

  • Multiple-animal continuations account for only 0–12% of beam mass at the Visual Anagram ambiguity boundary, and single-interpretation continuations are preferred across models.On Harper’s duck–rabbit, continuations favor the prefixed interpretation over enumerating both animals.Source: §B.1.4 · p. 16

B.2 Behavioral checks on non-LLaVA models

  • The qualitative pattern of modulability and predominantly exclusive reporting extends to four non-LLaVA models, though effect sizes and reversal profiles vary across architectures.Their rotation-related output shifts are more heterogeneous than within the LLaVA family.Source: §B.2 · p. 16, §B.2.1 · p. 17

B.2.1 Rotation

  • Rotation shifts relative animal-related outputs in non-LLaVA models on both Harper’s version and Visual Anagrams, with more heterogeneous profiles than in LLaVA models.Some models assign substantial probability to competing non-target outputs.Source: Figure 16 · p. 16, §B.2.1 · p. 17

B.2.2 Red-circle modulation

  • Red circles modulate outputs across non-LLaVA models, but the mouth-versus-beak spatial dissociation seen in LLaVA does not recur consistently.SmolVLM and IDEFICS2 shift toward rabbit-related outputs near the mouth; Qwen2-VL-7B and InstructBLIP align less clearly with that arrangement.Source: §B.2.2 · pp. 17–18

B.2.3 Top-down cues

Across non-LLaVA models, prefix cues generally have stronger effects than semantic cues, while beam continuations usually remain single-animal. Matched controls and Raven–Bear tests show where these patterns depend on visual support and stimulus type.

B.2.3 Top-down cues

  • Prefix cues generally produce stronger effects than semantic cues across non-LLaVA models, but susceptibility varies substantially by model.SmolVLM’s strongest prefix cue changes rabbit-related probability by less than 0.01, versus roughly 0.1–0.4 in other models.Source: §B.2.3 · pp. 18–19

B.2.4 Exclusivity under beam search

  • Multiple-animal continuations remain near zero across non-LLaVA models on Visual Anagrams, with single-animal continuations holding most beam mass.SmolVLM assigns a noticeable share to continuations naming no animal, but does not preferentially enumerate both interpretations.Source: §B.2.4 · p. 19, Figure 28 · p. 20

B.3 Matched non-ambiguous controls

  • Matched two-animal controls name both animals at essentially every tested angle, unlike ambiguous stimuli, and show neither rotation-driven reversal nor exclusive reports.The controls contain a duck and rabbit as separate objects and are paired with Visual Anagrams.Source: §B.3.1 · p. 21, §B.3 · p. 21, Figure 29 · p. 22

B.3.1 Rotation and exclusivity on two-animal controls

  • On two-animal controls, reports name both animals across the tested rotation range rather than reversing between exclusive interpretations.This contrasts with the ambiguous-stimulus pattern described in the passage.Source: §B.3.1 · p. 21, §B.3 · p. 21

B.3.2 Top-down cues on single-animal controls

  • When only the opposite animal is visible, cues leave the cued percept near zero, though they can alter other lexical outputs.For LLaVA-1.5-7B, prefix cues raise cued-percept mass to 0.23–0.43 on ambiguous images but only 0.03–0.05 on matched single-animal controls; a “d” cue can shift rabbit-only outputs toward deer, dog, and donkey.Source: §B.3.2 · p. 21

B.4 Generalization to figure–ground ambiguity

  • On Raven–Bear, rotation and linguistic cues still modulate reports, but cue effects are more asymmetric and exclusivity weakens in three of five LLaVA-family models.The stimulus is treated as a boundary condition rather than a uniform generalization of duck–rabbit results.Source: §B.4 · p. 21

B.4.1 Exclusivity under beam search

  • At the original Raven–Bear orientation, three LLaVA-family models assign 43–51% of beam mass to continuations naming both interpretations, while two remain fully exclusive.The fully exclusive models are LLaVA-v1.6-Mistral-7B and Llama3-LLaVA-Next-8B.Source: §B.4.1 · p. 21

B.4.2 Top-down prefix cues

  • On Raven–Bear, a “b” prefix raises bird-related tokens in every model, whereas “d” has weaker, less consistent effects on dog-related tokens.The “d” cue reverses the bird-versus-dog token ordering only in LLaVA-1.5-7B, so the modulation is less reliable than for duck–rabbit.Source: §B.4.2 · p. 21

B.4.3 Object count and exclusivity

On Harper’s version and Visual Anagrams, object-count judgments align with predominantly single-animal reports; on Raven–Bear, the relationship varies by model but broadly covaries with exclusivity.

B.4.3 Object count and exclusivity

  • The Raven–Bear beam analysis classifies continuations by the bird- or mammal-related readings, and its single-letter cue analysis tracks first-token probabilities at the centre angle.Llama3-LLaVA-Next-8B uses a longer continuation budget on this stimulus before naming an animal.Source: Figure 32 · p. 23, Figure 33 · p. 23
  • All five models strongly favor “1” and predominantly report one animal on Harper’s version and Visual Anagrams, while matched two-animal controls favor “2” and multiple-animal reports.The table treats object-count answers and beam-search exclusivity as separate measurements.Source: §B.4.3 · pp. 22–23
  • On Raven–Bear, LLaVA-1.5-7B and -13B favor “2” and often name both interpretations, while Mistral-7B and Llama3-LLaVA-Next-8B favor “1” and remain fully exclusive.LLaVA-v1.6-Vicuna-7B is intermediate; the association is broad, not one-to-one.Source: §B.4.3 · pp. 22–23

B.5 Robustness and boundary conditions across prompts

Prompt changes preserve rotation-related report shifts in several settings but can alter exclusivity, and supplemental analyses extend the reported mechanisms across LLaVA-family models with some implementation limits.

B.5.1 Alternative task prompts

  • Open-ended prompts preserve rotation-dependent shifts and generally favor single-animal descriptions, while an explicit two-animal hint reduces exclusivity by differing amounts across models.The hint raises multiple-animal beam mass to 37–56% in two LLaVA-1.5 models and 13–19% in several others; Qwen2-VL-7B, SmolVLM, and IDEFICS2 remain largely unaffected.Source: §B.5.1 · pp. 23–24, Figure 34 · p. 24

B.5.2 Negative top-down cues

  • Negated prefix cues have little effect, while semantic negation varies: “not avian” and “not terrestrial” often shift reports oppositely, but most other negated cues are weak.“Not Easter” does not reverse the rabbit bias associated with “Easter.”Source: §B.5.2 · pp. 24–25

C.1 Across LLaVA-family models

  • The extended mechanistic analyses cover five LLaVA-family models where applicable, but AnyRes image-token analyses use only the 576-token global base view.High-resolution tile tokens are left unchanged; the analyses use an internal-confidence threshold of 0.1 for dominant image tokens.Source: §C.1 · p. 25

C.1.1 Layerwise direct logit attribution

  • Across five LLaVA-family models, both interpretation logits rise mainly in middle-to-late layers, with substantial image-token attention contributions.The onset and relative attention and MLP contributions vary by model.Source: §C.1.1 · p. 25

C.1.2 Bottom-up modulation: dominant patches vs. final logit

  • Across models, the balance of bird- and rabbit-dominant image tokens broadly tracks the final output under rotation and red-circle modulation.This is consistent with bottom-up modulation being reflected in changes to image-side evidence.Source: §C.1.2 · pp. 25–26

C.1.3 Top-down modulation: resampling ablation across the residual stream, attention output, and MLP output

  • Across models, top-down cue effects concentrate in middle-to-late-layer transfer from the prefix token to the final token, with substantial MLP contributions earlier.Layerwise localization and effect magnitude vary across models.Source: §C.1.3 · p. 26

C.1.4 Query patching: image-attention contribution to top-down and bottom-up modulation

  • Query-side intervention explains only a small fraction of the top-down effect and a negligible fraction of the bottom-up effect in three tested models.The analysis excludes LLaVA-v1.6-Mistral-7B and Llama3-LLaVA-Next-8B because their grouped-query attention is incompatible with the implementation.Source: §C.1.4 · pp. 26–27

C.1.5 Object-count patching and exclusivity

Across five LLaVA-family models, image-token interventions link object-count processing with exclusive reporting, but the relationship is not exclusive to object-count encoding.

C.1.5 Object-count patching and exclusivity

  • Across five LLaVA-family models, interventions increasing the explicit “2”–“1” contrast also tend to increase the “and”–“.” continuation contrast.This supports an association between object-count processing and exclusive reporting.Source: §C.1.5 · p. 27
  • Figures 42–44 show top-down resampling ablations at resid_post, attn_out, and mlp_out, averaged over each model’s valid Visual Anagram stimuli.Source: Figure 42 · p. 28, Figure 43 · p. 28, Figure 44 · p. 28
  • Directly decoding “2” does not by itself indicate causal relevance: most image tokens have little individual effect on either measure.Source: §C.1.5 · p. 28
  • In Llama3-LLaVA-Next-8B, some image-token interventions substantially affect the continuation contrast with little effect on the explicit object-count contrast, so object-count encoding is not the sole factor in exclusivity.Source: §C.1.5 · pp. 28–29
  • Figure 45 compares image-attention contributions to top-down versus bottom-up modulation and omits two models with grouped-query attention.Source: Figure 45 · p. 29

D Additional Discussion and Future Directions

The discussion identifies boundaries of the study’s static, object-category focus and outlines extensions involving ongoing perception, prior perceptual history, other forms of ambiguity, and relations among figures.

D Additional Discussion and Future Directions

Switching dynamics and active visual sampling

  • The fixed-image setup does not capture spontaneous perceptual alternations or active visual sampling; a closer model-side analogue would acquire new evidence over time, such as through shifting fixation, cropping, or zooming.Source: D Additional Discussion and Future… · p. 29

History dependence across repeated presentations

Other forms of bistable perception

  • The study centers on static object-category ambiguity; depth, motion, and figure–ground ambiguities require different operationalizations, and broader generalization remains future work.Source: D Additional Discussion and Future… · p. 30

Relational reinterpretation of multiple ambiguous figures

  • The single-object exclusivity measure does not address relational interpretations across multiple ambiguous figures, such as assigning different roles under a relational prior.Source: D Additional Discussion and Future… · p. 30
Loading 2609.13254v1…