PaperlayerView exact revision on arXiv

Source-linked AI summary

How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning

Qian Yang, Ankur Sikarwar, Huy Le, Le Zhang, Zhuan Shi, Perouz Taslakian, Aishwarya Agrawal

arXiv:2605.27310v1cs.CV

TL;DR

Cross-view spatial reasoning remains difficult for VLMs, which may underuse intermediate thinking images. This paper introduces View Dropout and compares panoramic, top-down, and point-matching visual thinking, finding that panoramic thinking with VDrop achieves the best out-of-domain generalization.

  • Problem

    VLMs must reason across partially overlapping views, but visual thinking images can be underused during answer prediction.

  • Method

    The paper introduces View Dropout and compares panoramic, top-down, and point-matching thinking images using a learnability–informativeness framework.

  • Results

    A 6.7-point OOD gain over vanilla BAGEL was achieved with 8K synthetic samples, while panoramic thinking with VDrop was the only configuration both informative and learnable.

  • Takeaways & Limitations

    Panoramic visual thinking with VDrop delivers the strongest reported out-of-distribution cross-view spatial reasoning in this study.

  • Takeaways & Limitations

    The findings were validated on a single UMM architecture, so their transferability to other UMMs remains untested.

Abstract

from arXiv · show

Cross-view spatial reasoning remains a weak spot for vision-language models (VLMs): they often reason in language and lose the fine-grained geometry needed for the task. Thinking with images aims to address this by generating an intermediate thinking image, but recent work shows that models often ignore the visual evidence in these traces. We therefore ask how to make visual thinking matter, and what kind of visual thinking works best. We study these questions in unified multimodal models (UMMs), which natively support interleaved image-text generation. For the first question, we propose View Dropout (VDrop), a training-time intervention that hides parts of one input view from the answer span while keeping them visible to the thinking-image tokens. This encourages the model to use the thinking image when answering, instead of relying only on the input views. Once the thinking image is used for answer prediction, we study which type of visual thinking is most effective. We frame this as a learnability-informativeness tradeoff and compare three thinking-image variants: top-down, panoramic, and point-matching renderings. Trained on synthetic scenes and evaluated on five real-world out-of-domain benchmarks, panoramic visual thinking with VDrop is the only configuration that is both informative and learnable, and it achieves the best out-of-domain generalization.

1 Introduction

Cross-view spatial reasoning requires maintaining a consistent scene representation across partially overlapping viewpoints, yet strong VLMs remain near chance because language-only reasoning discards fine-grained geometry. The paper proposes View Dropout and compares visual thinking images, finding that panoramic thinking with VDrop best supports out-of-domain generalization.

  • Problem: Cross-view spatial reasoning asks VLMs to infer scene layout, object placement, and geometry across two partially overlapping views while maintaining a consistent scene representation.The capability underlies applications including embodied navigation and integrating temporally distant video frames.
  • Problem: Strong VLMs perform marginally above chance because language-only intermediate reasoning discards the fine-grained geometry required for inherently visual cross-view reasoning.Unlike VLMs, humans can construct internal spatial layouts in the visual domain.
  • Motivation: Standard supervised fine-tuning teaches plausible thinking-image generation but not its use for answering, so removing the image at inference barely changes accuracy.The thinking image therefore becomes a decorative by-product rather than a functional reasoning intermediate.
  • Method: View Dropout masks part of one input view from the answer span during training, forcing spatial evidence to pass through the generated thinking image without architectural changes.The method is agnostic to the thinking-image type and is designed to make visual thinking causally load-bearing.

2 Related Work

Cross-view spatial reasoning is a documented weakness of current VLMs, with strong open-source models scoring only marginally above chance. Related remedies primarily target the language pathway, while UMM research explores unified architectures for multimodal understanding and generation.

  • Cross-View Spatial Reasoning: Cross-view spatial reasoning covers object positions, distances, depth ordering, and viewpoint relationships across multiple views, yet remains a documented weakness of current VLMs.The cited literature reports this weakness across multiple recent studies.
  • Cross-View Spatial Reasoning: Even strong open-source VLMs score only marginally above chance on cross-view spatial reasoning.
  • Cross-View Spatial Reasoning: Existing remedies primarily target the language pathway rather than directly addressing the visual geometry of cross-view reasoning.
  • Unified Multimodal Models: TUNA builds a single continuous visual representation, cascading a VAE encoder with a representation encoder so understanding and generation share one feature space.
  • Unified Multimodal Models: BAGEL couples a multimodal understanding encoder with a diffusion-based image generator through a unified token interface.

3 Method

The method trains unified multimodal models to generate an intermediate thinking-image before answering, then introduces View Dropout to encourage answer tokens to use that image. It compares panoramic, top-down, and point-matching renderings as visual-thinking strategies for cross-view spatial reasoning.

  • Visual-thinking traces: A UMM generates an interleaved sequence in which a thinking-image Ivt precedes the final textual answer a, with both supervised end-to-end.The thinking-image is conditioned on the two input views and question; the answer is conditioned on the inputs and generated thinking-image.
  • View Dropout: View Dropout masks one input view from answer-side attention while preserving its visibility to thinking-image queries, forcing the answer to use Ivt rather than treating it as decorative.Only one view is masked, and the mask covers both ViT and VAE token streams so the answer cannot recover the hidden region through the other stream.
  • Training curriculum: VDrop uses a warmup followed by linear masking annealing because applying the mask immediately collapses learning before the thinking-image has learned what to encode.The masking probability is 0 during warmup, increases linearly during annealing, and reaches 1 thereafter.
  • Compatibility: VDrop changes only the answer-side attention mask, leaving the SFT objective and full-input thinking-image generation unchanged, and is compatible with any thinking-image strategy.The thinking-image remains supervised toward its ground-truth render as under standard SFT.
  • Thinking-image variants: The study compares panoramic, top-down, and point-matching renderings, respectively unifying views into one visual field, exposing a shared external frame, or annotating cross-view identity.The method avoids intermediates requiring auxiliary modules, such as depth maps or 3D reconstructions, to isolate the thinking-image contribution.
  • Training data: Training data from Infinigen Indoors pairs overlapping egocentric views with ground-truth answers and top-down, panoramic, and point-matching thinking-images derived from procedural 3D annotations.These annotations provide clean supervision for each visual-thinking strategy.

4 Experiments

Experiments show that visual thinking improves cross-view spatial reasoning, while VDrop consistently increases the usefulness of generated thinking-images and makes them load-bearing for answer prediction. Among the tested representations, panoramic visual thinking offers the strongest combination of informativeness and learnability, yielding the best out-of-domain performance.

  • Experimental setup: All models are fine-tuned from BAGEL and compared against direct-answer, textual-CoT, and prior BAGEL-based visual-thinking baselines.The experiments also compare visual-thinking variants trained with and without VDrop.
  • Impact of view dropout and visual thinking: VDrop raises the out-of-domain average for every thinking-image type, with panoramic improving from 37.6 to 40.0.The corresponding gains are +2.4 for panoramic, with VDrop also improving top-down and point-matching strategies.
  • Impact of view dropout and visual thinking: 40.0% OOD with 8K samples surpasses ThinkMorph at 37.2% and BAGEL-Zebra-CoT at 26.8%, despite those methods using 3× and 23× more samples.Because these methods fine-tune the same BAGEL backbone, the comparison supports the importance of the intermediate representation and VDrop training recipe.
  • Impact of view dropout and visual thinking: All three VDrop visual-thinking variants outperform non-visual baselines on in-domain and out-of-domain averages.Relative to No-Think at 35.1% OOD, panoramic, top-down, and point matching gain +4.9, +2.9, and +1.0 points; their gains over Text CoT are +7.3, +5.3, and +3.4.
  • Causal use of thinking-images: Generate-then-blind testing shows that VDrop-trained panoramic models suffer substantial accuracy drops when the thinking-image is blinded, whereas standard SFT is largely invariant.This confirms that VDrop makes the generated thinking-image load-bearing rather than decorative.
  • Thinking-image types: Panoramas are strong on both informativeness and learnability; top-down views are informative but only partially learnable, while point matching is mainly limited by low informativeness.Generated panoramas remain net-positive on Counting, Rel-Dist, and Rel-Dir, whereas top-down images help on only two spatial subtypes and regress on others.

5 Conclusion

The conclusion addresses how to make visual thinking matter in unified multimodal models for cross-view spatial reasoning and which intermediate representation is most effective. It identifies pervasive under-use, proposes View Dropout, and frames intermediate selection as a Learnability–Informativeness tradeoff.

  • The work studies how to make visual thinking matter and determine which kind is most effective for cross-view spatial reasoning in UMMs.
  • Under-use is identified as a pervasive failure of current visual-thinking pipelines.
  • View Dropout (VDrop) is proposed as a training-time intervention that makes the generated thinking image a load-bearing component of reasoning.
  • The choice of intermediate visual thinking representation is framed as a Learnability–Informativeness tradeoff.

Limitations … B Training Data C Experiments

The study validates VDrop and the Learnability–Informativeness framework only on BAGEL, leaving their transferability to other unified multimodal model architectures untested.

  • Limitations: The evaluation uses a single UMM, BAGEL, so whether the findings transfer across architectures remains untested.BAGEL is described as a representative, state-of-the-art open-source UMM and the backbone of ThinkMorph and BAGEL-Zebra-CoT; cross-architecture evaluation is left for future work.

A Evaluation Benchmarks

The evaluation uses one in-domain benchmark and five real-world out-of-domain benchmarks for cross-view spatial reasoning. All tasks are multiple-choice and scored by accuracy, generally using exact-match comparison of extracted answer tokens with ground truth.

  • Evaluation setup: The evaluation covers one in-domain benchmark and five real-world out-of-domain benchmarks spanning diverse cross-view spatial reasoning skills.All benchmarks use multiple-choice questions, and accuracy is the reported metric.
  • ID Benchmark: COSMIC is the in-domain benchmark, using Infinigen-generated scenes and testing object-level and relation-level cross-view reasoning.Object-level tasks include anchor recognition and global counting; relation-level tasks include relative distance reasoning.
  • OOD Benchmarks: MMSI-Bench contains 1,000 challenging multiple-choice questions created by six 3D-vision researchers, with distractors and stepwise reasoning processes.The strongest open-source models achieve roughly 30% accuracy, while GPT-5 reaches 41.9% and humans score 97.2%.
  • OOD Benchmarks: MindCube evaluates spatial mental-model construction from partial observations across Rotation, Around, and Among settings.These settings test holistic understanding, object permanence, depth-cue conversion, and spatial consistency.
  • OOD Benchmarks: The remaining OOD benchmarks assess higher-order spatial reasoning and perspective taking in OmniSpatial, non-egocentric viewpoint reasoning in STARE-Perspective, and multiview integration in BLINK-MultiView.OmniSpatial results are reported separately for Complex Logic and Perspective Taking.

B Training Data Details · B.1 Text Chain-of-Thought Annotation · C Experiments

The training pipeline uses procedurally rendered Infinigen scenes with 3D-grounded cross-view questions and synthetic visual/textual reasoning supervision. It separates oracle information from image-verifiable rationales while acknowledging that the resulting traces are not consistently high-quality.

  • B Training Data Details: Infinigen supplies ground-truth thinking images and object-level 3D annotations for automatically constructing unambiguous cross-view questions.The rendered variants include top-down maps, panoramic stitches, and point-matching overlays; annotations contain positions, bounding boxes, and categories.
  • B Training Data Details: The dataset renders two egocentric views with deliberately overlapping fields of view, so shared objects or regions connect the viewpoints.
  • B Training Data Details: Four generated question types target correspondence, counting, metric layout recovery, and reference-frame transformation.They are Anchor, Counting, Relative Distance, and Relative Direction.
  • B Training Data Details: The 8K training set is entirely procedurally rendered from Infinigen Indoors, with fewer samples allocated to Anchor and Counting than to the relative-spatial categories.The allocation reflects that Anchor is exercised across cross-view questions, whereas Counting often concerns objects in the overlapping region rather than genuine cross-view relations.
  • C Experiments: The supplied passages describe training-data construction and rationale annotation, but provide no substantive experimental results for section C Experiments.
  • B.1 Text Chain-of-Thought Annotation: Each sample has a 3D-derived multiple-choice answer but no natural-language rationale, so Qwen3-VL-235B-A22B-Instruct is prompted to synthesize one from the camera images and category-specific instructions.The prompt includes a reasoning-shape instruction block, one curated in-context example, and scene metadata.
  • B.1 Text Chain-of-Thought Annotation: The annotation protocol gives the model private oracle quantities for answer correctness while forbidding those quantities in the rationale.The oracle includes angles, distances, exact object counts, and the gold answer letter; rationales must instead use image-verifiable qualitative cues and relative-camera-pose hints.
  • B.1 Text Chain-of-Thought Annotation: An open-source annotator keeps the 8K-sample pipeline reproducible and less costly than proprietary API annotation, but the synthesized traces are not consistently high-quality.The cost concern arises from conditioning on two images and a long oracle-information context.

C.1 VDrop Mask Design Ablation

The VDrop mask ablation fixes Panoramic visual thinking and the LoRA SFT recipe while varying mask strategy, scope, drop ratio, and curriculum. Region masking on one view at 50% drop with warmup–anneal achieves the best OOD accuracy, while random or two-view masking reduces performance.

  • Ablation setup: The ablation holds Panoramic thinking images and LoRA SFT on 8K samples fixed while varying VDrop’s masking design axes.The compared axes include patch-selection strategy, masked scope, drop ratio, and curriculum.
  • Mask strategy: A 4.4-point OOD decrease from replacing Region with Random masking indicates that spatially coherent occlusion better forces localized structure into the thinking image.The comparison uses the same drop ratio.
  • Mask scope: A 3.2-point OOD decrease from masking both views instead of one suggests that retaining one view is beneficial.The ablation compares simultaneous two-view masking with masking a single primary view.

C.2 Generate-then-Blind Probe: Setup and Details

The generate-then-blind probe tests whether answer prediction causally depends on the generated thinking image by blocking answer-to-image attention after image generation. Compared with Standard SFT, VDrop SFT makes the thinking image load-bearing for visually aligning input views, producing a significant drop on MMSI Measurement questions.

  • Setup: The probe compares Standard SFT without VDrop against VDrop SFT, using the same BAGEL initialization, recipe, and 8K Infinigen training set.VDrop requires masked spatial evidence to pass through the generated thinking image before answer prediction.
  • Intervention procedure: After autoregressively generating the thinking image, the probe sets answer-to-image attention to −∞ before softmax, leaving only V1, V2, and the question available for answer decoding.The model still produces the image but cannot read it back when answering.
  • Evaluation: OOD benchmarks provide a cleaner causal-dependence signal than COSMIC, whose in-distribution accuracy is largely saturated by visual-thinking SFT.Each blinded run is compared with the same model’s unblinded baseline on four OOD benchmarks.
  • MMSI breakdown: +10.1 pp, z = 2.24, p = 0.025: VDrop SFT’s accuracy drops significantly when the thinking image is blinded on MMSI Measurement questions.Measurement requires visually aligning the two input views, whereas Named-region and Cardinal answers contain information a panorama cannot encode and show no positive blinding effect.
  • MMSI breakdown: Standard SFT remains unaffected across every MMSI evidence category, staying within ±2 pp of zero with max |z| = 0.69.This confirms that the generated thinking image is inert without VDrop.

C.3 Answer-Token Attention Probe: Setup and Details

The attention probe measures how answer-token visual attention is distributed across the input views and generated thinking-image tokens, comparing Standard SFT, VDrop SFT, and vanilla BAGEL. On STARE-Perspective, VDrop shifts attention toward the thinking image mainly in early and middle decoder layers, with smaller overall gains as models converge late.

  • Models compared: The probe compares Standard SFT, VDrop SFT, and vanilla BAGEL using answer-generation attention weights extracted across decoder layers.Vanilla BAGEL is forced to generate a comparable thinking-image span and serves only as an untrained reference.
  • STARE-Perspective: per-layer attention share: VDrop’s increased engagement with the thinking image is localized to early-to-middle layers rather than distributed uniformly across the decoder.This localization is consistent across BLINK and STARE.
  • Probe metric: Attention is normalized over visual tokens from input views V1 and V2 and generated thinking-image tokens VT_ALL.The per-layer share ρ_vt,ℓ averages answer-token attention mass across answer positions and heads, then across evaluation examples.
  • STARE-Perspective: per-layer attention share: +3.5 pp is VDrop’s increase in thinking-image attention over standard SFT across the first 14 decoder layers on 250 STARE-Perspective examples.This early-to-middle-layer pattern matches the probe’s BLINK observation.
  • STARE-Perspective: per-layer attention share: +0.9 pp is VDrop’s thinking-image attention gain over standard SFT when averaged across all decoder layers.The smaller overall gain reflects convergence between the models in late decoder layers.

C.4 Qualitative Analysis

Figure 7 qualitatively compares panoramic, point-matching, and top-down thinking images across four cross-view subtasks. The examples illustrate the informativeness–learnability analysis, with panoramic and top-down renderings providing genuinely new scene viewpoints.

  • Informativeness–learnability analysis: Panoramic and top-down thinking images render the scene from a genuinely new viewpoint, illustrating the informativeness axis of the analysis.These examples are used to illustrate both axes of the learnability–informativeness analysis.
  • Subtasks: The displayed questions cover shared visibility, object counting, farthest-object selection, and relative-direction judgments across two room views.The examples ask about a cabinet visible from both views, the total number of cabinets, the object farthest from a cabinet, and the desk’s direction.
  • Qualitative examples: Figure 7 presents one example each for Anchor, Counting, Relative Distance, and Relative Direction, alongside the question, options, input views, generated thinking image, prediction, and correctness.The gold option is marked in green, and each strategy’s predicted answer is shown with a correctness indicator.
Loading 2605.27310v1…