Source-linked AI summary
Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Mahtab Bigverdi, Linjie Li, Weikai Huang, Yiming Liu, Jaemin Cho, Tuhin Kundu, Chris Dongjoo Kim, Zelun Luo, Jieyu Zhang, Linda Shapiro, Ranjay Krishna
TL;DR
VLMs struggle with spatial reasoning when the needed structure is unobserved, motivating a method that externalizes imagined spatial views. The paper introduces IPT supervision across three tasks and finds consistent gains over label-only and textual chain-of-thought baselines, including when imaginations are not generated at inference.
Problem
VLMs struggle with viewpoint transformations, path visibility, and integrating partial observations when spatial structure is not directly observable.
Method
The paper introduces Imaginative Perception Tokens, task-specific visual intermediates for Perspective Taking, Path Tracing, and Multiview Counting, trained with BAGEL.
Results
IPT supervision consistently improves spatial reasoning over label-only and text chain-of-thought baselines, even when imaginations are not explicitly generated at inference.
Takeaways & Limitations
IPT provides a supervised intermediate representation for reasoning about unobserved spatial structure and can improve generalization beyond task-specific imagination targets.
Abstract
from arXiv · showhide
Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.
1 Introduction
VLMs struggle when spatial reasoning requires constructing unobserved structure rather than analyzing visible input. The paper introduces Imaginative Perception Tokens and three tasks to supervise such reasoning, improving performance over answer-only and often textual chain-of-thought training.
- VLMs often fail at three-dimensional relations, viewpoint changes, and integrating partial observations into coherent scene representations.
- Spatial reasoning frequently requires imagining missing structure from unseen perspectives or combining incomplete observations into a unified spatial map.
- Imaginative Perception Tokens represent what would be perceived under an alternative spatial configuration while remaining consistent with the observed scene.
- Perspective Taking, Path Tracing, and Multiview Counting require predicting novel views, visibility along paths, or object counts from multiple partial observations.
- Training with imaginative perception supervision improves spatial reasoning over answer-only supervision and often compares favorably with textual chain-of-thought approaches.
- The benefits persist without explicitly generating intermediate images at inference, although they vary across tasks and settings.
- Intermediate perceptual predictions offer a useful direction for spatial reasoning when required structure is not directly observable from the input.
2 Related Works
Prior benchmarks expose persistent spatial weaknesses in VLMs, while existing intermediate representations mainly support reasoning over observed structure. This work isolates constructive spatial imagination as a supervised intermediate substrate for unobserved configurations.
- Spatial benchmarks cover basic relations, multi-image consistency, video-based mental maps, and cross-view scene reconstruction.
- Most benchmarks evaluate discriminative understanding of observed views rather than constructive imagination of missing spatial structure.
- Visual thought methods use drawings, visualization traces, or interleaved text-image reasoning, while other approaches introduce latent visual scratchpads.
- Unlike prior intermediates that visualize visible structure, IPTs predict perception under an unobserved spatial configuration while remaining constrained by the input.
3 Spatial Imagination: Tasks and Datasets
The paper defines three tasks that require constructing missing spatial representations from incomplete observations and pairs them with task-specific ground-truth visual imaginations and answers.
- The three tasks use incomplete inputs and provide paired ground-truth spatial imaginations with final answers in task-specific datasets.Training sets contain 10k–50k examples per task, with human-filtered evaluation benchmarks.
- Perspective Taking: Perspective Taking asks models to answer about a scene from a marked unseen viewpoint, using a directly supervised novel-view rendering.Its categories cover distance changes and left/right relative-position changes, with accuracy averaged across six balanced sub-categories.
- Path Tracing: Path Tracing asks which object an agent would see from a queried side at a path midpoint not revealed by the map or endpoint views.The target is a ground-truth first-person sideview rendered from the midpoint.
- Multiview Counting: Multiview Counting requires resolving occlusions and duplicate appearances across first-person frames by constructing a unified top-down representation.
4 Method: Imaginative Perception Tokens
IPT decomposes spatial answering into generating a visually grounded representation of an unobserved configuration and then predicting the answer from it. BAGEL is trained jointly on the imaginative intermediate and final answer, with optional explicit generation at inference.
- IPT makes a visual representation of a non-observed spatial configuration a functional prerequisite for answering a spatial query.
- The model first predicts imaginative tokens from observed images and a query, then conditions answer generation on those tokens.
- BAGEL provides a unified decoder-only architecture with shared attention across multimodal understanding and image-generation experts.
- The generation target is a precise view imagination representing unobserved three-dimensional scene structure rather than a stylistic image.
- Training Objective: Training jointly optimizes flow matching for the imaginative latent and language modeling for the final answer.
- Training Objective: The flow-matching objective learns a velocity field that transforms Gaussian noise into the target latent representing the unobserved view.
- Training Objective: The language-modeling objective predicts answer tokens conditioned on observed context and ground-truth imaginative tokens.
- Inference: At inference, the model can answer directly in text-only mode or generate and re-encode a visual imagination before answering.
5 Experiments
Experiments evaluate IPT across three spatial reasoning tasks and show that imagination supervision improves performance, transfers across environments, and often outperforms textual chain-of-thought training. Results also show that resolution and inference mode matter, while answer-only inference can preserve gains without explicit imagination generation.
- Experimental setup: The evaluation covers PET, PT, and MVC on AI2-THOR, with additional transfer tests on Habitat, real-world images, and external benchmarks.Models use balanced multiple-choice evaluation, and PT accuracy averages across progressively informative input variants.
- Main results: Answer-only supervision raises Bagel accuracy from 40.3% to 97.5% on PET, 29.9% to 65.7% on PT, and 35.4% to 63.9% on MVC.The gains transfer to Habitat PET, where label-only reaches 82.0%.
- Main results: IPT reaches 67.3% on MVC versus 63.9% for label-only and 62.3% for Text CoT, while Mixed Training achieves the best PT results on synthetic and real benchmarks.IPT models are evaluated in answer-only mode, so the model need not generate an image at inference.
- Main results: Text CoT trails label-only and IPT, including PET at 83.1% versus 97.5% and MVC at 62.3% versus 67.3%.The authors attribute this pattern to language competing with answer prediction and poorly expressing viewpoint changes, occlusions, and cross-view correspondences.
- Ablations: Increasing resolution from Latent-4 to Latent-64 improves AI2-THOR PET from 87.4% to 96.8% and MVC from 53.5% to 63.1%.Habitat PET instead peaks at Latent-32 with 87.0% and drops to 83.3% at Latent-64, suggesting mild appearance-statistics overfitting.
- Ablations: On PT, IPT with answer-only inference reaches 61.1% versus 50.4% with explicit generation, and matches GPT-5 in answer-only mode.Ground-truth imaginations reveal remaining headroom, especially for PT, where accuracy rises from 50.4% to 86.7%.
- Transfer: IPT transfers to aligned external benchmarks, with Mixed Training reaching 63.6% on SAT and 37.0% on MessyTable.Fine-tuning on AI2-THOR MVC data also improves ScanNet from 40.5% to 52.0%, MindCube from 39.5% to 47.5%, and All-Angles from 40.0% to 50.0%.
6 Conclusion
The paper introduces Imaginative Perception Tokens to externalize reasoning about unobserved spatial structure and evaluates them across three dedicated spatial reasoning tasks. Experiments show consistent gains over label-only and text chain-of-thought supervision, while ablations identify imagination quality as a key determinant of accuracy.
- Imaginative Perception Tokens externalize spatial reasoning about unobserved structure in multimodal language models.The approach is studied through Perspective Taking, Path Tracing, and Multiview Counting, each paired with ground-truth intermediate imaginations and final answers.
- Imagination supervision consistently improves spatial reasoning over label-only and text chain-of-thought baselines, even without generating images at inference.
- Ground-truth intermediate imaginations reveal substantial headroom over current model-generated imagination quality.The ablations show that downstream accuracy is directly governed by the quality of the imagination.
A.1 Training Setup
The training setup compares visual, textual, label-only, and mixed output regimes, using task-specific prompts and supervision formats. Text chain-of-thought prompts structure spatial reasoning into explicit steps, while mixed training interleaves imaginative perception examples with direct-answer examples.
- Training modes: IPT training activates both visual-generation and language-modeling losses, whereas Text CoT and label-only training use only language-modeling loss.
- Training modes: Mixed training combines 50% IPT examples with 50% answer-only examples, allowing one checkpoint to switch between visual-token generation and direct answers.
- Text CoT: Text CoT prompts guide path tracing through path direction, midpoint orientation, egocentric left/right interpretation, and distractor elimination.
- Text CoT: Perspective Taking CoT explanations are generated using privileged correctness information but must be written as if only the original image and question were available.
- Text CoT: Multiview Counting CoT uses frame-by-frame explanations that describe newly visible objects without directly referencing hidden instance IDs.
A.2 Evaluation Setup
Evaluation uses inference settings matched to each training configuration and applies a unified answer-extraction and scoring procedure. IPT models are tested both with intermediate image generation and in answer-only mode, where no image is generated.
- Evaluation modes: IPT models are evaluated in imagination mode and answer-only mode, while other variants use the mode matching their training configuration.
- Evaluation modes: In answer-only mode, IPT models produce only a text answer without generating an image at inference.
- Inference configuration: Visual-generation models load VAE weights and encode inputs through both VAE and ViT pathways to match training-time processing.
- Answer extraction: Predicted answer letters are extracted through cascading tag, boxed-format, phrase, bold-format, and final-letter rules before unified scoring against ground truth.
B Data Curation Details
The path-tracing data combines synthetic and real-world environments with controlled scene, path, camera, rendering, and question-generation procedures. Filtering and debiasing enforce visibility, distractor, answer-quality, and answer-distribution constraints.
- Data sources: Path-tracing data comes from AI2-THOR synthetic scenes and Matterport3D real-world scenes for cross-domain evaluation.
- Scene selection: The synthetic scene pool uses 120 iTHOR scenes across four room types, with additional procedurally generated houses in training.
- Path sampling: Paths are sampled as feasible two-waypoint routes balanced across room types and short, medium, and long distance bins.Grid-based sampling uses 0.5 m spacing and requires at least 1.0 m waypoint separation.
- Rendering: Midpoint sideviews are selected from 147 candidates using object coverage and path-relative view-angle constraints.
- Curation: Question variants combine multiple map and viewpoint inputs, while TIFA-style filtering removes ambiguous or incorrectly verified samples and deterministic reshuffling reduces positional bias.
- Dataset statistics: The synthetic training set contains 11,204 examples.
Real-World Data (Matterport3D)
The paper constructs spatial reasoning data from synthetic, photorealistic, and real-world indoor scenes, using geometric filtering, automated verification, and human review to produce unambiguous viewpoint-transformation questions.
- Matterport3D images are collected as top-down per-floor views with interface elements removed and dark borders cropped.
- Walking paths use GPT-proposed start, midpoint, and endpoint waypoints, followed by verification of walkability and objects on each side.
- Geometric filters reject short, out-of-bounds, endpoint-adjacent, or dark-region paths before further verification.
- Surviving annotations receive TIFA checks and human review, with paths requiring at least two verified objects on each side.
- Perspective-taking data spans AI2-THOR, Habitat, and VST, while questions cover distance change and left/right relations after viewpoint transformations.
- Real-world VST examples pair indoor-scene images from different viewpoints with camera-motion questions and corresponding answers.
C Comparison with Related Spatial Benchmarks
IPT is presented as a broader spatial reasoning resource than prior benchmarks because it combines training data, scalable generation, paired visual and textual thoughts, and model training.
- IPT combines training data, scalable generation, paired ground-truth visual and textual thoughts, and a complete model training framework.
D Additional Results
Additional results illustrate how imaginative perception operates across path tracing, perspective taking, and multiview counting, with generated visual thoughts approximating task-specific spatial structure.
- Path tracing is evaluated across EgoDir, Path, PathArr, Real, and Real+Arr benchmark splits.
- The evaluation compares Show-o2, ThinkMorph, Bagel variants, human and random baselines, and a text-only Bagel variant.
- Path Tracing: Path tracing visual thoughts synthesize a midpoint sideview from a top-down map and endpoint views, requiring reasoning about camera height, object layout, and occlusion.
- Perspective Taking: Perspective taking generates a novel viewpoint from a marked first-person scene and predicts distance or left/right relations after movement.
- Multiview Counting: Multiview counting generates a top-down BEV map from four egocentric views along rotation or multi-camera trajectories before counting target objects.
G Imaginative Token Exploration with Different VLMs
Experiments with discrete IPTs across VLMs show benefits on path tracing but degraded reconstructions and remaining performance gaps that motivate unified continuous-latent modeling.
- VQGAN settings trade reconstruction fidelity against vocabulary size or sequence length through codebook size and spatial downsampling ratio.
- Discrete IPT experiments append VQGAN codebook tokens to Qwen2.5-VL and compare answer-only, textual CoT, image CoT, and zero-shot variants.
- IPT outperforms answer-only finetuning and Text CoT on Path Tracing, with CB 1K f=16 reaching 55.0 for 3B and 55.9 for 7B.
- On Perspective Taking, discrete-token gains are modest and near the zero-shot baseline, with lower data quality and reconstruction limits identified as bottlenecks.
- Decoded RGB and grayscale IPTs remain visually degraded and fail to preserve ground-truth spatial structure, motivating continuous latent representations in BAGEL.
- 55.0→59.6 on PT and 50.0→55.5 on PET when switching from RGB to grayscale tokens, while depth tokens perform comparably to RGB.