Source-linked AI summary
SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning
Jian Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan
TL;DR
VLMs struggle with reliable 3D spatial reasoning because they inadequately represent fine-grained geometry and spatial relationships, while existing integrations often discard hierarchical geometry features. SpatialStack addresses this gap by progressively aligning multi-level geometry with vision and language representations across the model hierarchy. Experiments and ablations report state-of-the-art performance among evaluated open-source models and strong zero-shot generalization across diverse 3D spatial reasoning tasks.
Problem
VLMs struggle to encode fine-grained 3D geometry and spatial relationships, while existing geometry-aware integrations commonly fuse only final-layer features and discard hierarchical cues.
Method
SpatialStack progressively aligns multi-level geometric features with vision and language representations across the model hierarchy, injecting them into corresponding LLM decoder layers.
Results
SpatialStack achieves state-of-the-art performance among evaluated open-source models and strong zero-shot generalization across diverse 3D spatial reasoning tasks.
Takeaways & Limitations
Hierarchical geometry-language fusion provides an effective and extensible paradigm for integrating local geometric precision with global spatial semantics.
Abstract
from arXiv · showhide
Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems. This limitation arises from their inability to capture fine-grained 3D geometry and spatial relationships. While recent efforts have introduced multi-view geometry transformers into VLMs, they typically fuse only the deep-layer features from vision and geometry encoders, discarding rich hierarchical signals and creating a fundamental bottleneck for spatial understanding. To overcome this, we propose SpatialStack, a general hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy. Moving beyond conventional late-stage vision-geometry fusion, SpatialStack stacks and synchronizes multi-level geometric features with the language backbone, enabling the model to capture both local geometric precision and global contextual semantics. Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks. Extensive experiments and ablations demonstrate that our multi-level fusion strategy consistently enhances 3D understanding and generalizes robustly across diverse spatial reasoning tasks, establishing SpatialStack as an effective and extensible design paradigm for vision-language-geometry integration in next-generation multimodal physical AI systems.
1. Introduction
SpatialStack addresses the gap between VLM semantic alignment and reliable 3D spatial reasoning by progressively aligning hierarchical geometry and language representations. The framework targets both fine-grained perception and high-level spatial understanding.
- Motivation: VLMs often fail to encode 3D geometry, spatial relationships, and language instructions reliably for physical-world tasks.These limitations affect relative-distance estimation, motion-related left-right distinctions, navigation, manipulation, and spatial assistance.
- Motivation: Existing approaches commonly prioritize image-level semantics or depend on externally preprocessed point clouds and depth maps.Multi-view geometry transformers offer a more unified alternative, but their integration with VLMs remains incomplete.
- Research gap: Most geometry-aware integrations fuse only final-layer features, discarding hierarchical cues from intermediate geometry representations.This single-level design can improve selected benchmarks but creates a bottleneck for 3D understanding.
- Approach: SpatialStack progressively aligns multi-level geometric and language features throughout the model hierarchy instead of fusing geometry only at deep encoder layers.The design aims to preserve detailed local geometry and global semantic context.
- Contribution: SpatialStack is presented as a model-agnostic framework, instantiated as VLM-SpatialStack, with experiments across multiple benchmarks and diverse 3D spatial reasoning tasks.The reported evaluations demonstrate strong performance on tasks requiring detailed perception and holistic spatial understanding.
2. Related Work
Prior work developed large multimodal models, spatial reasoning benchmarks, explicit 3D-data injection, and feed-forward geometry encoders. SpatialStack builds on these directions by using hierarchical geometry-language fusion rather than relying only on conventional visual features or late-stage integration.
- Large Multimodal Models: Large multimodal models evolved from joint vision-language pretraining toward instruction-tuned architectures connecting pretrained vision and language components.Examples include CLIP, Flamingo, BLIP, InstructBLIP, Qwen2.5-VL, LLaVA, and MiniGPT-4.
- Spatial Reasoning in VLMs: Spatial reasoning benchmarks quantify deficiencies in standard multimodal models across static, dynamic, and implicit 3D cognition settings.Representative benchmarks include VSI Bench, Spar Bench, BLINK, Cambrian-1, and VLM4D.
- Spatial Reasoning in VLMs: Explicit 3D-data approaches inject point clouds, RGB-D inputs, or related geometric representations into multimodal language models.These methods address spatial reasoning by adding structured geometric information to the multimodal pipeline.
- Vision-Language-Geometry Fusion: Feed-forward geometry encoders such as DUST3R, CUT3R, and VGGT infer dense geometric structure from unposed multi-view images or video.Their outputs provide rich geometric features for integration with multimodal models.
- Vision-Language-Geometry Fusion: SpatialStack follows implicit geometry fusion, using hierarchical geometric features to supply explicit spatial structure missing from internal visual features.This direction differs from methods that distill visual features into explicit 3D or 4D feature fields and maps.
3. How Multi-level Geometry Features Facilitate Spatial Reasoning
The analysis shows that geometry features at different depths support different levels of spatial reasoning: shallow features favor local perception, while deeper features favor global reasoning. This motivates evaluating injection depth and multi-level fusion rather than relying on a single layer.
- Task hierarchy: Spatial tasks range from low-level geometric perception to high-level multi-view spatial reasoning and 3D relational understanding.Examples include local depth or distance comparison versus cross-view object relations and path reasoning.
- Experimental setup: The study injects VGGT features from layers 4, 11, 17, and 23 into the last-layer vision features before concatenating them with text tokens.This single-layer geometry-vision fusion setup evaluates how injection depth affects low- and high-level tasks.
- Layer effects: Deeper injection layers improve high-level tasks, whereas low-level performance declines with depth and peaks at layer 11.The result indicates that shallow features provide local geometric cues while deeper features encode global structural and semantic relationships.
- Fusion comparison: Naive multi-layer fusion does not achieve the best overall behavior, instead compromising performance relative to the 11th-layer single-fusion configuration on low-level tasks.This finding motivates a structured fusion strategy rather than simply combining multiple geometry layers.
4. Where to fuse Multi-level Geometry Features
SpatialStack addresses the question of where to integrate multi-level geometry by moving fusion from the visual pathway into corresponding LLM layers. Its progressive geometry-language design preserves hierarchical spatial information and improves cross-task performance over conventional fusion strategies.
- Fusion location: Naive multi-layer fusion in the visual pathway yields suboptimal task-specific trade-offs, motivating a different integration location.Selecting a single geometry layer performs better for particular task levels than simply combining multiple layers visually.
- Layered fusion: The multimodal sequence combines merged visual tokens and text tokens before processing through stacked LLM decoder layers.Visual tokens are spatially merged, concatenated across frames, and combined with text to form the decoder input.
- Geometry features: VGGT supplies intermediate hidden states as multi-view geometric features rather than using only its final prediction outputs.The extracted states provide progressively richer geometric representations for fusion with the vision-language model.
- Layered fusion: SpatialStack injects geometric features from multiple geometry-encoder layers into corresponding LLM decoder layers as additive residuals.The framework maps VGGT layers 11, 17, and 23 to LLM decoder layers 0, 1, and 2 after layer-specific token merging.
- Empirical comparison: Across four spatial reasoning benchmarks, SpatialStack achieves the best overall average and highest scores on VSI-Bench, SPAR-Bench, and CV-Bench.The Qwen3.5 baseline remains strongest on BLINK-Spatial, while naive geometry-vision fusion suffers severe performance drops there.
5. Experiments
Experiments evaluate SpatialStack across spatial, general multimodal, and spatial-temporal benchmarks, alongside layer-selection and fusion-order ablations. Results show broad gains over comparable baselines, with hierarchical alignment and depth-spanning feature selection supported by the ablations.
- Evaluation setup: SpatialStack is evaluated on spatial and general multimodal benchmarks, including VSI-Bench, CV-Bench, SPAR-Bench, BLINK, and Video-MME.The evaluation covers depth, distance, object-relation, video-based spatial, and general multimodal understanding tasks.
- Spatial reasoning results: SpatialStack-5B establishes a new state-of-the-art among evaluated open-source models and surpasses all open-source systems on VSI-Bench without route-planning training data.The result is reported as robust zero-shot generalization for high-level spatial reasoning.
- Spatial reasoning results: SpatialStack surpasses similar-scale and same-base-model baselines on both the 2D and 3D CV-Bench subsets.The comparison supports multi-level geometry feature stacking for unified spatial perception.
- General capabilities: SpatialStack maintains robust general multimodal and spatial-temporal capabilities without catastrophic forgetting.The evaluated suite includes MMBench, Video-MME, BLINK, and TempCompass.
- Ablation studies: Broad sampling across network depth matters more than specific deep-layer indices in the VGGT layer-selection ablation.Replacing L23 with L21 or L22 produces no significant performance changes.
- Ablation studies: The progressive geometry-language mapping outperforms reverse and vision-fusion baselines on 3 of 4 benchmarks and achieves a higher overall score.The mapping pairs Geo-L11 with LLM-L0, Geo-L17 with LLM-L1, and Geo-L23 with LLM-L2.
6. Conclusion
SpatialStack hierarchically fuses vision, geometry, and language for 3D spatial reasoning. Its shallow-to-deep alignment preserves local geometric detail and global semantic context, supporting strong open-source performance and zero-shot generalization without compromising general multimodal capabilities.
- Conclusion: SpatialStack progressively aligns multi-level geometric features with the LLM decoder to preserve local precision and high-level relational semantics.The paper contrasts this with structural bottlenecks and feature interference from naive multi-layer geometry-vision fusion.
- Conclusion: Shallow geometry layers preserve fine-grained spatial details, whereas deeper layers capture global semantic context.This layer-wise correspondence motivates the framework’s hierarchical design.
- Conclusion: SpatialStack achieves state-of-the-art performance among open-source models and strong zero-shot generalization without compromising general multimodal capabilities.The conclusion presents these results across multiple 3D benchmarks and general multimodal evaluations.
Supplementary Material
The supplementary material provides implementation details and additional experiments for SpatialStack, organized around architecture, data, training, and evaluation protocols.
- Organization: Supplementary Section A details the architecture, including geometry token extraction and masked additive fusion.It also covers the geometry token extraction pipeline and masked additive fusion mechanism.
- Organization: Section B describes the composition and statistics of the training dataset mixture.
- Organization: Section C reports training details, including input processing and specific training configurations.
- Organization: Section D provides detailed evaluation protocols and benchmark specifications.
A. Architecture Details
The architecture extracts and spatially aligns multi-level geometry tokens, merges and projects them into language space, then adds them only to visual-token positions in the decoder. This preserves the separation between geometry-enhanced visual tokens and non-visual sequence elements.
- Architecture pipeline: The architecture processes visual features through geometry extraction, spatial alignment, geometry merging, language-space projection, and masked additive fusion.These stages connect multi-level geometric cues to the VLM decoder.
- Geometry token extraction and alignment: VGGT geometry outputs are reordered from row-major traversal into window-by-window order to align with merged vision tokens.With merge size s, groups of s^2 geometry tokens correspond to the same spatial region as one merged visual token.
- Geometry merging and projection: Normalized geometry tokens are grouped into non-overlapping spatial windows and concatenated along the channel dimension.The resulting merged geometry tokens are then prepared for projection into the language decoder dimension.
- Geometry merging and projection: Each flattened window token is projected to the language decoder dimension by a two-layer MLP.The projection uses the transformation G = W2 σ(W1 ˜Z + b1) + b2.
- Masked additive fusion: Additive fusion updates decoder hidden states only at positions identified as visual tokens by the binary mask Mvis.The projected geometry features are scattered into masked positions, while other positions remain unchanged.
- Masked additive fusion: During autoregressive generation, geometry fusion occurs at the initial prefill step before standard decoding proceeds.This confines geometry injection to the visual-token slice of the decoder state.
B. Dataset Details
The training mixture combines spatial reasoning, general instruction-following, and appearance-order data to support broad task coverage while keeping training computationally manageable.
- Approximately 200k samples combine SPAR, LLaVA-Hound, VLM-3R ScanNet, and VSI appearance-order data.The mixture blends spatial expertise with general instruction-following and adds approximately 2k appearance-order instances.
- The ScanNet split of VLM-3R covers six spatial question-answer types, including counting, distance, direction, size, and room-size reasoning.
- The combined VSI-Type Data organizes seven tasks into Configuration, Measurement, and Spatiotemporal groups.
- Sampling 60% of the main datasets and adding approximately 2k appearance-order instances balances broad coverage with manageable training scale.
C. Training Details
SpatialStack training uses geometry-aware instruction tuning with unified image and video preprocessing, patch-aligned inputs, and mixed spatial task coverage.
- Training updates the language tower and geometry-merger modules while keeping the vision tower and VGGT frozen.
- Videos are decomposed into frames, with K uniformly sampled indices determined from clip duration, temporal interval, and configured bounds.
- The sampled task distribution includes Configuration, Measurement, Relative Direction, Relative Distance, Object Count, and Absolute Distance categories.
- Images are resized to 518 pixels with aspect-ratio preservation and patch-aligned trimming for token merging.
- Training uses AdamW, cosine decay with warmup, bfloat16 precision, and DeepSpeed ZeRO-2 under distributed execution.
D. Evaluation Details
Evaluation uses frozen VGGT geometry features, aligned visual inputs, predefined decoder-layer fusion, and standardized generation settings.
- The evaluated models pair Qwen2.5-VL-3B or Qwen3.5-4B with frozen VGGT-1B and multi-depth SpatialStack fusion.
- Geometry-aware evaluation preprocesses visual inputs at 518 pixels and enforces patch/merge alignment when required.
- Geometry encoder inputs use the same visual content as the vision branch to maintain spatial correspondence.
- Geometry fusion is injected at predefined decoder layers after self-attention and MLP execution, replacing the vision-aligned slice.
- Main-paper benchmark results use greedy decoding with temperature 0, one beam, key/value caching, and task-specific generation limits.
E. More Results
Additional analyses show that geometry features preserve spatial structure across encoder depths more reliably than native vision features, supporting multi-level geometry fusion.
- SpatialStack consistently outperforms SpatialRGPT and Spatialbot across all CV-Bench metrics in zero-shot evaluation.
- Shallow geometry layers preserve fine-grained spatial distinctions and geometric boundaries, whereas deeper geometry layers become more homogeneous.
- Native vision features are noisy and spatially fragmented across depths, collapsing into nearly uniform responses at the deepest layers.
- These findings motivate multi-level geometry fusion because native visual features lack explicit spatial structure needed for relative-geometry reasoning.