PaperlayerView exact revision on arXiv

Source-linked AI summary

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang

arXiv:2602.21186v2cs.CV

TL;DR

VLMs remain limited in 3D spatial reasoning, and view-conditioned geometric priors leave them to infer global scene structure from partial cues. Spa3R learns a unified, view-invariant representation by predicting novel-view feature fields and integrates it into a VLM. Spa3-VLM averages 58.6% on VSI-Bench and improves over its Qwen2.5-VL baseline across three additional spatial reasoning benchmarks.

  • Problem

    VLMs receive partial, view-conditioned geometric cues and must integrate them into coherent 3D scene representations under sparse supervision.

    Source: §I · pp. 1–2

  • Method

    Spa3R self-supervises a unified, view-invariant spatial representation by encoding context views and predicting spatial feature fields at queried novel views.

    Source: §I · pp. 1–2

  • Results

    Spa3-VLM scores 58.6% on VSI-Bench and consistently improves over the Qwen2.5-VL baseline across three additional spatial reasoning benchmarks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Controlled ablations show gains from joint geometric–semantic targets, the PSFM representation, and cross-attention integration.

    Source: §V · p. 9

Abstract

from arXiv · show

Vision-language models excel at 2D visual understanding but remain limited in 3D spatial reasoning. Existing approaches either depend on explicit 3D modalities, which limits scalability, or inject partial, view-conditioned geometric priors and leave the language model to recover global scene structure from sparse cues. We introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation from unposed multi-view RGB images. Its Predictive Spatial Field Modeling objective compresses context views into a compact latent representation and predicts aligned geometric and semantic feature fields at novel viewpoints, thereby encouraging coherent encoding of scene geometry and layout. We integrate the pre-trained Spa3R Encoder into a vision-language model through a lightweight residual cross-attention adapter, yielding Spa3-VLM and grounding language reasoning in global spatial context. Spa3-VLM achieves an average score of 58.6% on VSI-Bench and delivers leading or competitive performance across three additional spatial reasoning benchmarks. These results demonstrate that predictive spatial representation learning provides an effective visual foundation for 3D reasoning.

I. INTRODUCTION

VLMs lack incentives to integrate multiple views into coherent 3D scene representations, while existing geometric priors leave global structure to be inferred from partial cues. Spa3R learns a view-invariant spatial representation by predicting novel-view feature fields, and Spa3-VLM integrates it into a VLM and improves spatial reasoning benchmark performance.

I. INTRODUCTION

  • VLMs remain limited in 3D geometry and spatial relations because pre-training provides little incentive to integrate multiple views into a coherent scene representation.View-conditioned priors also leave the language model to infer coherent scene structure from partial cues under sparse supervision.Source: §I · p. 1
  • Spa3R uses self-supervised Predictive Spatial Field Modeling to predict novel-view feature fields and learn unified, view-invariant scene geometry and layout.The representation is learned from visual observations without language supervision during pre-training.Source: §I · pp. 1–2
  • Spa3-VLM integrates the pre-trained Spa3R Encoder through a lightweight Residual Cross-Attention Adapter, letting native visual features query unified spatial context.The integration retains the base model’s pre-trained knowledge.Source: §I · pp. 1–2
  • 58.6%: Spa3-VLM’s average VSI-Bench score; it consistently improves over Qwen2.5-VL across three additional spatial reasoning benchmarks.The paper also reports competitive results against existing spatial models and consistent gains over matched baselines in controlled ablations.Source: §I · p. 2

II. RELATED WORK

Prior work develops generalizable 3D reconstruction and geometry priors, while 3D VLM approaches use explicit 3D data or partial view-conditioned cues. Spa3R instead learns a unified, view-invariant spatial representation for downstream reasoning.

A. 3D Reconstruction and Scene Representation

  • Generalizable reconstruction progressed from scene-specific neural fields such as NeRF and CT-NeRF to conditional fields such as pixelNeRF and MVSNeRF.Related directions include feed-forward 3D Gaussian Splatting, occupancy prediction, semantic scene completion, and self-supervised part-aware representations –.Source: A. 3D Reconstruction and Scene… · p. 2
  • LVSM and variants [27], [28] use Transformer scaling to infer geometry from 2D data, whereas Spa3R predicts spatially grounded feature fields for representation learning.Spa3R’s latent represents geometry and semantic relationships in a unified, view-invariant space for downstream spatial reasoning.Source: A. 3D Reconstruction and Scene… · p. 2

B. Vision-Language Models for 3D Reasoning

  • Explicit 3D grounding and reasoning methods use object-centric data, point clouds, depth, or preprocessed 3D representations, which can restrict scalability and applicability.Other approaches include viewpoint-aware grounding, 3D QA and scene graphs –, video BEV reconstruction [37], and spatial-data fine-tuning [38], [39].Source: B. Vision-Language Models for 3D… · p. 2
  • Geometry-prior VLMs typically expose partial, view-conditioned cues, leaving language models to integrate them; PSFM learns coherent scene-level context through self-supervised prediction.The unified representation is view-invariant.Source: B. Vision-Language Models for 3D… · p. 2

III. METHOD

PSFM models a scene as a spatial feature field and learns a compact latent from sparse context views by predicting aligned target-view features. Its bottleneck encourages view-invariant geometry and spatial relations, and the framework uses the resulting representation for downstream VLM reasoning.

III. METHOD

  • For downstream spatial reasoning, the pre-trained Spa3R Encoder supplies a latent that a lightweight adapter fuses with native VLM visual features through cross-attention.The overview depicts the encoder, novel-view decoder, and VLM integration stages.Source: §III · p. 2, Figure 2 · p. 3

A. Predictive Spatial Field Modeling

  • PSFM represents a 3D scene as a continuous feature field mapping viewpoints to view-centric feature maps.Each viewpoint is parameterized by an invertible lifted world-to-image projection matrix constructed from camera intrinsics and extrinsics.Source: A. Predictive Spatial Field Modeling · pp. 2–3
  • The Spa3R Encoder maps sparse context views to a unified latent, and the decoder predicts a target feature map at a queried camera projection conditioned on that latent.The context views are represented as projection-feature pairs.Source: A. Predictive Spatial Field Modeling · p. 3, Figure 2 · p. 3
  • Training partitions sampled scene views into context and target sets, then minimizes distance between predicted target features and their aligned targets.The architectural bottleneck encourages the latent to retain view-invariant geometry and spatial relations rather than view-specific appearance.Source: A. Predictive Spatial Field Modeling · p. 3

B. Learning a Unified Spatial Representation

Spa3R builds a canonical, view-aligned spatial feature field, compresses context features into a latent representation, and predicts target-view features from that representation and the queried camera. Separate geometric and semantic heads train the predictions against aligned target features.

B. Learning a Unified Spatial Representation

  • The Asymmetric View Aggregator aligns context and target features while preventing target-view information from leaking into context features.Target queries can attend to both context and target views, supporting construction of geometrically aligned target features; the frozen VGGT camera head predicts target camera parameters.Source: B. Learning a Unified Spatial… · p. 3
  • The Spa3R Encoder compresses context features into a compact latent representation, which the decoder conditions on with the target projection to synthesize target features.The decoder uses camera-space ray directions and relative 3D positional encoding to represent the target viewpoint; camera-associated tokens use PRoPE.Source: B. Learning a Unified Spatial… · p. 4
  • Separate geometric and semantic prediction heads are optimized against aligned target features, with losses computed per target patch and averaged over views and patches.The encoder, decoder, and prediction heads are trained jointly while VGGT, its camera head, and DINOv3 remain frozen; the two loss weights are equal.Source: B. Learning a Unified Spatial… · p. 4

C. Grounding Reasoning in Spatial Context

Spa3-VLM integrates the frozen Spa3R Encoder into Qwen2.5-VL through a residual cross-attention adapter. Native visual features query the spatial latent, and the resulting context is fused before language reasoning.

C. Grounding Reasoning in Spatial Context

  • The adapter fuses Spa3R’s unified spatial latent with Qwen2.5-VL’s native visual features using cross-attention and residual addition.Native visual features serve as queries over projected spatial latents; the normalized cross-attention output passes through a gated MLP before residual fusion.Source: C. Grounding Reasoning in Spatial… · p. 4
  • The Spa3R Encoder and VLM vision encoder remain frozen while the adapter and language model are fine-tuned, retaining base-model knowledge while grounding reasoning in global spatial context.The adapter’s MLP output projection is zero-initialized, initially preserving native visual features.Source: C. Grounding Reasoning in Spatial… · p. 4

IV. EXPERIMENTS

The experiments section introduces datasets and evaluation benchmarks, then implementation details, before presenting results and later ablations and qualitative analyses.

IV. EXPERIMENTS

  • The section first describes datasets and evaluation benchmarks, followed by Spa3R pre-training and Spa3-VLM instruction-tuning details.These are covered in Sections IV-A and IV-B, respectively.Source: §IV · p. 4
  • Main experimental results follow the setup and implementation details.The main results are reported in Section IV-C.Source: §IV · p. 4
  • Ablations and qualitative analyses are reported after the main results.They appear in Sections IV-E and IV-F.Source: §IV · p. 4

A. Setup

Spa3R pre-training uses real-world multi-view indoor RGB sequences, while evaluation spans video and image-based spatial reasoning benchmarks. The implementation describes pre-training and instruction-tuning data, models, and training procedures.

A. Setup

  • Spa3R is pre-trained on ScanNet and ScanNet++ training splits using RGB frames, without depth or ground-truth camera poses as inputs.Validation and test scenes from these datasets are excluded from optimization.Source: A. Setup · pp. 4–5
  • Evaluation covers VSI-Bench video questions and complementary CV-Bench, SPAR-Bench, and ViewSpatial-Bench formulations.VSI-Bench contains over 5,000 question-answer pairs from 288 indoor scene videos; accuracy is reported for multiple-choice tasks and mean relative accuracy for numerical tasks.Source: A. Setup · p. 5, Table I · p. 5, Table II · p. 5
  • Instruction tuning uses VSI-590K for VSI-Bench and a composite of SPAR-234K, LLaVA-Hound, and VLM3R for image-based benchmarks.The image-based composite follows prior work.Source: A. Setup · p. 5

B. Implementation Details

  • Spa3R uses frozen VGGT and DINOv3 feature targets, while Spa3-VLM fine-tunes an adapter and language model with its Spa3R Encoder and VLM vision encoder frozen.Spa3R pre-training samples 4–12 views and trains for 80K steps; instruction tuning uses Qwen2.5-VL-3B for one epoch and samples 4–8 frames per video.Source: B. Implementation Details · pp. 5–6

C. Main Results

Spa3-VLM leads or remains competitive across the reported spatial reasoning benchmarks, while Spa3R’s novel-view predictions recover geometric and semantic features. Ablations favor combined geometric and semantic targets, unified spatial features, residual cross-attention, a 50% target-view split, and PRoPE camera conditioning.

C. Main Results

  • 58.6% on VSI-Bench is 1.3 points above Cambrian-S-3B (57.3%); Spa3-VLM scores 75.6% on CV-Bench, 58.4% on SPAR-Bench, and 43.9% on ViewSpatial-Bench, 0.3 points below SpatialLadder-3B.On CV-Bench and SPAR-Bench, these results exceed the best previously reported results listed in Table II by 1.8 and 20.8 points, respectively.Source: C. Main Results · p. 6

D. Novel-View Spatial Field Prediction

  • With eight context views, Spa3R predicts novel-view fields with cosine similarities of 0.8806 for VGGT features and 0.7106 for DINOv3 features.The evaluation uses a dense novel-view trajectory; the results indicate recovery of both geometric and semantic targets beyond observed views.Source: D. Novel-View Spatial Field Prediction · p. 6, Table III · p. 6
  • AbsRel is 0.2492, RMSE is 0.5734 m, and δ1 accuracy is 0.6434; aggregated depth yields Chamfer-L1 of 0.1583 m and F-scores of 0.2541 at 5 cm and 0.5001 at 10 cm.The higher 10 cm F-score indicates more reliable recovery of coarse scene structure than fine surface detail; the encoder observes only context views.Source: D. Novel-View Spatial Field Prediction · p. 6

E. Ablation Studies

  • Combining VGGT and DINOv3 targets yields the highest average score, 58.6%, compared with 57.5% for VGGT alone, 56.7% for DINOv3 alone, and 51.9% for CLIP alone.The authors suggest the feature spaces provide complementary signals for geometric structure and high-level semantic context.Source: E. Ablation Studies · p. 6, Table IV · p. 6
  • Spa3R reaches 58.6%, compared with 55.1% for direct view-conditioned VGGT features and 50.9% for the matched Qwen2.5-VL-3B baseline.The comparison isolates the spatial branch under matched downstream training and supports the utility of compressing multi-view features through PSFM.Source: E. Ablation Studies · pp. 6–7, Table V · p. 7
  • Residual cross-attention reaches 58.6%, whereas sequence appending reaches 51.1%, only 0.2 points above the matched baseline.The authors hypothesize that sequence appending underutilizes spatial tokens, while cross-attention retrieves relevant spatial context before the residual update.Source: E. Ablation Studies · p. 7, Table VI · p. 7
  • A 50% target-view split gives the highest average score at 58.6%, versus 57.5% at 25% and 58.1% at 75%.The authors relate this pattern to a trade-off between context coverage and prediction difficulty.Source: E. Ablation Studies · p. 7, Table VII · p. 7
  • PRoPE reaches 58.6%, improving on the 57.6% score with Plücker coordinates for target-view conditioning.PRoPE encodes camera intrinsics, extrinsics, and image-plane patch positions through relative attention operators; Plücker embedding is injected at token level.Source: E. Ablation Studies · p. 7, Table VIII · p. 9

F. Qualitative Analysis

The qualitative analysis projects predicted features onto VGGT’s reconstructed point-cloud geometry and compares predictions with target features across views. The feature layouts are spatially coherent, and a detached depth probe recovers structure aligned with the reconstruction, including in highlighted occluded or unobserved regions.

F. Qualitative Analysis

  • Predicted features show spatial organization consistent with target features across views, with continuous, coherent region-level layouts in PCA projections.The comparison projects original RGB images and predicted feature maps onto canonical point-cloud geometry reconstructed by VGGT.Source: F. Qualitative Analysis · p. 9
  • A detached depth probe recovers scene structure visually aligned with the VGGT reconstruction, without backpropagating probe gradients to Spa3R.The probe is an MLP trained to regress depth maps from predicted features and serves only as a diagnostic.Source: F. Qualitative Analysis · p. 9
  • Predictions in highlighted occluded or unobserved regions suggest the decoder uses information aggregated across context views.The qualitative examples are complemented by quantitative comparison with sensor-derived geometry in Section IV-D.Source: F. Qualitative Analysis · p. 9

V. CONCLUSION

Spa3R learns a unified, view-invariant representation from unposed multi-view RGB observations, and Spa3-VLM integrates it through a residual cross-attention adapter. The study reports gains on VSI-Bench while remaining scoped to static indoor scenes and inherited geometry-model priors.

V. CONCLUSION

  • Spa3-VLM reaches 58.6% average on VSI-Bench and improves over the matched fine-tuning baseline by 7.7 points.Controlled ablations show gains from joint geometric–semantic targets, PSFM, and cross-attention integration.Source: §V · p. 9
  • The study focuses on static indoor scenes and inherits 3D priors and pose estimates from a frozen geometry foundation model.It does not yet model calibrated uncertainty, establish formal disentanglement, or demonstrate complete recovery of unobserved scene geometry.Source: §V · p. 9
  • Code is available at the paper’s cited repository.https://github.com/hustvl/Spa3RSource: §V · p. 9

APPENDIX ADDITIONAL QUALITATIVE RESULTS AND RESOURCE REQUIREMENTS

The appendix reports Spa3-VLM’s parameter and memory requirements and illustrates selected VSI-Bench predictions. The examples show frame aggregation for repeated-instance counting and scale estimation.

A. Parameter Counts and Inference Memory

B. Qualitative Spa3-VLM Predictions

  • On selected VSI-Bench examples, Spa3-VLM aggregates observations across frames to count repeated instances and estimate object and room scale.Fig. 5 shows six representative frames from each 32-frame evaluation input alongside the task, question, answer, and ground truth.Source: B. Qualitative Spa3-VLM Predictions · p. 9, Figure 5 · p. 10
Loading 2602.21186v2…