Source-linked AI summary
SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation
Jinsheng Quan, Jianhua Li, Siyi Xie, Xuanke Shi, Kewang Deng, Zukai Chen, Feifei Shao, Lei Yang, Quan Wang, Yawei Luo
TL;DR
Existing systems often separate geometric perception, correspondence, and spatial reasoning, limiting unified learning across complementary scene representations. SPARGen casts these capabilities as instruction-conditioned generation tasks in one native multimodal model, achieving competitive performance across visual-geometry and optical-flow benchmarks and the strongest spatial-reasoning performance among compared models.
Problem
Existing approaches typically separate dense geometric perception, correspondence, and spatial reasoning, limiting unified use of complementary supervision from the same physical scene.
Method
SPARGen formulates 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation using native token-sequence and image-aligned field outputs.
Results
SPARGen remains competitive with specialized geometry and flow models and achieves the strongest spatial-reasoning performance among compared models across heterogeneous benchmarks.
Takeaways & Limitations
The results demonstrate the feasibility of supporting spatial perception and reasoning within a single native multimodal model, with preliminary evidence of complementary supervision effects.
Takeaways & Limitations
SPARGen’s frozen VAE spatial compression creates a bottleneck for geometric edges and high-precision physical quantities.
Abstract
from arXiv · showhide
Spatial perception and reasoning from visual observations require recovering geometric structure, establishing correspondences, and understanding spatial relations. Existing approaches typically address these capabilities separately using task-specific architectures or external geometric modules, limiting knowledge transfer among complementary representations of the same physical scene. We introduce SPARGen, a unified multimodal framework that casts 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation tasks. SPARGen serializes compact structured and linguistic outputs as token sequences while generating dense geometric fields in image-aligned forms, enabling spatial supervision to jointly shape shared representations within a native multimodal generative model. Experiments across benchmarks for 3D reconstruction, correspondence, and spatial reasoning show that SPARGen achieves competitive performance across heterogeneous spatial tasks within a single native multimodal generative framework.
1 Introduction
SPARGen unifies 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation tasks within one native multimodal framework. It combines image-aligned generation of dense geometric fields with token generation of structured outputs and answers through a shared MoT backbone.
- Motivation: Unifying geometry and reasoning can enable knowledge transfer because depth, camera motion, point maps, and optical flow provide complementary supervision for spatial understanding.These signals characterize scene structure and variation, while spatial question answering abstracts geometric evidence into semantic relations.
- Problem: Existing methods typically separate accurate dense geometry prediction from instruction-following spatial reasoning, limiting shared supervision across these capabilities.Feed-forward geometry models generally lack spatial question answering, whereas multimodal models receive limited dense geometry and correspondence supervision.
- SPARGen: SPARGen formulates 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation tasks for constructing spatial representations and answering conditioned queries.Dense outputs are represented as image-aligned fields, while compact structured outputs and spatial answers are serialized as token sequences.
- Architecture: A shared Mixture-of-Transformer-Experts backbone integrates native image-generation and autoregressive pathways for heterogeneous spatial outputs without task-specific regression heads or external geometric modules.Dense spatial fields use the image-generation pathway, while camera poses and spatial answers use the autoregressive pathway.
- Novelty: Unlike approaches specialized for dense geometry or language-based reasoning, SPARGen supports both capabilities within a single instruction-conditioned multimodal architecture.Its design allows geometric, correspondence, and semantic supervision to jointly shape multimodal representations.
2 Related Work
Related work spans feed-forward 3D reconstruction and dense correspondence, geometry-aware spatial reasoning, and unified multimodal understanding and generation. These approaches increasingly use learned geometric representations, spatial supervision, and shared multimodal architectures.
- 3D Reconstruction and Dense Correspondence: Feed-forward geometry models broaden reconstruction beyond traditional camera-motion and scene-structure recovery.DUSt3R formulates uncalibrated stereo reconstruction as pointmap regression, while VGGT jointly predicts camera poses, depth, point maps, and tracks.
- 3D Reconstruction and Dense Correspondence: Learning-based methods increasingly replace individual correspondence-pipeline stages with direct dense prediction.LoFTR performs detector-free transformer matching, while DKM and RoMa directly predict dense correspondence fields.
- Geometry-Aware Spatial Reasoning: Geometry-aware spatial reasoning methods augment vision-language models with spatial question-answering data, region-level supervision, depth, or reconstruction priors.SpatialVLM uses large-scale spatial question-answering data, SpatialRGPT combines region supervision with a plug-in depth representation, and VLM-3R derives implicit spatial and camera tokens from geometry.
- Unified Multimodal Understanding and Generation: Unified multimodal models seek shared systems for understanding and generation through interleaved tokens, combined modeling pathways, or shared backbones.Chameleon uses early-fusion interleaved discrete image-text tokens; Show-o combines autoregressive language modeling with discrete diffusion; Janus separates visual encoding pathways while sharing an autoregressive backbone; Bagel scales unified pretraining.
3 Method
SPARGen unifies spatial tasks by conditioning a native multimodal generative model on images and instructions, then generating either serialized token sequences or image-aligned dense fields. Its joint training combines autoregressive sequence generation with rectified-flow matching for dense fields.
- Problem formulation and representations: SPARGen predicts spatial targets for instruction-conditioned tasks in Tseq ∪ Tfield, using token sequences for structured outputs and dense fields for geometric outputs.This unified formulation avoids separate task-specific modules while preserving each output type’s structural properties.
- Unified architecture: The MoT-based architecture encodes images with a ViT, instructions as text, and lets both streams interact through joint multimodal self-attention.Sequence targets use autoregressive decoding, whereas dense fields are generated in VAE latent space through rectified flow.
- Unified architecture: For dense-field generation, conditioning concatenates text, ViT image, and frozen-VAE image tokens, while sequence generation uses text and ViT-derived context.The dense target is VAE-encoded and generated by predicting the velocity field along a linear path from clean latent to Gaussian noise.
- Sparse and structured outputs: Sparse geometric quantities, textual answers, and structured camera poses are serialized into canonical token sequences for deterministic autoregressive decoding.Camera pose scalars are quantized at a resolution of 10^-3 and emitted in a fixed order.
- Dense outputs: Dense outputs are transformed into image-aligned fields matching RGB-image shape, including normalized relative depth, shared-coordinate point maps, and normalized optical flow.Point-map normalization shares center and scale across a sequence to preserve relative geometry across views.
- Training objectives: SPARGen jointly trains on mixed tasks with target-position cross-entropy for serialized sequences and rectified-flow matching for dense fields, weighted by λ in the overall objective.Each example activates the loss corresponding to its target representation.
4 Experiments
SPARGen is evaluated across visual geometry, spatial reasoning, and optical flow, where it improves over unified baselines, leads spatial reasoning, and remains competitive with specialized models. Ablations further indicate complementary benefits among geometry, correspondence, and reasoning supervision.
- Visual Geometry: SPARGen improves over G2VLM on most visual-geometry metrics, including depth estimation, 7Scenes reconstruction, and all three CO3D v2 camera-pose metrics.VGGT remains stronger on several reconstruction metrics, but SPARGen substantially narrows the gap with specialized geometry models.
- Spatial Understanding and Reasoning: SPARGen achieves the highest average score on all four spatial-reasoning benchmarks and ranks first in 13 of 15 reported categories among compared non-proprietary models.Its average-score gains over the strongest competing result are 9.85 points on MindCube, 1.97 on OmniSpatial, 4.99 on OST, and 24.71 on SPAR.
- Optical Flow Estimation: 4.09 EPE and 13.34 F1-all are SPARGen’s zero-shot KITTI optical-flow results, outperforming the compared methods on both metrics.These results support effective dense-correspondence estimation through SPARGen’s native generation pathway.
- Qualitative Results: Qualitatively, SPARGen recovers coherent global 3D layouts, captures dominant foreground-vehicle motion, estimates static backgrounds consistently, and improves residual flow misalignments through refinement.The reported examples cover point-map reconstruction, optical flow, and depth estimation.
- Ablation Study: Removing geometry supervision degrades optical flow and spatial reasoning, removing optical flow supervision worsens 7Scenes reconstruction errors and average SPAR score, and removing reasoning supervision slightly degrades reconstruction.The ablations are designed to test whether complementary supervision types transfer structural, correspondence, and semantic information across tasks.
5 Conclusion
SPARGen unifies spatial perception and reasoning as instruction-conditioned generation within a shared multimodal backbone, combining tokenized structured outputs with image-aligned geometric fields. Its reliance on a frozen VAE introduces a bottleneck for geometric edges and high-precision physical quantities.
- Framework: SPARGen formulates spatial perception and reasoning as instruction-conditioned generation in a unified multimodal framework.The framework uses a shared MoT backbone without external modules.
- Output representations: Compact structured and linguistic outputs are serialized as token sequences, while dense geometric and correspondence fields use image-aligned outputs.This design leverages the backbone’s native autoregressive and rectified-flow pathways.
- Evaluation: Across visual-geometry, optical-flow, and spatial-reasoning benchmarks, SPARGen evaluates the unified framework across heterogeneous spatial tasks.The supplied passage identifies these benchmark categories but does not report numerical results.
- Limitations: A frozen VAE’s spatial compression bottlenecks geometric edges and high-precision physical quantities.Although the frozen VAE enables reuse of pretrained multimodal generative pathways, its compression limits geometric fidelity.