Reinforcing Dual-Path Reasoning in Spatial Vision Language Models

Yatai Ji; An-Chieh Cheng; Yang Fu; Yukang Chen; Han Zhang; Zhaojing Yang; Wei Huang; Ka Chun Cheung; Song Han; Vidya Nariyambut Murali; Pavlo Molchanov; Jan Kautz; Simon See; Hongxu Yin; Ping Luo; Sifei Liu

Source-linked AI summary

Reinforcing Dual-Path Reasoning in Spatial Vision Language Models

Yatai Ji, An-Chieh Cheng, Yang Fu, Yukang Chen, Han Zhang, Zhaojing Yang, Wei Huang, Ka Chun Cheung, Song Han, Vidya Nariyambut Murali, Pavlo Molchanov, Jan Kautz, Simon See, Hongxu Yin, Ping Luo, Sifei Liu

arXiv:2606.17539v1Jun 16, 2026cs.CVcs.AI

TL;DR

Complex spatial queries require either linguistic multi-step deduction or explicit 3D grounding, but spatial VLMs need a unified way to support both. SR-REAL combines Language-Only Reasoning and Detect-Then-Reason with supervised initialization and reinforcement learning, outperforming spatial VLM baselines across region-grounded, global, and out-of-distribution benchmarks.

Problem
Complex spatial reasoning requires multi-step inference, and different queries call for linguistic deduction or explicit 3D grounding.
Method
SR-REAL jointly trains linguistic and geometry-aware reasoning paths using cold-start supervision, region-to-3D grounding, and reinforcement learning.
Results
SR-REAL consistently outperforms spatial understanding baselines across region-grounded, global, and out-of-distribution benchmarks, with DTR strongest on region-based tasks.
Takeaways & Limitations
A single jointly trained model can support both reasoning modes and generalize across datasets and domains without per-task tuning.
Takeaways & Limitations
SR-REAL requires depth maps and camera intrinsics/extrinsics at inference time, limiting use when geometric metadata is unavailable.

Abstract

from arXiv · show

Spatial VLMs have made substantial progress in geometric perception, yet complex spatial reasoning requiring multi-step inference over depth, distance, and scene relations remains challenging. Moreover, different spatial queries call for fundamentally different strategies: some are best addressed through purely linguistic, step-by-step deduction, while others require explicit 3D grounding before quantitative inference. We present Dual-Path Spatial Reasoning via Reinforcement Learning for Spatial VLMs (SR-REAL), a unified framework that equips a spatial VLM with two complementary reasoning paths: Language-Only Reasoning (LOR), which performs step-by-step linguistic deduction, and Detect-Then-Reason (DTR), which detects 3D geometric cues (e.g., centers or bounding boxes) via region tokens before explicit geometric inference. SR-REAL begins with a cold-start supervised fine-tuning stage that constructs LOR and DTR chain-of-thought supervision and exposes a region-to-3D interface, followed by RL that optimizes the policy model with accuracy and format rewards; for DTR, a discrete center-based detection reward further refines geometric alignment. Across diverse spatial benchmarks, SR-REAL significantly outperforms spatial VLM baselines: (i) a single RL-trained model supports both reasoning paths, with DTR excelling in region-aware tasks through precise 3D localization and LOR enhancing general spatial reasoning; (ii) jointly training both paths fosters mutual reinforcement; (iii) high-quality, blended cold-start data is crucial for stable RL optimization; and (iv) the model generalizes across datasets and domains without per-task tuning, demonstrating positive transfer between LOR and DTR.

1 The University of Hong Kong · 2 NVIDIA 3 University of California, San Diego · 1. Introduction

SR-ReaL addresses persistent spatial limitations in vision-language models by combining linguistic deduction with explicit 3D grounding and reinforcement learning. Its evaluation shows versatile spatial reasoning, performance gains, and cross-domain generalization across region-grounded, global, and out-of-distribution benchmarks.

2 NVIDIA 3 University of California, San Diego: The paper is affiliated with the University of Hong Kong, NVIDIA, and the University of California, San Diego.The authors note that the work was completed during an NVIDIA internship.
1. Introduction: Large vision-language models remain limited in understanding 3D layout, depth, occlusion, and viewpoint-dependent spatial relationships.The introduction frames these weaknesses as persistent despite rapid progress in visual interpretation and reasoning.
1. Introduction: Complex spatial queries require multi-step inference over relations, global context, localized cues, or quantitative 3D comparisons, motivating distinct reasoning strategies.Some queries are best solved through purely linguistic step-by-step deduction, whereas others require explicit geometric grounding.
1. Introduction: SR-ReaL equips one spatial VLM with Language-Only Reasoning for linguistic deduction and Detect-Then-Reason for 3D cues before quantitative inference.DTR can use centers or oriented boxes, while LOR reasons directly over scene relations.
1. Introduction: Cold-start supervision constructs structured chain-of-thought datasets for both paths and exposes a region-to-3D interface through centers or bounding boxes.LOR traces derive answers from scene relations; DTR traces combine explicit geometry with subsequent reasoning steps.
1. Introduction: Reinforcement learning jointly optimizes both paths with DAPO-style GRPO, online filtering, accuracy and format rewards, and a discretized detection reward for DTR.Prompt guidance selects LOR or DTR, with DTR additionally requiring extracted region tokens and 3D predictions.
1. Introduction: SR-ReaL consistently outperforms spatial understanding baselines across region-grounded, global, and out-of-distribution spatial benchmarks.The evaluation emphasizes versatility across localized references, whole-scene reasoning, and shifted imagery or question styles.
1. Introduction: The results show significant performance gains and strong cross-domain generalization, with region grounding, data quality, and RL supervision jointly shaping spatial reasoning.The findings position proper grounding and initialization as important for strengthening both linguistic and geometric reasoning.

2. Related Work

Related work progresses from object-centric spatial perception and 2D spatial question answering toward region prompting, depth cues, explicit 3D representations, and reinforcement-learning-based reasoning. In parallel, supervised chain-of-thought initialization and R1-style reinforcement learning have become general approaches for visual and multimodal reasoning.

Spatial VLMs: Early spatial VLMs model object-centric properties, including position, size, orientation, pairwise distance, and direction, while later systems scale 2D spatial QA and introduce region prompting with depth cues.The passage identifies SpatialVLM as scaling spatial QA in 2D and SpatialRGPT as introducing region prompting and depth cues.
RL for VLM Reasoning: VLM reasoning initially relied on supervised chain-of-thought learning, while rule-based reinforcement learning and R1-style GRPO expanded long-form reasoning to multimodal mathematical and visual tasks.DeepSeek-R1 demonstrates that rule-based RL alone can induce long-form reasoning without human-labelled trajectories; Vision-R1 and MM-Eureka adapt R1-style GRPO to multimodal tasks.
Spatial Reasoning VLMs: Recent spatial reasoning VLMs combine structured spatial supervision, explicit 3D cues, and reinforcement learning to strengthen reasoning in multi-view 3D scenes and explicit scene representations.3D-R1 applies R1-style training to multi-view 3D scenes with known camera poses and detailed textual grounding.

3. Methods

SR-REAL combines a 3D-aware spatial VLM interface with complementary language-only and geometry-grounded reasoning paths. Its cold-start data construction and blended reinforcement learning jointly teach structured spatial inference, region-to-3D grounding, and task-aligned output behavior.

Spatial VLM Architecture: The model extends SR-3D with interleaved text, image, and region tokens for single-view spatially grounded reasoning.It supports both single-image and multi-image settings.
Spatial VLM Architecture: A positional embedding formulation fuses 2D coordinates, depth, and camera intrinsics and extrinsics into visual features as 3D-aware positional bias.The pixel-wise 3D position map is encoded and added to corresponding visual embeddings.
Spatial VLM Architecture: Region prompts bridge 2D visual priors and 3D localization, enabling DTR to predict geometric cues for explicit coordinate-based reasoning.Region tokens interact with vision features through cross-attention, while direct language-to-3D grounding performs substantially worse.
Cold-Start Data Construction: Cold-start supervision constructs concise step-by-step LOR traces and DTR chains using object centers or 3D bounding boxes for explicit spatial computation.For annotated 2D regions, 3D locations are obtained by projecting EmbodiedScan annotations into the image plane with known camera parameters.
Cold-Start Data Construction: The method augments template-style questions with navigation, interaction, and layout tasks, then filters generated traces for answer agreement and logical consistency.Sampled batches are also checked for annotation formatting and region–object alignment.
Reinforcement Learning: GRPO reinforcement learning uses format and accuracy rewards, while DTR additionally receives discretized rewards based on predicted-center distances to 3D ground truth.Required formats are think–answer for LOR and detect–think–answer for DTR.

4. Experiments

Experiments show that SR-REAL’s unified LOR/DTR model substantially improves standard spatial performance, generalizes under distribution shift, and benefits from joint training, explicit 3D detection, and blended cold-start supervision.

Performance on Standard Benchmarks: Ours-DTR reaches 61.9 average accuracy on SPAR-Bench versus 33.4 for SR-3D, a +28.5 improvement.On Single-View, DTR also delivers a +5.0 gain, supporting the value of explicit 3D coordinates for precise calculation.
Performance on Standard Benchmarks: DTR remains highly competitive on Spatial Relation and Imagination tasks, often exceeding LOR through explicit object detection.The model can solve spatial relations either through linguistic deduction or coordinate comparison, while DTR is particularly advantageous for multi-view metric distance estimation.
Generalization on OOD Benchmarks: Direct inference preserves strong OOD performance, including 88.1% on CVBench, despite lower performance when applying chain-of-thought reasoning to shifted distributions.The evaluated OOD benchmarks are BLINK, RealWorldQA, and CVBench, covering unseen imagery and question styles.
Impact of Joint LOR & DTR Training: Joint LOR and DTR training consistently outperforms single-mode baselines across metrics while supporting both inference modes in one model.For example, LOR inference improves from 58.0 to 58.7 on SPAR-Bench.
Ablation on DTR Designs: Removing the discrete DTR detection reward increases 3D localization error from 0.45 to 0.78, reducing downstream reasoning performance.The ablation uses DTR inference on a 400-instance SPAR-Det validation set and confirms the importance of the detection reward and region-to-3D interface.
Impact of Cold-Start SFT and RL Phases: Cold-start SFT improves SPAR-Bench but can hurt unseen-task performance, whereas adding RL improves reasoning and generalization; blended data progressively strengthens EmbSpatial and OOD behavior.RL-only training improves metrics but often produces illogical or answer-inconsistent CoTs, while combining SPAR, CA-1M, general-purpose multimodal, and region-related data improves robustness.

5. Conclusion

SR-REAL improves spatial reasoning in Vision–Language Models through reinforcement learning that jointly supports complementary linguistic and geometric reasoning paths. Its region-to-3D grounding and discrete detection rewards contribute to strong benchmark performance and cross-domain generalization.

5. Conclusion: SR-REAL integrates complementary Language-Only Reasoning and Detect-Then-Reason paths for linguistic and geometric spatial reasoning.The framework uses region-to-3D grounding and discrete detection rewards to improve spatial accuracy.
5. Conclusion: SR-REAL demonstrates reinforcement learning’s effectiveness for enhancing spatial reasoning in Vision–Language Models.The approach improves both linguistic and geometric reasoning.
5. Conclusion: SR-REAL excels across diverse spatial benchmarks and achieves strong cross-domain generalization.These results indicate performance across spatial tasks and domains without requiring task-specific adaptation.

A. Implementation Details

SR-REAL is trained in two stages: cold-start supervised fine-tuning on approximately 1M blended samples, followed by GRPO-based reinforcement learning on approximately 200k spatial samples. Both stages use cosine-decay learning rates and run on 32 NVIDIA A100 GPUs.

Stage 1: Cold-Start SFT: Cold-start SFT fine-tunes the SR-3D base model for 2 epochs on approximately 1M blended samples, using a 5 × 10−6 learning rate and batch size 128.The learning rate follows cosine decay scheduling.
Stage 2: Reinforcement Learning: GRPO-based reinforcement learning runs for 200 steps on approximately 200k spatial samples, with rollout batch size 512 and learning rate 1 × 10−6.The learning rate uses cosine decay scheduling.

B. Preliminary knowledge of GRPO

GRPO optimizes policies without a value-function critic by estimating advantages from relative rewards among responses sampled for the same query. Its clipped surrogate objective and direct KL penalty promote stable updates while constraining deviation from a reference policy.

Critic-free policy optimization: GRPO eliminates the value-function critic used in methods such as PPO, reducing memory usage and computational overhead during reinforcement-learning training.It is the core reinforcement-learning algorithm used in this work.
Group-relative advantage estimation: For each input query, GRPO compares rewards across a group of sampled responses to estimate relative advantages.The responses are generated from the same input query, and reward normalization provides the comparison baseline.
Group-relative advantage estimation: GRPO normalizes each response’s reward against group statistics, reinforcing outputs that outperform the group average.The normalized advantage is applied at the token level for each response.
Stable policy updates: Its surrogate objective combines importance-sampling ratios, PPO-style clipping, and a direct KL-divergence penalty for stable policy updates.The KL term limits excessive deviation of the trained policy from the reference policy at the token level.

C. CoT Data Construction

The CoT instruction data combines balanced LOR and DTR samples for fundamental spatial perception with complex-scene reasoning samples. Prompt-controlled inference distinguishes linguistic deduction from detection-first 3D reasoning.

CoT Data Composition: The dataset contains approximately 10k LOR and 10k DTR samples sourced from SPAR, plus 20k complex spatial-task samples from CA-1M and NuScenes.The complex subset targets navigation, object manipulation, and spatial planning, while LOR and DTR cover fundamental spatial tasks.
CoT Data Composition: The LOR and DTR subsets cover depth, distance, object relations, spatial imagination, and position matching, providing basic geometric coverage.Together with the complex subset, the composition balances geometric reasoning and scene-level inference.
Prompts: Input prompts select the inference pathway, with DTR requiring 3D detection information inside a <detect> block before reasoning.LOR and DTR use distinct system prompts to specify their reasoning procedures.

D. More Results

Additional analyses show that DTR leads detailed SPAR-Bench performance through explicit 3D grounding, while LOR remains competitive on relational tasks. They also identify RL enhancement mechanisms and cold-start grounding supervision as important factors for training and localization quality.

Detailed results on SPAR-Bench: Ours-DTR achieves the best overall SPAR-Bench average, 61.9, and leads most depth- and distance-related dimensions.Ours-LOR remains competitive on relation and spatial imagination tasks, while both modes substantially outperform SR-3D across nearly all dimensions.
Training mechanisms during RL: The RL analysis examines online filtering and cosine-decayed KL coefficients as mechanisms for improving GRPO-based spatial-task training.KL coefficient decay constrains the updated policy to remain close to the reference model by annealing the KL term’s coefficient.
Grounding data: Removing auxiliary grounding supervision causes inaccurate localization and performance degradation across all DTR benchmarks, especially EmbSpatial and CVBench.The decline is relatively minor on SPAR-Bench because its cold-start dataset already contains DTR chain-of-thought samples derived from grounding data.

E. More Visualization

Additional qualitative examples show that the model performs accurately with both LOR and DTR, while DTR resolves localization-sensitive distance estimation more reliably. The supplemental failure analysis emphasizes that diverse cold-start data stabilizes out-of-domain reasoning and that omitting cold-start can cause reasoning–answer inconsistency.

Qualitative examples: Qualitative examples demonstrate accurate results under both the LOR and DTR reasoning pathways.Figure S2 presents additional examples spanning both paradigms.
Qualitative examples: DTR accurately computes distance through precise spatial coordinate detection when LOR misestimates it because of inaccurate object localization.The comparison comes from the first distance-measurement example.
Cold-start failure analysis: Diverse cold-start data facilitates more stable reasoning on out-of-domain questions, whereas direct RL without cold-start frequently produces reasoning–answer inconsistency.The failure analysis reports cases where the model selects the correct option but produces inconsistent reasoning and answers.

F. Limitations

SR-REAL remains constrained by its dependence on geometric metadata and region annotations at inference time. These requirements limit deployment on unconstrained images and prevent DTR from operating when query-specific regions are unavailable.

Geometric metadata dependence: SR-REAL requires depth maps and camera intrinsics/extrinsics at inference, limiting applicability to images without such geometric metadata.This includes unconstrained in-the-wild images captured without depth sensors.
Region annotation dependence: DTR depends on region tokens from 2D bounding boxes or masks, so it cannot operate without region annotations and falls back to LOR.This limitation arises on global benchmarks such as SAT.
Open challenge: Automatically generating reliable region proposals for arbitrary queries remains an open challenge.The limitation is especially relevant when required region annotations are absent.

G. Broader Impact

The paper highlights broad benefits of improved spatial reasoning for embodied and scientific applications, while warning that spatial VLMs may also enable privacy-sensitive misuse and concentrate capability development.

Positive impacts: Enhanced spatial understanding can benefit embodied AI, assistive robotics, autonomous driving, augmented reality, medical imaging, and remote sensing.These applications may improve accessibility, quality of life, and interpretation of 3D spatial data.
Potential negative impacts: Spatial reasoning improvements could be misused for surveillance or automated physical-space monitoring without appropriate consent.The paper advocates responsible deployment and clear usage guidelines in sensitive or privacy-sensitive environments.
Potential negative impacts: Reliance on proprietary data-generation pipelines may concentrate capability development among well-resourced organizations.