Source-linked AI summary
Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning
Yuhong Liu, Beichen Zhang, Yuhang Zang, Yuhang Cao, Long Xing, Xiaoyi Dong, Haodong Duan, Dahua Lin, Jiaqi Wang
TL;DR
LVLM spatial understanding remains weaker than desired, while existing SFT and RLVR pipelines rely on costly supervision, specialized tools, or constrained environments. Spatial-SSRL turns intrinsic structure in ordinary RGB and RGB-D images into verifiable RL supervision through five self-supervised spatial tasks. Across seven spatial benchmarks, it improves accuracy for both 3B and 7B models while preserving general visual capabilities.
Problem
LVLM spatial understanding remains substantially below human performance, while existing SFT and RLVR pipelines depend on costly supervision, specialized tools, or constrained environments.
Method
Spatial-SSRL uses five self-supervised tasks derived from intrinsic structure in ordinary RGB and RGB-D images as deterministically verifiable rewards for LVLM reinforcement learning.
Results
4.63% average accuracy gain for 3B models and 3.89% for 7B models was achieved across seven spatial understanding benchmarks, with general visual capabilities preserved.
Takeaways & Limitations
Intrinsic image supervision provides a cost-effective, scalable, and practical route to stronger LVLM spatial intelligence through RLVR.
Takeaways & Limitations
Relative 3D position prediction assumes object orientation is parallel to the ground plane and ignores vertical displacement because image y-coordinates do not reliably encode real-world height.
Abstract
from arXiv · showhide
Spatial understanding remains a weakness of Large Vision-Language Models (LVLMs). Existing supervised fine-tuning (SFT) and recent reinforcement learning with verifiable rewards (RLVR) pipelines depend on costly supervision, specialized tools, or constrained environments that limit scale. We introduce Spatial-SSRL, a self-supervised RL paradigm that derives verifiable signals directly from ordinary RGB or RGB-D images. Spatial-SSRL automatically formulates five pretext tasks that capture 2D and 3D spatial structure: shuffled patch reordering, flipped patch recognition, cropped patch inpainting, regional depth ordering, and relative 3D position prediction. These tasks provide ground-truth answers that are easy to verify and require no human or LVLM annotation. Training on our tasks substantially improves spatial reasoning while preserving general visual capabilities. On seven spatial understanding benchmarks in both image and video settings, Spatial-SSRL delivers average accuracy gains of 4.63% (3B) and 3.89% (7B) over the Qwen2.5-VL baselines. Our results show that simple, intrinsic supervision enables RLVR at scale and provides a practical route to stronger spatial intelligence in LVLMs.
1. Introduction
Spatial-SSRL addresses weak LVLM spatial understanding by converting intrinsic structure in ordinary RGB and RGB-D images into verifiable rewards for reinforcement learning. Across seven spatial benchmarks, it improves spatial performance while preserving general visual capabilities.
- Spatial understanding remains substantially below human performance despite near-saturated LVLM results on several general vision and reasoning tasks.Spatial reasoning over depth, distance, azimuth, and relative object positions supports reconstruction of 3D environments and applications such as autonomous driving and robot manipulation.
- Existing SFT and RLVR approaches are limited by memorization, inherited perception errors, costly curation, specialized environments, and substantial pipeline engineering.These constraints restrict the scalability and domain coverage of spatial training.
- The framework derives deterministically verifiable supervision from intrinsic consistency signals in ordinary 2D or RGB-D images and uses them as RL rewards.This repurposes self-supervised objectives from visual representation learning to directly optimize LVLM behavior.
- Five self-supervised task families cover depth-free 2D structure and depth-based relative depth or 3D relation consistency.The framework is organized around tasks involving relative position, permutation ordering, cross-view correspondence, depth ranking, and 3D relations.
- Automated curation uses only raw RGB or RGB-D images, without human labels or auxiliary proprietary models, making the pipeline cost-effective and scalable.The pipeline is also described as reproducible end-to-end and extensible with new pretext tasks.
- Spatial-SSRL improves accuracy across seven spatial understanding benchmarks in both image and video settings, with no regression in general visual capabilities.The method also improves or preserves general VQA, multi-image understanding, and hallucination diagnostics.
2. Related Work
Prior spatial-understanding methods rely heavily on annotations, specialized tools, or simulated scenes, limiting cost, scalability, and real-world consistency. Spatial-SSRL instead scales verifiable supervision from intrinsic image structure through a tool-free, self-supervised RL paradigm.
- Tool-based approaches add computationally complex pipelines through detection, segmentation, and depth-estimation models, while simulation-based approaches can diverge from real-world scenarios.These two paradigms represent the dominant approaches for constructing large-scale spatial question-answer data.
- Spatial-SSRL provides a tool-free, real-world-consistent, cost-effective, and highly scalable alternative for spatial training data.Its supervision is derived from intrinsic image structure rather than manual annotations or specialized curation assets.
- Self-supervised learning obtains supervision from inherent data structure without manual labels, traditionally supporting visual representation learning through contrastive or task-based objectives.Spatial-SSRL extends this principle to verifiable reinforcement-learning rewards for LVLM spatial understanding.
- Compared with prior self-supervised post-training methods, Spatial-SSRL covers multiple 2D and RGB-D spatial tasks rather than focusing only on jigsaw objectives or 2D tasks.Jigsaw is one component of Spatial-SSRL, while its supervisory signals target spatial understanding across both 2D and RGB-D images.
- Existing spatial methods depend mainly on annotation-rich datasets and tool-heavy pipelines, limiting the quality and scale of RL training data.The specialized assets used for curation constrain how fully RL advantages can be exploited.
3. Spatial-SSRL
Spatial-SSRL combines automatically curated self-supervised spatial tasks with verifiable-reward reinforcement learning, using RGB and RGB-D image structure without human or LLM annotations. Its five complementary tasks target 2D layout, orientation, structural consistency, depth ordering, and 3D spatial relations.
- Task Design: Spatial-SSRL uses five complementary pretext tasks to train spatial understanding through automatically derived image-structure supervision.The tasks cover shuffled patch reordering, flipped patch recognition, cropped-patch inpainting, regional depth ordering, and relative 3D position prediction.
- Task Design: The framework separates depth-free RGB tasks for 2D structure from depth-based tasks for 3D scene understanding.Depth-free tasks emphasize layout, structural consistency, and texture-context correspondence, while depth-based tasks target ordinal depth and egocentric spatial relations.
- Depth-free Tasks: Shuffled patch reordering predicts the inverse permutation that restores the original image layout from randomly reordered patches.Optional masking prevents reliance on trivial edge matching and encourages global layout reasoning.
- Depth-free Tasks: Flipped patch recognition identifies both the location and direction of a vertically or horizontally flipped patch.The task uses a reconstructed image containing one randomly flipped patch and requires the answer tuple [t, d].
- Depth-free Tasks: Cropped patch inpainting selects the correct patch for a masked region using texture continuity, semantic context, and structural consistency.Visually similar rotated, interior, and exterior distractors are used to discourage low-level texture matching.
- Depth-based Tasks: Depth-based tasks construct spatial supervision from depth maps by selecting regions with consistent internal depth and separated inter-region depth.Regional depth ordering adopts a camera-centric ordinal perspective, supporting depth and distance reasoning.
- RL Training: Training begins with a small SFT cold-start before jointly applying GRPO across all five tasks with answer-correctness and format rewards.The approximately 3,600-sample warm-up addresses instability and invalid formatted responses observed when applying RL directly.
4. Experiments
Spatial-SSRL is evaluated across diverse spatial and general visual benchmarks using standardized protocols, with ablations testing complementary task contributions. The method improves spatial understanding across image and video settings while preserving general visual capabilities.
- 4.2.1. Spatial Understanding: +5.65% (3B) and +1.21% (7B) on VSI-Bench show transfer from static-image training to video spatial understanding.The evaluation spans seven spatial benchmarks across image and video modalities.
- 4.2.2. General Visual Capabilities: General visual understanding gains +0.65% on average, while OCR and chart understanding change by +0.12% for 3B and +1.22% for 7B models.The 3B model shows a -1.0% change on ChartQA, described as within typical evaluation variance.
- 4.3. Ablation Studies: Combining all five tasks achieves the best performance on four of seven benchmarks, while no single task dominates every evaluation dimension.The results characterize depth-free and depth-based supervision as complementary.
- 4.3. Ablation Studies: Depth-free tasks improve 2D layout and general VQA, whereas depth-based tasks target ordinal depth and egocentric spatial relations.Combining both depth-based tasks gives the best multi-object performance.
- 4.3. Ablation Studies: Practitioners can prioritize Regional Depth for 3D height-related capabilities, while holistic spatial intelligence benefits from diverse task combinations.This recommendation follows the reported task-specific ablation patterns.
5. Conclusion
Spatial-SSRL derives verifiable supervision from intrinsic structure in ordinary RGB and RGB-D images and improves spatial reasoning without requiring human or proprietary-model annotations. Across seven spatial benchmarks, it raises average accuracy while preserving general visual and fine-grained perception capabilities.
- 5. Conclusion: +4.63% average accuracy for 3B models and +3.89% for 7B models across seven spatial understanding benchmarks.The benchmarks include image and video settings and show particularly strong gains on complex spatial reasoning tasks.
- 5. Conclusion: Spatial-SSRL uses intrinsic image structure to provide verifiable rewards for self-supervised reinforcement learning.The framework uses ordinary RGB and RGB-D images rather than manual annotation.
- 5. Conclusion: The method improves spatial capabilities while preserving fine-grained perception and general visual understanding.The conclusion identifies preservation of these capabilities as a critical outcome.
A.1. Statistics of Spatial-SSRL-81k
Spatial-SSRL-81k is an 81,053-sample dataset evenly distributed across five self-supervised tasks, using RGB or RGB-D inputs and automated task-specific transformations.
- 81,053 samples comprise Spatial-SSRL-81k, with the five self-supervised tasks forming its complete training dataset.Shuffled Patch Reordering and Flipped Patch Recognition are grouped as one large task, while three other tasks are mixed evenly.
- Shuffled Patch Reordering: Images are divided into M × N patches, with configurations spanning 2D grids, horizontal strips, and vertical strips, optionally including one masked patch.The listed configurations include M = 2, N = 2; M = 1, N = 3 or 4; and M = 3 or 4, N = 1.
- Cropped Patch Inpainting: Cropped Patch Inpainting selects the patch cropped from a blackened image region as the correct option and uses visually similar alternatives as distractors.Distractors can be internal or external regions of the ground-truth patch, or rotated versions of it.
- Relative Position Prediction: Relative Position Prediction defines camera- and object-based coordinate systems, with x to the right, y vertically downward, and z indicating orientation.The formulation uses thresholds for horizontal and depth directions to avoid ambiguous ground-truth answers.
- The five tasks use fixed answer formats, including patch arrangements, flip label-direction pairs, multiple-choice selections, depth orderings, and spatial relation choices.Training appends a format prompt so answers can be extracted for accuracy-based reward computation.
B.1. Spatial Benchmarks
The spatial evaluation covers image and video benchmarks measuring 2D relations, 3D distances, multi-perspective reasoning, and egocentric spatial understanding with benchmark-specific metrics and prompts.
- 3DSRBench and SpatialEval: The evaluation includes 3DSRBench and SpatialEval, which are multiple-choice benchmarks evaluated in reasoning-free and reasoning-required settings.The official VLMEvalkit code is used for these benchmarks.
- QSpatial-plus: QSpatial-plus evaluates quantitative 3D distance prediction and requires outputs containing both a scalar and a distance unit.Official prompts are used for both non-reasoning and reasoning evaluation variants.
- Spatial457 and What’sUp: Spatial457 uses non-multiple-choice questions that require chain-of-thought responses, while What’sUp measures unambiguous 2D object relations by exact option matching.Spatial457 evaluation uses the original prompt for reasoning-required baselines and the training format prompt for Spatial-SSRL models.
- ViewSpatial: ViewSpatial tests 3D reconstruction and perspective transformation through multi-perspective spatial reasoning questions.The evaluation defines separate reasoning-free and reasoning prompts to standardize answer formats.
- The benchmark suite evaluates spatial understanding across multiple-choice, numerical, image, and video tasks using exact matching or benchmark-defined numerical metrics.VSI-Bench uses exact matching for multiple-choice answers and Mean Relative Accuracy for numerical answers.
B.2. General Visual Benchmarks
General visual capabilities are evaluated with VLMEvalkit using original benchmark prompts for both baseline and Spatial-SSRL models.
- VLMEvalkit implements the full evaluation of general visual capability benchmarks for both baseline and Spatial-SSRL models.Baselines use each toolkit’s original prompt, while Spatial-SSRL models use original prompts in general VQA benchmarks.
B.3. Results on Qwen3-VL-based Models
Additional experiments train Spatial-SSRL-4B from Qwen3-VL-4B-Instruct and compare it with the baseline on spatial and general VQA benchmarks.
- Spatial-SSRL-4B is initialized from Qwen3-VL-4B-Instruct and trained solely on the Spatial-SSRL-81k self-supervised QA dataset.Both non-reasoning and reasoning variants are evaluated for spatial understanding.
- Spatial understanding results for Qwen3-VL-4B and Spatial-SSRL-4B are reported in Table 9.
- Spatial-SSRL-4B achieves an average accuracy gain of 1.18% over Qwen3-VL-4B on general VQA benchmarks.The comparison is reported in Table 10.
C. Qualitative Analysis
The qualitative analysis compares Spatial-SSRL-7B with its base model across diverse spatial reasoning tasks, including depth, orientation, location, and object relationships. The examples illustrate both correct and incorrect answers on these tasks.
- The examples span 3D location, depth comparison, orientation recognition, multi-object relationships, and viewpoint transformation.
- Additional examples test higher 3D location and vertical alignment, including a kite versus buildings and a clock relative to a station roof.
- Depth-comparison examples show competing judgments about whether the boat or fire hydrant is farther from the camera.
- Orientation examples compare judgments about which side of an airplane faces the camera, including front- and left-facing answers.
- Object-relationship examples evaluate whether a kid and teddy bear face similar directions and whether a dog lies in front of or behind an RV.