Source-linked AI summary
Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoning
Yijun Yang, Shenghe Zheng, Wenbo Li, Jianhui Liu, Haoze Sun, Yanbing Zhang, Jiaxiu Jiang, Lin Song, Haoyang Huang, Nan Duan, Lei Zhu
TL;DR
VLMs remain limited in spatial reasoning because conventional learning treats dynamic physical scenes largely as flat 2D observations, leaving 3D geometry and temporal continuity underrepresented. FactoSR uses reinforcement learning with factorized XY, Z, and T rewards to impose explicit geometric and temporal constraints. Across diverse 3D and 4D benchmarks, it reports substantial spatial-reasoning gains, including 5.9% on VSI-Bench and 4.5% on All-Angles-Bench.
Problem
VLMs struggle with spatial reasoning because existing approaches rely heavily on single-view 2D patterns and lack explicit 4D physical-world perception.
Method
FactoSR combines supervised spatial fine-tuning with reinforcement learning using factorized XY correspondence, Z depth, and T temporal rewards.
Results
5.9% on VSI-Bench and 4.5% on All-Angles-Bench are reported improvements, while FactoSR-8B-RL averages 62.0 across 3D/4D spatial benchmarks.
Takeaways & Limitations
FactoSR supports reasoning across planar correspondence, depth, and temporal dimensions underlying visual observations.
Abstract
from arXiv · showhide
Despite the remarkable prowess of Vision-Language Models (VLMs) in general multimodal tasks, they remain fundamentally ``flat'' when reasoning about the physical world. We argue that this spatial bottleneck stems from a profound dimensional mismatch: while VLMs are trained to interpret 2D projections, true spatial reasoning demands the recovery of latent 3D geometry and temporal continuity. To conquer this high-dimensional complexity, we advocate a shift from monolithic learning to a ``divide and conquer'' paradigm. We present FactoSR, a factorized reinforcement learning framework that explicitly interpret the dimensions collapsed by visual projection. At its core, FactoSR decomposes the monolithic problem of world-consistent reasoning into three orthogonal, geometric sub-objectives: planar correspondence ($XY$), depth consistency ($Z$), and temporal reversibility ($T$). By optimizing these verifiable constraints within a unified policy learning mechanism, we effectively transform an ill-posed projection recovery problem into a series of tangible reasoning steps. Extensive evaluations on multi-view and video benchmarks demonstrate that this elegant decomposition yields substantial gains in 3D and 4D reasoning, achieving a 5.9% boost on VSI-Bench and 4.5% on All-Angles-Bench. Our findings suggest that reinforcing explicit, factorized 4D consistency is a critical step toward evolving VLMs into robust, world-aware reasoners.
1 Introduction
The paper identifies spatial reasoning as a persistent VLM limitation caused by reasoning over dynamic worlds through incomplete 2D projections. FactoSR addresses this challenge by factorizing 4D reasoning into verifiable planar, depth, and temporal objectives, yielding gains on spatial benchmarks.
- Motivation: Spatial reasoning remains elusive for VLMs despite their success on general visual tasks, limiting deployment in dynamic spatial-intelligence applications.The cited applications include autonomous driving, robotics navigation, and world models.
- Motivation: Existing supervised and token-based approaches require substantial 3D data and transfer poorly to 4D scenes because they learn mainly from single-view 2D patterns.The paper links this static learning to hallucination and heuristic spatial decisions without geometric correspondence, depth, or temporal reasoning.
- Motivation: Existing spatial RLVR methods emphasize final correctness and single-view learning, rather than explicitly modeling the dynamics of latent worlds.The paper characterizes their rewards as inherited from general understanding and insufficient for depth-distorted dynamics.
- FactoSR: FactoSR factorizes spatial reasoning into XY correspondence, Z depth, and T temporal objectives within online policy reinforcement learning.Its rewards respectively constrain cross-view reprojection, 3D localization and depth ordering, and temporal cycle consistency with reversible camera-motion reasoning.
- Results: 5.9% on VSI-Bench and 4.5% on All-Angles-Bench are reported improvements from the proposed factorized framework.The introduction reports these gains alongside preserved general multimodal capabilities.
2 Related Work
Prior work improves spatial intelligence through architectural changes, spatial representations, supervised or progressive training, and multimodal reinforcement learning. The paper argues that these directions leave depth-temporal learning pathways incomplete and motivate structured, decoupled rewards.
- Multimodal Reinforcement Learning: Prior multimodal RL work uses iterative SFT–RL cycles and self-reflection, but the paper identifies a gap in jointly addressing depth and temporal analysis.The proposed response is a reinforcement learning framework with decoupled rewards.
- Reinforcement Learning for Spatial Intelligence: Spatial-intelligence RL research uses verifiable rewards, intermediate stages, perception-oriented rewards, progressive training, and explicit 3D representations.The paper concludes that long chain-of-thought reasoning or sparse rewards alone is insufficient, motivating multiple verifiable rewards.
3 Preliminaries
The paper formulates 4D spatial-temporal intelligence as reasoning over time-ordered visual observations and an implicit representation of geometric and temporal properties. It situates RLVR and GRPO as the optimization preliminaries for verifiable policy learning.
- Problem Formulation: Each dataset sample contains a visual sequence, a spatial-temporal query, and a ground-truth answer, with the sequence encoding dynamic 3D scene evolution.The formulation denotes the dataset as D = {x1, x2, . . . , xN} and each sample as xi = (V1:t, Qst, a).
- Problem Formulation: 4D intelligence requires an implicit spatio-temporal representation capturing geometric attributes and temporal dynamics before generating a textual answer.Geometric examples include distance, orientation, occlusion, and topology; temporal examples include motion, interaction, and state transitions.
- RLVR: RLVR derives rewards directly from ground-truth correctness rather than a learned reward model, reducing auxiliary estimation and reward-hacking risks.The cited passage also states that this simplifies training and reduces computational cost.
- GRPO: GRPO combines reward computation with group-based relative advantage estimation for stable policy updates without requiring a value model.The method is described as improving stability while reducing memory and computational cost.
- GRPO: The GRPO objective averages clipped policy-ratio advantages across sampled responses while penalizing divergence from a reference policy.The displayed objective includes a KL-divergence penalty and response-length normalization.
4 Methodology
FactoSR uses progressive training: supervised fine-tuning builds spatial perception, then factorized reinforcement learning optimizes explicit correspondence, depth, and temporal consistency. Its geometric rewards turn multi-view reasoning into verifiable constraints involving reprojection, visibility, depth order, and temporal reversibility.
- Stage 1: Spatial Perception Fine-tuning: FactoSR first uses supervised fine-tuning to develop spatial grounding and foundational perception from short-form multimodal and spatial-task data.The stage emphasizes localization, correspondence, and spatial relations, and runs for a single epoch to inject spatial priors.
- Stage 2: Factorized Reinforcement Learning: Stage 2 applies factorized reinforcement learning with Accuracy, XY, Z, and T rewards to refine structured spatial reasoning.The framework transitions from static supervision to rule-based verification over planar correspondence, depth, and temporal cycle-consistency.
- XY-plane correspondence: The XY reward scores cross-view point correspondence by comparing predictions with geometry-based reprojections in normalized image coordinates.The reprojection uses depth, camera intrinsics, and camera poses, with a distance-aware reward and a cutoff that suppresses grossly incorrect matches.
- XY-plane correspondence: Visibility-aware overlap gating assigns zero reward to correspondences outside the depth-consistent region, discouraging matches on occluded or non-overlapping areas.The overlap mask is derived from reprojection and depth consistency between the two views.
- Z-depth reasoning: The Z reward evaluates front–back object ordering by comparing predicted and ground-truth depth sequences with pairwise concordance.Matched 3D boxes provide object-center depths, and Kendall-τ summarizes concordant and discordant pair orders.
- Temporal reasoning and total objective: The final objective combines semantic accuracy with geometric constraints while requiring valid formatting, and the T reward enforces reversibility across forward and inverse motion sequences.The factorized reward is intended to shift reasoning from pattern matching toward physically grounded 4D scene understanding.
5 Experiments
FactoSR is evaluated across 4D, 3D, general multimodal, qualitative, and ablation settings. Results show strong benchmark performance, targeted gains from factorized rewards, and more consistent spatial reasoning across views, depth, and motion.
- 4D Reasoning Analysis: FactoSR-8B-RL achieves the best performance on All-Angles-Bench and VSI-Bench among compared open-source VLMs.The model reaches 61.5% on VSI and 55.4% on AllAngles, outperforming strongest open-source baselines by +5.9 and +2.5 points, respectively.
- Overall Benchmark Comparison: FactoSR-8B-RL achieves an average accuracy of 62.0 across all 3D/4D spatial benchmarks, improving over the base model by +2.9 points.It also maintains competitive performance on general multimodal benchmarks such as MMBench-CN and MMBench-EN.
- Ablation Study: XY improves correspondence reasoning by ∆Cor +2.7, Z improves depth understanding by ∆Depth +1.3, and T improves temporal reasoning by ∆t +7.9.The specialized evaluation groups measure correspondence, depth, and camera-motion capabilities across nine spatial benchmarks.
- Ablation Study: Combining XY, Z, and T rewards achieves the best overall improvement of 1.7%, indicating complementary supervision across the three dimensions.The Z reward particularly improves Relative Depth by +3.2, while the vanilla grounding reward causes overall performance to drop by -0.2.
- Qualitative Analysis: After reinforcement learning, the model uses cross-view, depth-aware, and temporal chain-of-thought patterns instead of appearance or size heuristics.The qualitative analysis describes anchor-transfer-verify reasoning for correspondence, occlusion and layering for depth, and temporal reasoning for camera motion.
- Generalization to Video Route Plan: FactoSR-RL constructs coherent route plans from egocentric video by reconstructing spatial layout and camera motion.The model uses motion parallax, yaw rotation, and cross-frame verification to identify the path consistent with the reconstructed 4D navigation structure.
6 Conclusion
FactoSR factorizes spatial reasoning into complementary plane, depth, and time dimensions, enabling models to reason over latent spatial structure. Results across diverse 3D and 4D benchmarks show improved spatial reasoning and interaction with the physical world.
- FactoSR factorizes spatial reasoning into complementary plane, depth, and time dimensions.
- The framework enables models to reason over latent spatial structure underlying visual observations.
- Results on diverse 3D and 4D spatial benchmarks show that FactoSR advances spatial reasoning.
- FactoSR encourages multimodal models to observe, reason, and interact with the latent structure of the physical world.