Source-linked AI summary
Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model
Fuhao Li, Wenxuan Song, Han Zhao, Jingbo Wang, Pengxiang Ding, Donglin Wang, Long Zeng, Haoang Li
TL;DR
2D-pretrained VLA models lack precise spatial awareness, while explicit 3D inputs and estimated 3D cues face practical limitations. Spatial Forcing aligns intermediate VLA visual embeddings with spatial representations from a pretrained 3D foundation model. The method achieves state-of-the-art performance across simulation and real-world evaluations, with up to 3.8× faster training and improved data efficiency.
Problem
2D-pretrained VLA backbones lack precise spatial awareness, while sensor-based and estimated 3D approaches face data, sensor, and estimator limitations.
Method
Spatial Forcing aligns intermediate VLA visual embeddings with external spatial representations extracted from pretrained 3D foundation models.
Results
SF achieves state-of-the-art performance across LIBERO, RoboTwin, and real-world evaluations, with up to 3.8× training acceleration and improved data efficiency.
Takeaways & Limitations
SF provides implicit spatial comprehension without explicit 3D inputs or depth estimators and operates without additional inference structures or computational overhead.
Takeaways & Limitations
The approach assumes that 3D information is implicitly embedded within VLA visual embeddings and can be accessed by action tokens through autoregressive inference.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have recently shown strong potential in enabling robots to follow language instructions and execute precise actions. However, most VLAs are built upon vision-language models pretrained solely on 2D data, which lack accurate spatial awareness and hinder their ability to operate in the 3D physical world. Existing solutions attempt to incorporate explicit 3D sensor inputs such as depth maps or point clouds, but these approaches face challenges due to sensor noise, hardware heterogeneity, and incomplete depth coverage in existing datasets. Alternative methods that estimate 3D cues from 2D images also suffer from the limited performance of depth estimators. We propose Spatial Forcing (SF), a simple yet effective alignment strategy that implicitly forces VLA models to develop spatial comprehension capabilities without relying on explicit 3D inputs or depth estimators. SF aligns intermediate visual embeddings of VLAs with geometric representations produced by pretrained 3D foundation models. By enforcing alignment at intermediate layers, SF guides VLAs to encode richer spatial representations that enhance action precision. Extensive experiments in simulation and real-world environments demonstrate that SF achieves state-of-the-art results, surpassing both 2D- and 3D-based VLAs. SF further accelerates training by up to 3.8x and improves data efficiency across diverse robotic tasks. Project page is at https://spatial-forcing.github.io/
1 INTRODUCTION
VLA models built on 2D-pretrained vision-language backbones lack precise spatial awareness, while existing 3D-input and estimated-3D approaches face scalability and quality challenges. Spatial Forcing addresses this gap by aligning VLA visual embeddings with external spatial representations, improving performance, training speed, and data efficiency.
- 2D-pretrained VLA backbones lack precise spatial awareness for operating in the 3D physical world.
- Existing 3D VLA approaches use depth maps or point clouds, but sensor-data limitations hinder a universal and scalable strategy.
- Depth probing shows that unaligned visual embeddings learned from 2D images fail to produce meaningful spatial structures.
- 3.8× training acceleration and improved data efficiency accompany enhanced performance across diverse robotic tasks.
- Spatial Forcing aligns intermediate VLA visual embeddings with external spatial representations to implicitly enforce spatial knowledge integration.
2 METHOD
The method treats visual tokens as intermediate scene representations and supervises them with spatial features from a pretrained 3D foundation model. Spatial Forcing combines this alignment with action training, applies supervision at effective intermediate layers, and adds no inference overhead.
- 2.1 PRELIMINARIES: VLA models process vision, language, and action tokens, generating action tokens autoregressively from preceding visual and linguistic tokens.
- 2.1 PRELIMINARIES: Visual tokens serve as intermediate scene representations that can be supervised to improve action generation.
- 2.1 PRELIMINARIES: VGGT produces camera, point-map, depth-map, and 3D-track attributes from sequences of 2D images.
- 2.3 SPATIAL FORCING: Spatial Forcing aligns per-pixel VLA visual tokens with VGGT spatial representations using positional embeddings, normalization, an MLP, and cosine similarity.
- 2.3 SPATIAL FORCING: Supervising relatively deep but not deepest causal-attention layers is most effective because deeper layers lose more vision-specific features.
- 2.3 SPATIAL FORCING: The final objective combines action-generation loss and alignment loss as LSF = Laction + αLalign.
- 2.3 SPATIAL FORCING: During inference, SF operates identically to a standard VLA without additional structures or computational overhead.
3 SIMULATION EXPERIMENTS
Simulation experiments show that Spatial Forcing improves performance across LIBERO and RoboTwin while matching methods using additional 3D inputs. Component analyses further indicate faster convergence, stronger data efficiency, and alignment that preserves the VLA’s representational identity.
- Benchmark comparisons: SF achieves the best performance across all four LIBERO task suites and remains competitive with methods using extra 3D sensor inputs.The comparison follows the same primary- and wrist-camera setup as OpenVLA-OFT.
- Benchmark comparisons: SF achieves the highest average success rate on RoboTwin and substantially improves over the base model π0 across tasks.The strongest gains on hard tasks indicate better capture of object locations and relative spatial relationships than shortcut visual cues.
- Component analysis: Supervising relatively deep but not deepest VLA layers provides the most effective alignment.The analysis suggests that deep-layer supervision also encourages alignment in shallower features.
- Training efficiency: 3.8× faster convergence enables SF to reach the same success rates as the base model in fewer training iterations.The authors attribute this acceleration to an efficient learning pathway for acquiring visual understanding and spatial relationships.
- Data efficiency: SF reaches 75.8% success with only 5% of the data, delivers 25.8% higher success at equal data amounts, and is 5.9× more efficient at equal success rates.The experiments use uniformly sampled 1%, 5%, and 33% subsets with cosine-annealing scheduling.
- Representation analysis: After alignment, VLA features adopt nearly the target’s distribution shape while retaining an independent cluster center.This indicates spatial alignment without removing the VLA’s original representational identity.
4 REAL-WORLD EXPERIMENTS
Real-world experiments evaluate Spatial Forcing across variable visual, spatial, embodiment, and bimanual conditions. SF achieves higher success across the tested tasks, including settings designed to probe spatial comprehension and data efficiency.
- Experimental setup: The real-world evaluation uses unified models across varied single-arm conditions and bimanual tasks measuring horizontal balance.The setup includes a bimanual AgileX platform with 6-DoF Piper arms, grippers, and body- and wrist-mounted cameras.
- Overall results: SF achieves higher success rates across all real-world tasks and shows considerable improvements in data efficiency.The authors state that this capability is critical for real-world deployment.
- Single-arm tasks: SF improves glass-cup stacking success by 47.5% over the base model under deceptive lighting variations.The task tests whether the model captures spatial relationships rather than overfitting lighting correlations.
- Single-arm tasks: SF achieves an 85% success rate when green-block placement heights vary.This task requires precise estimation of spatial height information.
- Bimanual tasks: Bimanual pot-lifting results indicate adaptability to a new configuration and awareness of horizontal balance needed to prevent tilting.The task probes spatial reasoning under dual-arm control.
5 RELATED WORK
Prior vision-language-action work combines vision-language models with action experts but primarily relies on 2D information. Representation supervision work instead adapts latent features through reconstruction or alignment for downstream tasks.
- Vision-language-action Models: VLA models map language instructions and scene observations to executable robot actions, supporting localization, reasoning, planning, and generalized action-language knowledge.OpenVLA is identified as an open-source VLA pretrained on large-scale robotic datasets.
- Vision-language-action Models: Most prior VLA methods focus on 2D image information and lack accurate comprehension of the 3D physical world.Recent studies address spatial perception through depth estimation and related approaches.
- Representation Supervision: Representation supervision adapts latent features through reconstruction or alignment to facilitate downstream task adaptation.Examples include generative reconstruction, denoising-based visual-embedding supervision, and related VLA representation methods.
6 CONCLUSION
The paper investigates implicit development of 3D perception in VLAs through Spatial Forcing, which aligns VLA visual embeddings with spatial representations from 3D foundation models. Simulation and real-world experiments report gains in performance, training speed, data efficiency, and spatial comprehension.
- Conclusion: A lightweight depth-probing experiment motivates Spatial Forcing by examining insufficient spatial reasoning in current VLA models.SF aligns VLA visual embeddings with external spatial representations extracted from 3D foundation models.
- Conclusion: Simulation experiments report improved performance, accelerated training, and better data efficiency after applying SF.The conclusion summarizes these benefits without relying on explicit 3D inputs in the stated method description.
- Conclusion: Real-world experiments report spatial comprehension capabilities across diverse robotic tasks.The conclusion presents this as evidence from varied real-world evaluations.
A WEIGHT FACTOR
The alignment-loss weight α controls the trade-off between developing spatial comprehension and preserving stable robot action prediction. The reported best setting is α = 0.5.
- α controls the relative weight of the alignment loss.The alignment loss is used to implicitly force spatial comprehension.
- Excessively large alignment-loss weights may destabilize the VLA visual modality and interfere with robot action prediction.
- α = 0.5 produces the best model performance and is used as the default setting in other experiments.
B EXPLANATIONS OF T-SNE RESULTS
The t-SNE analysis indicates that SF alignment makes VLA features resemble the target’s relational geometry while preserving an independent feature-space center. This supports manifold-level spatial representation alignment rather than simple feature copying.
- t-SNE visualizes high-dimensional features in a low-dimensional space by modeling pairwise neighborhood similarities.It is a nonlinear dimensionality-reduction technique typically producing 2D or 3D visualizations.
- SF-aligned VLA features exhibit nearly the same cluster distribution shape as the target features.Similar distribution shapes indicate that the relational structure among features has become isomorphic.
- The matching distribution shape suggests that SF forces VLA features to learn the target spatial representation’s underlying manifold.The model learns how the feature space is structured with respect to spatial concepts, not only individual feature locations.
- The aligned VLA cluster center remains independent from the target cluster center, indicating that alignment does not cause representational collapse.
C REAL-WORLD DATA COLLECTION
Real-world task datasets are collected by teleoperatively guiding puppet arms with master arms while recording camera images and absolute joint angles at 30Hz. Models are then fine-tuned separately for each task.
- Master arms are teleoperated to guide puppet arms through task completion.
- Camera images and absolute joint angles are recorded at 30Hz as task-specific datasets.
- Training and inference fine-tune the model separately for each task to control the puppet arms.
D DETAILS OF COMPARED MODELS
The compared models span generalist VLAs, diffusion-based policies, action-tokenization methods, video-based latent-action learning, visual reasoning, and explicit 3D spatial architectures. Their designs differ in action generation, data use, temporal reasoning, and geometric representation.
- π0 combines a pretrained VLM with a flow-matching action expert for continuous, high-frequency robot actions.
- Diffusion Policy iteratively denoises sampled actions to model multimodal action distributions and high-dimensional action spaces.
- CoT-VLA generates a future subgoal image as an intermediate visual chain-of-thought step before predicting actions.
- π0-FAST uses DCT-based frequency-space action tokenization to represent high-frequency action trajectories efficiently.
- UniVLA learns task-centric latent actions from videos with a VQ-VAE and uses them to pretrain an autoregressive vision-language policy.
- SpatialVLA injects depth-derived 3D context through Ego3D Position Encoding and discretizes movements with Adaptive Action Grids.
- GeoVLA processes 2D vision and language alongside point-cloud geometry, then fuses them with a 3D-enhanced action expert.
- 3D-CAVLA combines narrative prompts, point-cloud 3D features, and task-oriented region-of-interest pooling to improve spatial reasoning.