PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model

Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, Xuelong Li

arXiv:2501.15830v5cs.ROcs.AI

TL;DR

Existing VLA models lack precise 3D spatial understanding and face observation and action mismatches across robot embodiments. SpatialVLA addresses these issues with Ego3D Position Encoding and Adaptive Action Grids, and extensive evaluations report improved generalization, transfer, and adaptation, while the authors note inference remains slower than desired.

  • Problem

    Existing VLA models primarily rely on 2D observations, while heterogeneous robots have non-aligned 3D observation spaces and differing action characteristics.

  • Method

    SpatialVLA injects egocentric 3D context into visual features and represents continuous robot actions with adaptive, distribution-based spatial action grids that can be re-discretized for new setups.

  • Results

    SpatialVLA shows stronger generalization and transfer across diverse manipulation tasks and robot embodiments, including reported gains of +36.5% and +42.1% over linear 256-bin discretization in Google Robot variant aggregation and visual matching success rates.

  • Takeaways & Limitations

    Adaptive spatial representations support zero-shot robot control and flexible post-training transfer across new robot environments and embodiments.

  • Takeaways & Limitations

    SpatialVLA achieves 21Hz inference speed but remains slower than the comparison target described by the authors, and discretization performance plateaus or can decline beyond 6166 grid resolution.

Abstract

from arXiv · show

In this paper, we claim that spatial understanding is the keypoint in robot manipulation, and propose SpatialVLA to explore effective spatial representations for the robot foundation model. Specifically, we introduce Ego3D Position Encoding to inject 3D information into the input observations of the visual-language-action model, and propose Adaptive Action Grids to represent spatial robot movement actions with adaptive discretized action grids, facilitating learning generalizable and transferrable spatial action knowledge for cross-robot control. SpatialVLA is first pre-trained on top of a vision-language model with 1.1 Million real-world robot episodes, to learn a generalist manipulation policy across multiple robot environments and tasks. After pre-training, SpatialVLA is directly applied to perform numerous tasks in a zero-shot manner. The superior results in both simulation and real-world robots demonstrate its advantage of inferring complex robot motion trajectories and its strong in-domain multi-task generalization ability. We further show the proposed Adaptive Action Grids offer a new and effective way to fine-tune the pre-trained SpatialVLA model for new simulation and real-world setups, where the pre-learned action grids are re-discretized to capture robot-specific spatial action movements of new setups. The superior results from extensive evaluations demonstrate the exceptional in-distribution generalization and out-of-distribution adaptation capability, highlighting the crucial benefit of the proposed spatial-aware representations for generalist robot policy learning. All the details and codes will be open-sourced.

I. INTRODUCTION

SpatialVLA addresses the lack of 3D spatial understanding and cross-robot alignment in existing VLA models by introducing robot-agnostic spatial representations for observations and actions. It combines Ego3D Position Encoding with Adaptive Action Grids and evaluates generalization across diverse tasks, embodiments, and environments.

  • Existing VLA models primarily use 2D observations and lack precise perception of the 3D physical world for manipulation.
  • Heterogeneous robots produce non-aligned 3D observations and exhibit different action characteristics because of sensor placement, embodiment, controllers, workspaces, and task complexity.
  • SpatialVLA integrates egocentric 3D context with visual semantics through Ego3D Position Encoding, avoiding robot-camera extrinsic calibration across embodiments.
  • Adaptive Action Grids discretize continuous robot actions into spatial tokens using dataset action distributions, aligning cross-robot movements with 3D structure.
  • Evaluation covers 24 real-robot tasks and 3 simulation environments, including zero-shot control and adaptation under changes in views, textures, lighting, objects, environments, and spatial layouts.
  • The model represents one-step 7D robot actions with three spatial action tokens and supports adaptive grid re-discretization for new robot setups.

B. The Pre-training and Post-training Scheme

SpatialVLA uses a two-stage training scheme: pre-training learns generalizable knowledge across diverse robots and tasks, while post-training adapts the model to new embodiments or tasks. Its adaptive action-grid mechanism transfers pretrained spatial action knowledge by initializing new action-token embeddings from neighboring pretrained grids.

  • Pre-training Procedure: Pre-training learns generalizable robot knowledge from a large-scale cross-robot dataset mixture, followed by post-training for new embodiments or tasks.The pre-training procedure uses a Paligemma2 backbone and a dataset mixture containing 1.1 Million episodes.
  • Post-training Adaptation: New translation-grid token embeddings are initialized by trilinear interpolation from adjacent pretrained grids using normalized centroid distances.Rotation-grid embeddings are initialized in the same way, allowing the new tokenizer to transfer pretrained spatial action knowledge to new robot setups.
  • Post-training Adaptation: The interpolated action-token initialization enables a new action tokenizer to transfer pretrained spatial action knowledge to new robot setups.The same initialization strategy applies to new rotation action tokens.

IV. EXPERIMENT

The experiments test whether SpatialVLA provides generalist robot control after pre-training, adapts to new setups, and handles tasks requiring spatial understanding. Evaluations span simulation and real-world robots, while ablations examine the proposed spatial representations.

  • Evaluation Scope: The study evaluates both out-of-the-box control and fine-tuning efficacy across simulation and real-world settings.The experiments compare pretrained performance on matching setups with adaptation to new robot environments and tasks.
  • Research Questions: The evaluation asks whether SpatialVLA performs zero-shot in-distribution control, adapts to new setups and tasks, and improves spatial understanding.A fourth question specifically examines the contributions of Egocentric 3D Spatial Representations and Adaptive Spatial Action Grids.
  • Evaluation Scope: Experiments cover 7 robot learning scenarios, 24 real-robot tasks, and 3 simulation environments.The setups include SimplerEnv, WidowX, LIBERO, and new Franka robot environments.
  • Evaluation Scope: The evaluation tests instruction following and 3D scene-structure understanding under visual, object, robot-environment, and spatial-layout changes.These tests include unseen objects, unseen robot environments, and challenging spatial-layout changes.

A. Performing Zero-shot Robot Control

SpatialVLA is evaluated for zero-shot control across simulated and real-world robot settings with varied visual conditions, task suites, and competing generalist policies. It achieves the strongest reported performance in the supplied comparisons, including visual-matching gains on Google Robot and high success rates on WidowX.

  • Real-world WidowX: The real-world WidowX evaluation uses 7 task suites covering language grounding, semantic understanding, and motion distractors across 77 rollouts.The task suites include unseen backgrounds and poses, manually moved objects, and moderately complex manipulation tasks.
  • SimplerEnv Google Robot: 71.9% and 75.1% Visual Matching scores are achieved by SpatialVLA in zero-shot and fine-tuning settings on Google Robot tasks.These results surpass the second-best RoboVLM by +15.6% and +11.7% margins, respectively.
  • SimplerEnv Google Robot: SpatialVLA achieves the highest overall visual matching and variant aggregation performance on the Google Robot setup.The benchmark varies lighting, colors, textures, and robot camera poses to test robustness across visual appearances.
  • SimplerEnv WidowX: 34.4% and 42.7% overall success rates are reported for SpatialVLA on the WidowX setup.The passage states that these results surpass the state-of-the-art RoboVLM policy.
  • SimplerEnv WidowX: 100% success is achieved after BridgeV2 fine-tuning on the “Put Eggplant in Yellow Basket” task.This result is reported in the WidowX evaluation as evidence of strong manipulation performance.
  • Real-world WidowX: SpatialVLA is more robust than OpenVLA to motion disturbances in the real-world WidowX evaluation.Other policies frequently encounter object misidentification and grasp failures on moderately complex tasks.

B. Adapting to New Robot Setups

SpatialVLA is evaluated across simulation and real-robot setups, where spatial representations and adaptive action grids support generalization and adaptation to new tasks and embodiments.

  • Evaluation Setup: SpatialVLA is evaluated on LIBERO simulation suites and 13 Franka tasks spanning single-task, instruction-following, and multi-task settings.The LIBERO suites cover spatial, object, goal, and long-horizon generalization; Franka evaluations compare mainstream generalist policies.
  • Simulation Results: Fine-tuned SpatialVLA achieves the highest average success rate and ranking across the four LIBERO task suites.The benchmark averages success rates over three random seeds and 500 trials.
  • Franka Results: +12% over OpenVLA in instruction following and 57% accuracy in multi-tasks demonstrate stronger performance across these Franka setups.Diffusion Policy reaches 26% success in instruction following, while SpatialVLA achieves the highest reported multi-task accuracy.
  • Action-Grid Ablations: Adaptive spatial action grids improve Google Robot variant aggregation and visual matching success rates by +36.5% and +42.1% over linear 256-bin discretization.Initializing grid partitioning from dataset distributions also focuses learning on high-frequency action spaces and further improves generalization.
  • Action-Grid Ablations: 8194-resolution action grids improve success rates by +31.2% and +33.3% over 1026-resolution grids on Move Near and Put Eggplant in Yellow Basket.Lower-resolution models learn smaller actions and exhibit slow motion, whereas higher resolution improves fine-tuning transfer performance.
  • Spatial-Representation Ablations: Ego3D position encoding improves robustness to lighting, color, texture, and camera-pose changes, while removing it reduces variant aggregation from 81.6% and 79.2% to 68.9% and 66.7%.The ablation incorporates 3D point-cloud features into the spatial representation.
  • Post-Training Adaptation: Spatial embedding adaptation yields +4.6%, +5.1%, +2.2%, and +5.4% on LIBERO-Spatial, Object, Goal, and Long, respectively.The adaptation aligns pretrained spatial features with the target model, improving initialization and accelerating convergence; large-scale datasets show only +2.9% on Move Near visual matching.

V. DISCUSSION, LIMITATIONS, AND FUTURE WORK

The discussion identifies inference speed, long-horizon modeling, and heterogeneous-data quality as limitations, while proposing future improvements in decoding, history modeling, and data composition.

  • Inference and Decoding: SpatialVLA achieves 21Hz inference but is slower than diffusion decoding methods that produce multiple consecutive actions.The paper proposes integrating diffusion decoding with spatial grid action representation and dynamically varying action-token counts.
  • Long-Horizon Modeling: Relying on current-frame observations and history tokens creates challenges for long-horizon tasks and motivates more efficient historical-information perception.The proposed direction targets stronger long-sequence modeling and real-time task switching.
  • Data Quality: Variable quality in OXE data can hinder training, motivating better data composition and high-quality subset distillation from heterogeneous robot collections.The limitation concerns the quality and composition of the pre-training data.

APPENDIX

The appendix details SpatialVLA’s architecture, adaptive action-grid encoding and decoding, dataset mixture, and discretization trade-offs. It also reports that fine-grained manipulation remains effective, while higher-dimensional action spaces increase vocabulary overhead.

  • Dataset Mixture: The training mixture prominently includes Bridge, Fractal, Droid, BC-Z, Kuka, RH20T, Stanford Hydra, and Language Table datasets, with RH20T added to OpenVLA’s mixture.Several datasets were down-weighted during mixing, including Kuka, Toto, Berkeley Fanuc Manipulation, and FMB.
  • Model Architecture: SpatialVLA combines PaliGemma2, SigLIP, ZoeDepth, an Ego3D Position Embedding MLP, and spatial embeddings to process RGB, depth, and language inputs.The 224×224 RGB image is separately encoded for visual features and depth prediction before multimodal fusion.
  • Action Representation: Adaptive Action Grids discretize translational and rotational actions using CDF/PPF-based partitions, then linearize them into 8194 spatial action-token embeddings.The action representation includes translation, rotation, and gripper components, with embeddings sharing parameters with the language-model vocabulary.
  • Discretization Ablations: Performance improves as grid resolution rises from 1026 to 8194, then plateaus or declines beyond 6166, while adaptive partitioning enables fine-grained manipulation.The method reportedly outperforms uniform partitioning #1 U8196 with half the partitions in the cited ablation.
  • Action Encoding and Decoding: The action pipeline digitizes normalized continuous actions into translation and rotation grids, predicts tokens autoregressively, then gridifies and unnormalizes decoded actions.The gripper token is encoded separately from translation and rotation tokens.
  • Limitations: Higher-resolution or higher-dimensional action spaces trade precision for efficiency by adding parameters to the 128k vocabulary, especially for robots with more degrees of freedom.The appendix suggests sharing action grids across embodiments as a future direction.

Q2. Handle more complex task beyond pick-and-place?

SpatialVLA is evaluated beyond pick-and-place through fine-grained dexterity tasks and depth-input comparisons. The reported results support effective manipulation with discrete actions and smoother ZoeDepth inputs than noisy sensor depth.

  • Fine-grained manipulation: 72.7% vs 54.5%: SpatialVLA significantly outperforms OpenVLA on four complex fine-grained dexterity tasks.The authors attribute this result to probability-adaptive action-grid partitioning.
  • Depth inputs: ZoeDepth produces smoother inputs than sensor depth, which the authors describe as noisy and disruptive.The comparison is reported for depth ablations associated with the fine-grained manipulation evaluation.

D. Model Pre-training and Deployment.

SpatialVLA is pretrained on large-scale cross-robot data and evaluated across zero-shot control, spatial understanding, and adaptation settings. The appendix specifies the real-robot tasks used for these evaluations on WidowX and Franka systems.

  • Pre-training: SpatialVLA is pretrained on 1.1 Million real robot demonstrations from diverse embodiments, scenes, and tasks using a two-stage training procedure.Pretraining learns cross-task and cross-robot knowledge, while post-training adapts the model to new embodiments or tasks.
  • Franka adaptation: Franka adaptation evaluates four methods across 13 real-world tasks, with Diffusion Policy trained from scratch and Octo, OpenVLA, and SpatialVLA fine-tuned.Each task uses 11 trials.
  • WidowX tasks: WidowX evaluations include articulated-object manipulation, pick-and-place, color-conditioned placement, object transfer, and a combined push/place/put/close multitask setting.The listed tasks include closing drawers, manipulating teapot handles, placing colored cubes, and transferring food objects.
  • Spatial understanding: Spatial-understanding tasks vary object proximity, articulated-door angles, object heights, target elevation, and cluttered object identities or colors.These changes test whether manipulation trajectories adjust to spatial configurations and object-selection requirements.

F. More Detailed Evaluation Results.

Detailed evaluations show strong performance for SpatialVLA in simulation, single-task adaptation, multitask fine-tuning, and instruction-following fine-tuning. The strongest reported pattern is robust generalization across diverse manipulation scenarios.

  • SimplerEnv: SpatialVLA achieves the highest average visual-matching and variant-aggregation performance in the SimplerEnv evaluation.The evaluation includes Coke-can picking and drawer-opening or closing tasks.
  • Single-task fine-tuning: SpatialVLA achieves state-of-the-art performance on 2 out of 4 single-task fine-tuning tasks.It trails Diffusion Policy on Close the Drawer but exceeds it on the challenging Push the Teapot Handle Aside task.
  • Multitask fine-tuning: SpatialVLA achieves the highest success rate in 3 out of 4 multitask fine-tuning tasks, with only a marginal gap to OpenVLA on White Pot placement.Diffusion Policy trained from scratch shows significant degradation in the multitask setting.
  • Instruction following: SpatialVLA achieves the highest success rate in 3 out of 5 instruction-following fine-tuning tasks.These tasks require both physical manipulation and language understanding.

4. Spatial Understanding Capability Evaluation on Franka and WidowX Robot

The evaluation covers spatial understanding across fine-tuning and zero-shot robot tasks, including Google Robot and BridgeV2 WidowX settings. SpatialVLA is reported as achieving strong performance across these evaluations.

  • The evaluation includes one fine-tuning task and three zero-shot tasks designed to test spatial reasoning abilities.
  • The fine-tuning comparison includes Diffusion Policy, Octo-Base, and OpenVLA baselines.
  • SpatialVLA achieves state-of-the-art performance on all 7 real-world BridgeV2 WidowX zero-shot tasks.

G. Visulizatiob of Dataset Statistic.

The dataset visualization shows concentrated, ellipsoidal translation and rotation distributions, motivating adaptive rather than uniform action encoding. Across fine-tuning evaluations, SpatialVLA achieves state-of-the-art results on multiple single-task, multi-task, and instruction-following tasks.

  • Visulizatiob of Dataset Statistic.: Translation and rotation samples form ellipsoidal clusters around the center, with projections concentrated within the 2-sigma action space.
  • Visulizatiob of Dataset Statistic.: The concentrated action distribution motivates adaptive division instead of uniform encoding across the entire [-1,1] range.
  • Visulizatiob of Dataset Statistic.: Gaussian distributions are adopted for generality and simplicity rather than kernel density estimations.
  • Visulizatiob of Dataset Statistic.: SpatialVLA achieves state-of-the-art performance on 2 of 4 single-task fine-tuning tasks tested on Franka Emika Panda.
  • Visulizatiob of Dataset Statistic.: SpatialVLA achieves state-of-the-art performance on 3 of 4 multi-task and 3 of 5 instruction-following fine-tuning tasks on Franka Emika Panda.
  • Visulizatiob of Dataset Statistic.: SpatialVLA achieves state-of-the-art performance in one fine-tuning and three zero-shot spatial-understanding tasks.
Loading 2501.15830v5…