PaperlayerView exact revision on arXiv

Source-linked AI summary

DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoning

Tianyuan Yuan, Yicheng Liu, Chenhao Lu, Zhuoguang Chen, Tao Jiang, Hang Zhao

arXiv:2510.13375v1cs.CV

TL;DR

VLAs struggle with precise spatial reasoning because VLMs provide limited geometric understanding and existing grounding approaches rely heavily on action-data pretraining. DepthVLA adds a pretrained depth expert to a shared-attention mixture-of-transformers model, and it outperforms existing approaches across real-world and simulated evaluations. Its main scope boundary is the difficulty of monocular depth prediction in challenging visual conditions.

  • Problem

    VLAs inherit limited spatial reasoning from VLMs, while extensive action-data pretraining remains inefficient and insufficient for accurate 3D understanding.

    Source: §I · pp. 1–2

  • Method

    DepthVLA integrates a pretrained depth prediction expert with a VLM and action expert through a mixture-of-transformers architecture with shared attentions.

    Source: §I · p. 2

  • Results

    DepthVLA outperforms existing approaches across real-world, LIBERO, and Simpler evaluations, achieving 78.5% vs. 65.0%, 94.9% vs. 93.6%, and 74.8% vs. 58.8%, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    DepthVLA improves precise manipulation, collision avoidance, and fine-grained grasping while preserving strong language-following capabilities.

    Source: §V · p. 7

    Monocular depth prediction can struggle with tiny edges, reflective or transparent objects, and texture-less surfaces, affecting action generation.

    Source: §V · p. 7

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have recently shown impressive generalization and language-guided manipulation capabilities. However, their performance degrades on tasks requiring precise spatial reasoning due to limited spatial reasoning inherited from Vision-Language Models (VLMs). Existing VLAs rely on extensive action-data pretraining to ground VLMs in 3D space, which reduces training efficiency and is still insufficient for accurate spatial understanding. In this work, we present DepthVLA, a simple yet effective VLA architecture that explicitly incorporates spatial awareness through a pretrained depth prediction module. DepthVLA adopts a mixture-of-transformers design that unifies a VLM, a depth transformer, and an action expert with fully shared attentions, forming an end-to-end model with enhanced spatial reasoning. Extensive evaluations in both real-world and simulated environments show that DepthVLA outperforms state-of-the-art approaches, achieving 78.5% vs. 65.0% progress in real-world tasks, 94.9% vs. 93.6% in the LIBERO simulator, and 74.8% vs. 58.8% in the Simpler simulator. Our code will be made publicly available.

I. INTRODUCTION

DepthVLA addresses weak spatial reasoning in VLAs by integrating a pretrained depth expert with semantic and action components. It reports gains across real-world and simulated manipulation tasks while enabling separate expert pretraining.

  • VLAs inherit limited spatial reasoning from VLMs, constraining precise manipulation and motivating explicit 3D geometric representations.Source: §I · pp. 1–2
  • DepthVLA integrates a pretrained depth prediction expert with a VLM and flow-matching action expert in a mixture-of-transformers architecture.Source: §I · p. 2
  • 78.5% vs. 65.0% success in real-world tasks, 94.9% vs. 93.6% in LIBERO, and 74.8% vs. 58.8% in Simpler demonstrate DepthVLA’s stronger performance across evaluated environments.Source: §I · p. 2
  • The architecture supports separate pretraining of experts on diverse datasets beyond embodied action data, improving training scalability.Source: §I · p. 2
  • Evaluations report gains in grasping accuracy and collision avoidance on real-world and simulated tasks.Source: §I · p. 2

II. RELATED WORK

Prior VLA research progressed from generalist action policies toward methods for spatial awareness, but action-data demands, limited geometry, and inference latency remain concerns.

A. Generalist Robot Manipulation Policies

  • Generalist VLAs improve task and embodiment coverage but still require large-scale action-data pretraining, limiting efficiency and fine-grained spatial understanding.Source: A. Generalist Robot Manipulation… · p. 2

B. VLAs with Spatial Awareness

  • Additional LiDAR or RGB-D inputs reduce cross-platform generalizability, while off-the-shelf depth estimation is not optimized end-to-end with the VLA.Source: B. VLAs with Spatial Awareness · p. 2
  • World-model approaches improve planning by simulating futures but do little to improve current-scene encoding, while CoT depth-token generation adds over 2 seconds of latency.Source: B. VLAs with Spatial Awareness · p. 2

C. 3D Geometry Perception

Recent 3D perception advances motivate DepthVLA’s explicit geometric reasoning within a unified VLA policy. Its MoT combines semantic, spatial, and action experts while preserving pretrained capabilities through masking.

C. 3D Geometry Perception

  • 3D vision foundation models show strong monocular or multi-view geometric estimation, motivating their integration into VLAs without additional sensors.Source: C. 3D Geometry Perception · p. 2

A. Problem Formulation and Model Overview

B. Depth Expert

The depth expert is designed as a pretrained spatial reasoner whose intermediate geometric features guide action generation.

B. Depth Expert

  • The depth expert uses a transformer backbone with separate weights and dimensions to provide geometric cues for the VLA.Source: B. Depth Expert · p. 3
  • DINOv2 features initialized from Depth Anything V2 [19] support fine-grained geometry, while intermediate representations guide actions beyond a final depth map.Source: B. Depth Expert · p. 3
  • The expert is pretrained on diverse 3D datasets with a monocular depth task before VLA integration.Source: B. Depth Expert · p. 3
  • The scale-invariant log loss uses ground-truth metric depth d, predicted depth ˆd, and λ to balance the scale term, with λ set to 0.5 by default.Source: B. Depth Expert · p. 3
  • DepthVLA achieves the highest average performance on the Simpler WidowX benchmark under zero-shot evaluation after BridgeData V2 training.Source: Table I · p. 4

C. DepthVLA Policy Training

DepthVLA trains with flow matching for continuous action trajectories while retaining depth prediction supervision. The combined objective jointly optimizes spatial reasoning and action generation end to end.

C. DepthVLA Policy Training

  • Flow matching models continuous and diverse action trajectories by predicting a flow toward the target action trajectory.The model predicts vθ, while u is the target flow derived from the action trajectory.Source: C. DepthVLA Policy Training · p. 4
  • Flow-matching timesteps use τ ∈[0,1], while subscripts denote robot timesteps and superscripts denote flow-matching timesteps.The passages distinguish robot-time indexing from flow-matching-time indexing.Source: C. DepthVLA Policy Training · p. 4
  • The training objective preserves depth supervision during VLA training to maintain the depth expert’s spatial reasoning.Depth prediction loss is retained alongside the flow-matching loss.Source: C. DepthVLA Policy Training · p. 4
  • DepthVLA jointly optimizes spatial reasoning and action generation end to end by combining depth and flow-matching losses.The final objective is L = Lsi + Lflow.Source: C. DepthVLA Policy Training · p. 4

A. Implementation Details

The implementation combines pretrained VLM, depth, and action components and evaluates them across Simpler WidowX and LIBERO. DepthVLA achieves the strongest reported simulation performance, including against pretrained and spatially enhanced baselines.

A. Implementation Details

  • The depth expert uses a DINOv2-L encoder initialized from Depth Anything V2, while its decoder and the action expert each contain approximately 300M parameters.Paligemma-3B serves as the VLM backbone.Source: A. Implementation Details · p. 4
  • DepthVLA pretrains its depth expert on WildRGB-D, ScanNet, ScanNet++, and HyperSim before VLA training.The depth pretraining uses 50k steps, batch size 1024, and an initial learning rate of 5×10−5.Source: A. Implementation Details · p. 4

B. Simulation Benchmarking

  • Simpler WidowX mirrors BridgeData V2, and DepthVLA is trained on BridgeData V2 before zero-shot evaluation across four task suites.Each suite is tested with 120 trials under different random seeds.Source: B. Simulation Benchmarking · p. 4
  • DepthVLA achieves the highest average success rate on Simpler WidowX, outperforming its no-depth counterpart and SpatialVLA.The gains include block stacking and eggplant picking, which require spatial reasoning and collision avoidance.Source: B. Simulation Benchmarking · pp. 4–5
  • DepthVLA achieves the highest average LIBERO success rate, surpassing models with additional action-data pretraining, spatial baselines, and world-model approaches.The evaluation jointly trains one model across LIBERO-Spatial, Object, Goal, and Long.Source: B. Simulation Benchmarking · p. 5

C. Real-World Benchmarking

The real-world evaluation tests DepthVLA on a dual-arm mobile robot across grasping, collision avoidance, and stacking tasks. DepthVLA outperforms the baseline in standard and few-shot fine-tuning, while matching it on simpler grasping.

C. Real-World Benchmarking

  • The evaluation uses the Galaxea R1 Lite, a dual-arm mobile platform with two 6-DoF arms, wrist cameras, and a head camera.DepthVLA and the re-implemented π0 are pretrained on 100k Galaxea Open-World trajectories before task-specific fine-tuning.Source: C. Real-World Benchmarking · p. 5
  • 79% vs. 65% average progress favors DepthVLA over the baseline under standard fine-tuning, while few-shot performance is 63% vs. 45%.Scores average successful substeps over 20 runs per task.Source: C. Real-World Benchmarking · p. 5
  • DepthVLA improves collision avoidance during microwave operation and spatial perception during block stacking, even with limited fine-tuning data.These tasks evaluate collision avoidance and precise pick-and-place respectively.Source: C. Real-World Benchmarking · p. 5
  • On table bussing, DepthVLA performs comparably to the baseline, indicating that both models handle relatively simple small-object grasping tasks effectively.The task organizes a cluttered desk by moving pens, headphones, and a book.Source: C. Real-World Benchmarking · p. 5

D. Ablation Studies

The ablations test depth-expert pretraining, training loss, freezing, attention masking, and depth-input design across simulated manipulation settings. Results identify the depth components that support performance and show that predicted depth can outperform direct ground-truth depth input.

Ablation Studies

  • Ablation settings (i)–(iv) use BridgeData V2 and Simpler, whereas the ground-truth-depth comparison uses LIBERO inference maps.The settings distinguish random initialization, removing depth loss, freezing the expert, cross-attention, and ground-truth depth input.Source: D. Ablation Studies · p. 5
  • The LIBERO table reports success rates across four task suites and compares models with or without additional robot-action pretraining.Its caption states that DepthVLA outperforms all baselines across these diverse manipulation tasks.Source: Table II · p. 6
  • DepthVLA improves over the baseline on three bimanual tasks requiring precise spatial reasoning and collision avoidance, while maintaining comparable performance on simpler small-object manipulation.Source: Figure 4 · p. 6
  • Predicting depth yields stronger performance than consuming ground-truth depth directly.The authors attribute this to modality competence: internal prediction may avoid over-reliance on externally supplied depth signals.Source: Table IV · p. 6, D. Ablation Studies · p. 6
  • Each evaluated depth-expert component proves essential, while freezing the depth expert causes little performance impact.This supports using a robust pretrained spatial representation without requiring depth ground truth during fine-tuning.Source: D. Ablation Studies · pp. 5–6

E. Visualization of Depth Prediction

DepthVLA’s visualized predictions show that its depth expert extracts detailed 3D structure from monocular RGB input. These geometric cues support grasping, collision avoidance, object alignment, and positioning in cluttered real and simulated scenes.

Visualization of Depth Prediction

  • Predicted depth captures object boundaries, distances, and occlusions that support precise manipulation.In cluttered microwave scenes, it estimates relative object positions for grasping and collision avoidance.Source: E. Visualization of Depth Prediction · p. 6, Figure 5 · p. 7
  • Depth predictions provide fine-grained geometric cues for object alignment and positioning in Simpler block stacking and LIBERO-long tasks.The action expert uses these cues to improve spatially precise manipulation.Source: E. Visualization of Depth Prediction · p. 7
  • The depth expert extracts 3D spatial information from monocular RGB and complements the VLM’s semantic grounding.The paper links this depth-aware representation to improvements across real and simulated environments.Source: E. Visualization of Depth Prediction · p. 7

V. CONCLUSION

DepthVLA combines a pretrained depth expert with a VLM and action expert in a unified mixture-of-transformers framework. Across real-world and simulated tasks, it improves precise manipulation while retaining language-following capabilities, but monocular depth remains a challenging limitation.

V. CONCLUSION

  • DepthVLA integrates a pretrained depth expert, VLM, and action expert to enhance spatial reasoning in a unified mixture-of-transformers model.The conclusion identifies depth pretraining, depth loss, and attention design as critical for robust 3D perception.Source: §V · p. 7
  • DepthVLA improves precise manipulation, collision avoidance, and fine-grained grasping while preserving strong language-following capabilities.These findings are reported across both real-world and simulated environments.Source: §V · p. 7
  • Monocular depth prediction remains ill-posed and can struggle with tiny edges, reflective or transparent objects, and texture-less surfaces.The paper suggests multi-view depth or pointmap prediction as future directions for improved spatial accuracy and robustness.Source: §V · p. 7
Loading 2510.13375v1…