PaperlayerView exact revision on arXiv

Source-linked AI summary

Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding

Tao Lin, Gen Li, Yilei Zhong, Yanwen Zou, Yuxin Du, Jiting Liu, Encheng Gu, Bo Zhao

arXiv:2507.00416v3cs.ROcs.CV

TL;DR

Existing VLA models lack precise 3D spatial understanding, while explicit 3D inputs can require additional sensors or estimation models. Evo-0 implicitly injects VGGT-derived geometric features into VLA visual representations while retaining RGB inputs. Across simulation, real-world, and disturbance evaluations, the approach consistently improves performance over baseline VLA models.

  • Problem

    Existing VLA models lack precise 3D spatial understanding because their training relies mainly on 2D image-text alignments and RGB observations.

  • Method

    Evo-0 uses VGGT-derived 3D tokens and a lightweight cross-attention fuser to inject geometric priors into VLA visual representations from RGB images.

  • Results

    Evo-0 consistently improves spatial understanding and outperforms baseline VLA models across simulation, real-world, and disturbance evaluations.

  • Takeaways & Limitations

    Implicit geometric-prior integration offers a simple and efficient way to enhance spatial understanding in VLA systems without explicit 3D inputs.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as a promising framework for enabling generalist robots capable of perceiving, reasoning, and acting in the real world. These models usually build upon pretrained Vision-Language Models (VLMs), which excel at semantic understanding due to large-scale image and text pretraining. However, existing VLMs typically lack precise spatial understanding capabilities, as they are primarily tuned on 2D image-text pairs without 3D supervision. To address this limitation, recent approaches have incorporated explicit 3D inputs such as point clouds or depth maps, but this necessitates additional depth sensors or pre-trained depth estimation models, which may yield defective results. In contrast, our work introduces a plug-and-play module that implicitly incorporates 3D geometry features into VLA models by leveraging an off-the-shelf visual geometry foundation model. This integration provides the model with depth-aware visual representations, improving its ability to understand the geometric structure of the scene and the spatial relationships among objects from RGB images alone. We evaluate our method on a set of spatially challenging tasks in both simulation and the real world. Extensive evaluations show that our method significantly improves the performance of state-of-the-art VLA models across diverse scenarios.

I. INTRODUCTION

VLA models achieve strong semantic performance but lack precise 3D spatial understanding because their training relies mainly on 2D image-text data and RGB-only robotic observations. Evo-0 addresses this gap by implicitly injecting geometric features into VLA models and evaluating the result across simulation, real-world, and disturbance tasks.

  • VLA models struggle with precise geometric and spatial relationships needed for effective physical interaction.
  • 2D image-text pretraining and RGB-only robotic fine-tuning leave existing VLA models without sufficient 3D spatial information.
  • Explicit depth-based approaches can require additional depth sensors or depth-estimation networks, creating practical challenges.
  • Evo-0 uses a spatial encoder trained on 2D–3D paired data to extract geometric features from RGB images and fuse them with VLM visual tokens.
  • Evo-0 is evaluated on 5 simulation tasks, 5 real-world manipulation tasks, and 5 disturbance conditions, consistently improving spatial understanding over state-of-the-art VLA models.

II. RELATED WORK

Related work develops general-purpose VLA policies from pretrained vision-language models and increasingly incorporates 3D information for spatial reasoning. Evo-0 instead integrates VGGT while retaining RGB inputs, avoiding explicit 3D inputs and their deployment constraints.

  • Vision-Language-Action Models: VLA models extend pretrained VLMs with action prediction to support general-purpose robot policies and few-shot generalization.
  • Vision-Language-Action Models: OpenVLA, π0, and GR00T represent different strategies for scaling VLA policies across demonstrations, continuous action prediction, and multimodal co-training.
  • Robot Learning with 3D Information: Most existing VLA models rely primarily on 2D visual inputs, limiting spatial reasoning in complex manipulation tasks.
  • Robot Learning with 3D Information: Current 3D-aware VLA methods often require explicit depth maps or point clouds from specialized sensors or auxiliary estimation models.
  • Robot Learning with 3D Information: Evo-0 integrates VGGT into existing VLA models to model 3D structure from RGB images without additional sensors or depth-estimation modules.

A. Preliminaries

VLA systems combine pretrained multimodal understanding with action prediction, while visual geometry foundation models provide 3D structural cues from image observations. Evo-0 combines these streams through a 2D image encoder, VGGT spatial encoder, fusion layer, VLM, and action module.

  • Vision-Language-Action Models: The VLM provides general-purpose semantic encoding, while an action expert maps fused representations to robot control commands.
  • Vision-Language-Action Models: VLA models encode visual observations and language instructions with a VLM, then combine the embedding with robot states to generate control commands.
  • Vision-Language-Action Models: VLA models reuse pretrained multimodal understanding to improve semantic grounding, modality fusion, and generalization beyond task-specific imitation learning.
  • Visual Geometry Foundation Models: VGGT accepts an arbitrary number of image views and alternates frame-wise and global self-attention to model spatial consistency.
  • Evo-0 Architecture: Evo-0 processes RGB inputs through 2D and VGGT encoders, fuses their features, and passes the spatially enriched representation to a VLM and action module.
  • Visual Geometry Foundation Models: Given multi-view RGB images, VGGT predicts camera poses, depth maps, point maps, and 3D point tracks for each frame.

B. Proposed VLA Architecture

Evo-0 augments a VLA with geometry-aware features extracted from RGB images by a spatial encoder, then fuses them into the visual-language pipeline. The design uses lightweight adaptation to preserve the pretrained backbone while adding spatial information.

  • Design motivation: The approach implicitly adds spatial awareness without explicit point clouds, depth maps, or additional depth sensors.This design addresses the sensitivity and preprocessing requirements associated with explicit 3D inputs.
  • Geometry-aware representation: Evo-0 uses VGGT as a spatial encoder to extract 3D tokens from RGB-derived visual representations.The tokens provide depth-aware context, temporally consistent object trajectories, and spatial correspondences across views.
  • Feature fusion: A lightweight cross-attention fuser combines Vision Transformer tokens with VGGT-derived geometry tokens.The 2D tokens serve as queries, while the geometry tokens provide keys and values before the fused representation is passed onward.
  • Evaluation context: The architecture is evaluated in simulation using five RLBench tasks requiring precise spatial manipulation, with per-task and average success rates.The experiments use multi-task training and official RLBench success criteria.
  • Feature fusion: Each view is processed independently, and the resulting cross-attended tokens are concatenated into the fused output.The projection matrices are trainable and shared across views.
  • Action prediction: The fused tokens are forwarded to PaliGemma, which jointly attends to geometry-enhanced visual and language tokens to predict actions.The action prediction pipeline therefore receives both visual-language and geometry-aware representations.

IV. EXPERIMENTS

The experiments evaluate Evo-0 in simulation and real-world settings, including baseline comparisons, hyperparameter analyses, and precise manipulation tasks. The section reports consistent improvements across the evaluated settings.

  • Evaluation scope: Evo-0 is evaluated in both simulation and real-world settings to assess spatial perception for manipulation.The evaluation spans multiple task environments rather than a single benchmark.
  • Simulation evaluation: The experiments compare Evo-0 with competitive baselines in RLBench simulation tasks focused on fine-grained grasping and transport.The section also analyzes key hyperparameters affecting manipulation accuracy.
  • Evaluation scope: The evaluation program includes real-world tests of precise manipulation in addition to simulation and hyperparameter studies.This broad setup is used to examine performance across different experimental conditions.

A. Simulation Experiments

The simulation study evaluates Evo-0 on five precise-manipulation tasks and compares it with OpenVLA-OFT and π0. Evo-0 achieves the highest reported average success rate, with gains over both baselines.

  • Task setup: The RLBench suite contains five tasks spanning precise grasping, placement, transport, and motion under height or translation variation.The tasks use a Franka Panda with front, wrist, and overhead cameras, and training collects 100 demonstration trajectories per task.
  • Real-world evaluation: The real-world task setup includes cylinder centering, peg-in-hole insertion, bottle grasping, can pick-and-place, and transparent-object pick-and-place.These tasks are presented as a separate five-task real-world evaluation set.
  • Inference speed: Evo-0 operates at 6.94 Hz versus 11.3 Hz for π0 on an NVIDIA RTX 4090 GPU.The lower control frequency is attributed primarily to the additional VGGT image-encoding cost, while remaining within the stated real-time control range.

B. Hyperparameter Experiments

The hyperparameter study examines training duration and executed horizon on the five RLBench tasks. Evo-0 improves as training proceeds and shows training-efficiency advantages relative to π0.

  • Executed horizon: The study evaluates executed horizon as a second factor affecting task success rates.Evo-0 trained with 20k steps is tested under five different horizon settings.
  • Training steps: Evo-0 success rates consistently improve and gradually converge across checkpoints from 15k to 25k training steps.The study evaluates checkpoints saved at 15k, 20k, 23k, and 25k steps.
  • Training steps: Evo-0 trained for 15k steps already outperforms π0 trained for 20k steps.This comparison is presented as evidence of higher training efficiency for Evo-0.

C. Real-World Experiments

The real-world evaluation covers five spatially demanding manipulation tasks, including precise alignment, grasping, placement, and transparent-object interaction. Results are assessed with task completion metrics, including fine-grained scoring for cylinder centering, and qualitative execution sequences illustrate manipulation performance.

  • Task Setup: Five real-world tasks test fine-grained alignment, peg insertion, selective grasping, can placement, and transparent-object manipulation.The tasks impose low tolerance for spatial error; transparent objects add glare and RGB-perception challenges.
  • Evaluation: Task 1 uses a five-point average score rather than binary completion to capture differences in cylinder-centering precision.The innermost ring scores 5, outer rings score 4 to 1, and failed grasps score 0.
  • Qualitative Results: The qualitative sequences visualize step-by-step execution across all five real-world evaluation tasks.These visualizations complement the quantitative task results.
  • Qualitative Results: Evo-0 reliably performs stable grasping and precise alignment in cylinder-centering and peg-in-hole tasks, while π0 often fails to establish a proper cylinder grasp.The comparison is reported from qualitative execution visualizations.

D. Robustness Experiments

The robustness study progressively introduces five disturbance conditions outside the training distribution to test generalization to environmental variation and unseen distractors. Evo-0 outperforms π0 across object selection, full manipulation, background, position, and height variations in the reported comparisons.

  • Experimental Setup: Five out-of-distribution disturbances test unseen distractors, background color, target position, target height, and camera viewpoint variation.The setup begins from a simplified can pick-and-place task and progressively adds disturbances.
  • Unseen Distractor Object: Evo-0 reaches 100% object-selection success versus π0’s 60%, and 70% full grasp-and-place success versus π0’s 20%, with an unseen distractor.The condition places the target can and distractor bottle on opposite sides of the shelf.
  • Target Position Variation: Evo-0 achieves approximately 10% higher success rates than π0 under 20 mm and 30 mm target-position shifts.The mat is shifted backward along the shelf across separate test groups.
  • Target Height Variation: Evo-0 exceeds π0 by 20% and 10% under 10 mm and 15 mm target-height increases, respectively, although both models degrade as height increases.The results indicate stronger robustness under vertical perturbations.

V. CONCLUSION

The paper explores implicit 3D representations for improving spatial understanding in VLA models. It uses VGGT features to inject geometric priors without explicit 3D inputs and reports significant improvements over baseline models across challenging simulation and real-world tasks.

  • Conclusion: Evo-0 uses VGGT features trained on large-scale 2D–3D paired data to inject geometric priors into VLA models without explicit 3D inputs.The approach is presented as an implicit strategy for enhancing spatial understanding.
  • Conclusion: Extensive experiments across spatially challenging simulation and real-world tasks show significant improvements over baseline models.The conclusion describes the approach as a simple and efficient solution for enhancing spatial understanding in VLA systems.
Loading 2507.00416v3…