PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoVLA: Empowering 3D Representations in Vision-Language-Action Models

Lin Sun, Bin Xie, Yingfei Liu, Hao Shi, Tiancai Wang, Jiale Cao

arXiv:2508.09071v2cs.RO

TL;DR

VLA models often rely on 2D inputs, limiting their use of 3D geometric information for robotic manipulation. GeoVLA independently encodes visual-language and point-cloud features, fuses them in a spatial-aware action expert, and reports strong simulation and real-world performance with robustness to spatial variations.

  • Problem

    Predominantly 2D VLA inputs overlook 3D geometric information needed for depth cues, spatial understanding, and viewpoint robustness.

  • Method

    GeoVLA uses a VLM for vision-language features, PEN for independently encoded point-cloud geometry, and 3DAE to fuse both modalities for action generation.

  • Results

    GeoVLA achieves state-of-the-art results on LIBERO and ManiSkill2 and reports 86.3% average success across eight real-world tasks, outperforming π0 by 28.8%.

  • Takeaways & Limitations

    The reported results show robustness to height adaptation, scale awareness, viewpoint generalization, and other real-world spatial variations.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as a promising approach for enabling robots to follow language instructions and predict corresponding actions. However, current VLA models mainly rely on 2D visual inputs, neglecting the rich geometric information in the 3D physical world, which limits their spatial awareness and adaptability. In this paper, we present GeoVLA, a novel VLA framework that effectively integrates 3D information to advance robotic manipulation. It uses a vision-language model (VLM) to process images and language instructions,extracting fused vision-language embeddings. In parallel, it converts depth maps into point clouds and employs a customized point encoder, called Point Embedding Network, to generate 3D geometric embeddings independently. These produced embeddings are then concatenated and processed by our proposed spatial-aware action expert, called 3D-enhanced Action Expert, which combines information from different sensor modalities to produce precise action sequences. Through extensive experiments in both simulation and real-world environments, GeoVLA demonstrates superior performance and robustness. It achieves state-of-the-art results in the LIBERO and ManiSkill2 simulation benchmarks and shows remarkable robustness in real-world tasks requiring height adaptability, scale awareness and viewpoint invariance.

1 Introduction

GeoVLA addresses the limited 3D geometric awareness of predominantly 2D VLA models with parallel visual-language and point-cloud processing. Its 3D-enhanced action generation achieves strong benchmark and real-world results with robustness to spatial variations.

  • Current VLA models mainly rely on 2D visual inputs, overlooking geometric information that provides depth cues and spatial understanding.
  • GeoVLA processes vision-language and 3D geometric information through parallel branches, preserving VLM knowledge while enabling multimodal action generation.
  • GeoVLA combines a Point Embedding Network with a 3D-enhanced Action Expert to generate precise action sequences from visual and point-cloud features.
  • 2.4% and 11% improvements over OpenVLA-OFT and Dita are reported on LIBERO and ManiSkill2, respectively.
  • 86.3% average success across eight real-world tasks exceeds π0 by 28.8%, alongside robustness to height, scale, and viewpoint variations.

2 Related Works

Related work explores 3D representations in VLA models and geometry injection into action heads, but existing approaches face alignment, data, or adaptation challenges. GeoVLA independently encodes point clouds and explicitly fuses them with visual-language features in a specialized action expert.

  • Recent VLA research explores 3D representations to improve generalizable robotic manipulation.
  • PointVLA injects point-cloud features into an action policy but can hinder adaptation to the newly introduced point-cloud modality.
  • GeoVLA independently processes 3D inputs with PEN and models visual-language–point-cloud interactions through modality-specific experts in 3DAE.

3 Methodology

GeoVLA processes RGB-language and depth-derived point-cloud inputs in parallel, then fuses their representations for action generation. PEN extracts end-effector-centered geometric features, while 3DAE uses diffusion and modality-aware experts to generate action chunks.

  • 3.1 Problem Definition: GeoVLA combines visual observations, language instructions, and point clouds to condition sequence-level robot actions.
  • 3.2 Overview: A VLM extracts vision-language features while PEN independently encodes depth-derived point clouds into geometric features for multimodal action generation.
  • 3.3 Point Embedding Network: PEN uses a dual-path architecture with large-kernel convolutions for geometric tokens and RoPE for positional information.
  • 3.3 Point Embedding Network: PEN selects the end-effector token as an anchor and outputs its updated representation as the geometric feature supplied to the action expert.
  • 3.4 3D-enhanced Action Expert: 3DAE applies a diffusion-transformer action head to concatenated visual-language and geometric tokens, progressively recovering action sequences during inference.
  • 3.4 3D-enhanced Action Expert: A modality-specific mixture-of-experts design and static routing strategy address imbalance between pretrained visual-language and newly initialized point-cloud branches.

4 Simulation Experiments

GeoVLA is evaluated on LIBERO and ManiSkill2, where it achieves leading success rates and ablations show benefits from its point encoder, token selection, positional encoding, and MoE action expert.

  • LIBERO: GeoVLA achieves an average LIBERO success rate of 97.7%, outperforming CogACT at 93.2% and OpenVLA-OFT at 95.3%.
  • ManiSkill2: On ManiSkill2, GeoVLA reaches 77% overall success, surpassing CogACT at 69% and Dita at 66%.
  • ManiSkill2: GeoVLA reaches 45% on PickClutterYCB versus Dita’s 36%, while point-cloud observations support spatial awareness under greater object diversity.
  • Ablation Study: The PEN encoder achieves 97.7% success, exceeding a 3-layer MLP at 95.8% and PointNet at 95.2%.
  • Ablation Study: End Effector token selection achieves 97.7%, outperforming Max at 96.3% and Mean at 95.9%.
  • Ablation Study: RoPE increases success from 95.4% to 97.7%, while the MoE-based 3DAE improves performance from 96.0% to 97.7%.

5 Real-world Experiments

Real-world experiments evaluate GeoVLA on basic and 3D-aware manipulation tasks, including controlled changes in height, scale, viewpoint, and scene support. GeoVLA achieves an overall 86.3% success rate and shows stronger robustness than comparison methods under these variations.

  • Task Definition: The evaluation covers basic manipulation and 3D-aware tasks that test spatial perception and precise manipulation.Tasks include Pick Carrot, Stack Block, Stack Cup, Insert Circle, Hang Cup, Put Basketball, Cover Matryoshka, and Put Hairclip.
  • Main Results: GeoVLA achieves 95.0% average success on basic tasks, 77.5% on 3D-aware tasks, and 86.3% overall.The overall rate exceeds π0 by 28.8% and CogACT by 10.0%.
  • Variant Results: At unseen basket heights, GeoVLA maintains reasonable performance and reaches 60% success at the highest position, while 2D-VLA baselines drop sharply.CogACT reaches 20% and 30% at the two lower tested positions, whereas GeoVLA remains robust across the variations.
  • Variant Results: GeoVLA consistently achieves higher success rates than other 2D-VLA models when the Matryoshka doll is enlarged.The evaluation compares a medium training size with smaller, slightly larger, and significantly larger inference sizes.
  • Variant Results: When the camera shifts to 45°, GeoVLA maintains high performance while CogACT drops significantly.Camera viewpoints are evaluated at 15°, 30°, and 45°.
  • Variant Results: With the sponge mat removed, GeoVLA achieves more stable and successful carrot grasps than methods that grasp from above.Removing the mat lowers the carrot placement during inference.

6 Conclusion

The conclusion presents GeoVLA as a 3D-enhanced VLA that combines point-cloud geometry with vision-language features for robotic action generation.

  • 6 Conclusion: GeoVLA uses a dual-path Point Embedding Network to extract compact geometric representations from noisy point clouds.The network selects an end-effector-position token as an anchor for geometric features.
  • 6 Conclusion: A diffusion-based 3D-enhanced Action Expert processes geometric and vision-language features together.

A Simulation Environments

The simulation evaluation uses LIBERO and ManiSkill2 to cover diverse robotic manipulation settings. LIBERO spans five task suites, while ManiSkill2 focuses primarily on pick-and-place scenarios in 3D space.

  • Simulation Environments: The simulation benchmarks are LIBERO and ManiSkill2.
  • Simulation Environments: LIBERO contains five task suites focusing on different skills.
  • Simulation Environments: ManiSkill2 evaluates pick-and-place tasks requiring objects to be moved to locations marked in 3D space.

B Visualization of Simulation Results

The simulation visualizations present qualitative results from GeoVLA across LIBERO and ManiSkill2 tasks. They illustrate object recognition and interaction across different tasks and environments.

  • Qualitative Findings: The visualizations show accurate object recognition and interaction across different tasks and environments.
  • LIBERO: Figure 6 presents GeoVLA qualitative results on the five LIBERO task suites.
  • ManiSkill2: Figure 7 presents GeoVLA qualitative results on five ManiSkill2 tasks.
  • Real-world Setup: The real-world setup includes a depth camera and a robotic arm.

C Real-World Environments

GeoVLA is evaluated on base manipulation tasks and more challenging 3D-aware tasks in real-world experiments using a WidowX-250s arm and a RealSense-435i depth camera. The task set targets basic pick-and-place, spatial reasoning, object-pose estimation, and robustness or generalization.

  • Real-world experiments use a six-degree-of-freedom WidowX-250s arm and a RealSense-435i depth camera positioned approximately 0.8 meters away.
  • Base Tasks: Four base tasks evaluate fundamental pick-and-place capabilities: Pick Carrot, Stack Block, Stack Cup, and Insert Circle.These tasks are jointly trained to improve training efficiency and facilitate shared skill learning.
  • The real-world evaluation divides tasks into base tasks, 3D-aware tasks, and inference-time variations for robustness and generalization.
  • 3D-Aware Tasks: The 3D-aware task set includes Hang Cup, Put Basketball, Cover Matryoshka, and Put Hairclip, requiring accurate object-pose and spatial-relationship perception.

D Visualization of Real-World Results

Qualitative results show high precision and success across GeoVLA’s base and 3D-aware tasks. The visualizations particularly highlight geometric reasoning for missing-RGB operation, position changes, object scaling, and camera viewpoint changes.

  • GeoVLA demonstrates high precision and success rates across both base and 3D-aware real-world tasks.
  • For Put Hairclip, point-cloud inputs provide detailed geometric structures that support robust performance without RGB inputs.
  • GeoVLA outperforms π0 and CogACT on Put Basketball variations by recognizing different 3D object positions.
  • GeoVLA maintains strong performance when doll sizes change and remains robust to camera viewpoint changes.

E More Experiments on Generalization

Additional experiments examine GeoVLA under background and lighting changes and across task variations involving position, scale, and viewpoint. The reported visualizations indicate robustness across these generalization conditions.

  • Background and lighting variations are evaluated on three manipulation tasks using success-rate measurements.
  • GeoVLA performs well when basketball positions are higher or lower in Put Basketball variation tasks.
  • GeoVLA performs well when Matryoshka doll sizes are larger or smaller in Cover Matryoshka variation tasks.
  • GeoVLA performs better than π0 and CogACT when the camera viewpoint changes in Stack Block variations.
  • GeoVLA shows robustness to background and light changes on Stack Cup, Insert Circle, and Stack Block.
Loading 2508.09071v2…