PaperlayerView exact revision on arXiv

Source-linked AI summary

BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models

Peiyan Li, Yixiang Chen, Hongtao Wu, Xiao Ma, Xiangnan Wu, Yan Huang, Liang Wang, Tao Kong, Tieniu Tan

arXiv:2506.07961v2cs.ROcs.AI

TL;DR

Existing VLA approaches provide effective manipulation learning but often use 2D inputs or fail to exploit the spatial structure of 3D actions efficiently. BridgeVLA projects 3D observations into multiple 2D views, predicts 2D heatmaps, and pre-trains heatmap-based object grounding before policy learning. It outperforms baselines across simulation and real-robot evaluations, including 96.8% success on 10+ tasks with only 3 trajectories per task.

  • Problem

    Existing 3D VLA methods do not fully leverage 3D spatial structure, while 3D action fine-tuning is misaligned with the 2D inputs used during VLM pre-training.

  • Method

    BridgeVLA projects 3D point clouds into multiple orthographic 2D images, predicts 2D action heatmaps, and pre-trains text-conditioned heatmap object grounding.

  • Results

    BridgeVLA outperforms state-of-the-art baselines across three simulation benchmarks and surpasses a state-of-the-art method by 32% on average in real-robot experiments.

  • Takeaways & Limitations

    BridgeVLA learns 3D manipulation efficiently and effectively while generalizing across visual disturbances, unseen categories, and other out-of-distribution settings.

  • Takeaways & Limitations

    Real-robot Category performance remains limited, with failures linked to differences between third-person pre-training views and robot projection images and between object localization and manipulation keypoints.

Abstract

from arXiv · show

Recently, leveraging pre-trained vision-language models (VLMs) for building vision-language-action (VLA) models has emerged as a promising approach to effective robot manipulation learning. However, only few methods incorporate 3D signals into VLMs for action prediction, and they do not fully leverage the spatial structure inherent in 3D data, leading to low sample efficiency. In this paper, we introduce BridgeVLA, a novel 3D VLA model that (1) projects 3D inputs to multiple 2D images, ensuring input alignment with the VLM backbone, and (2) utilizes 2D heatmaps for action prediction, unifying the input and output spaces within a consistent 2D image space. In addition, we propose a scalable pre-training method that equips the VLM backbone with the capability to predict 2D heatmaps before downstream policy learning. Extensive experiments show the proposed method is able to learn 3D manipulation efficiently and effectively. BridgeVLA outperforms state-of-the-art baseline methods across three simulation benchmarks. In RLBench, it improves the average success rate from 81.4% to 88.2%. In COLOSSEUM, it demonstrates significantly better performance in challenging generalization settings, boosting the average success rate from 56.7% to 64.0%. In GemBench, it surpasses all the comparing baseline methods in terms of average success rate. In real-robot experiments, BridgeVLA outperforms a state-of-the-art baseline method by 32% on average. It generalizes robustly in multiple out-of-distribution settings, including visual disturbances and unseen instructions. Remarkably, it is able to achieve a success rate of 96.8% on 10+ tasks with only 3 trajectories per task, highlighting its extraordinary sample efficiency. Project Website:https://bridgevla.github.io/

1 Introduction

BridgeVLA addresses inefficient and misaligned 3D VLA learning by aligning 3D observations and action outputs within a shared 2D image space. Its experiments show strong simulation and real-world performance with high sample efficiency.

  • Motivation: Existing 3D VLA methods often tokenize actions without spatial structure, while 3D inputs also misalign with VLM pre-training inputs.These issues can reduce the use of 3D structural priors and create a distributional shift from original VLM pre-training.
  • Approach: BridgeVLA projects 3D point clouds into multiple orthographic 2D images to align observations with the pre-trained VLM backbone.The projected views preserve a shared image-based representation for downstream manipulation learning.
  • Approach: BridgeVLA predicts 2D heatmaps for translational actions, aligning input observations and output actions within a unified spatial structure.The heatmaps are generated from projection-image tokens and share the images’ resolution.
  • Approach: A scalable object-grounding pre-training method equips the VLM to predict text-conditioned heatmaps before action-policy fine-tuning.This pre-training addresses the incompatibility between token-sequence pre-training and heatmap-based action prediction.
  • Results: 88.2% average success in RLBench versus 81.4% for baselines, and 64.0% versus 56.7% in COLOSSEUM, demonstrate stronger simulation performance.BridgeVLA also surpasses comparing baselines in GemBench and a state-of-the-art method by 32% on average in real-robot experiments.
  • Results: 96.8% success across 10+ real-robot tasks with only 3 trajectories per task highlights BridgeVLA’s sample efficiency.The real-world evaluations also span visual perturbations and unseen object categories, demonstrating generalization across multiple out-of-distribution settings.

2 Related Work

Prior work includes effective 2D VLA policies, sample-efficient 3D manipulation policies, and emerging 3D VLA models. These approaches differ in how they represent visual observations, spatial structure, and actions.

  • Language-Conditioned Visuomotor Policies: Language-conditioned visuomotor policies commonly use transformers to process 2D visual inputs and generate 3D manipulation actions.Pre-trained VLMs have become popular foundations for learning complex manipulation tasks.
  • Language-Conditioned Visuomotor Policies: 2D image-based policies often require hundreds of trajectories per task, whereas 3D policies exploit spatial structure for more efficient learning.Point-cloud methods represent 3D observations directly; Act3D, for example, predicts translational actions by classifying observation points.
  • 3D Vision-Language-Action Models: 3D VLA models remain relatively under-explored and include approaches for 3D reasoning, multimodal goal generation, planning, and robotic representation learning.Examples include 3D-VLA, Lift3D, and FP3, which combine language with different 3D or multimodal representations.

3 BridgeVLA

BridgeVLA uses a dual-phase recipe that aligns 3D manipulation inputs and action outputs within a shared 2D image space. It pre-trains heatmap prediction, then fine-tunes on projected point-cloud views to predict translation and other action components.

  • 3 BridgeVLA: BridgeVLA maps observations and language instructions to 6-DoF pose, gripper, and collision actions at the next key frame.The action includes an end-effector pose, target gripper state, and collision flag.
  • 3.2 2D-Heatmap Pre-training: During pre-training, the model predicts heatmaps that localize text-specified objects in 2D images.Ground-truth heatmaps are constructed from object-detection bounding boxes and fused across objects by averaging and normalization.
  • 3.2 2D-Heatmap Pre-training: BridgeVLA uses learned convex upsampling to reconstruct input-resolution heatmaps from spatially arranged image tokens.The heatmap pipeline is trained with cross-entropy loss to localize all objects of interest.
  • 3.3 3D Action Fine-tuning: During fine-tuning, three orthographic point-cloud projections serve as VLM inputs, while heatmaps estimate translation and an MLP predicts rotation, gripper, and collision outputs.The three views are top, front, and right; translation is estimated by back-projecting view heatmaps to 3D point grids.
  • 3.3 3D Action Fine-tuning: A coarse-to-fine strategy crops and zooms the point cloud around the initial predicted translation before a second action-prediction pass.The second-pass prediction is used for execution.
  • 3.3 3D Action Fine-tuning: Fine-tuning combines translational, rotational, gripper, and collision losses, with random rigid-body transformations applied to improve geometric robustness.Translation and rotation use cross-entropy losses, while gripper and collision prediction use binary cross-entropy.

4 Experiments

Experiments evaluate BridgeVLA across simulation benchmarks, perturbation and semantic generalization settings, real-robot tasks, and targeted ablations. Results show strong benchmark performance and sample efficiency, while identifying weaknesses in long-horizon tasks, category generalization, projection-view coverage, and direct position regression.

  • 4.1.1 Experiments on RLBench: 88.2% average success rate across 18 RLBench tasks establishes BridgeVLA as the benchmark’s state-of-the-art method.Its average rank is 1.9 across all tasks.
  • 4.1.1 Experiments on RLBench: BridgeVLA improves the best RLBench baseline by large margins on Insert Peg, 88.0% vs 40.0%, and Sort Shape, 60.8% vs 35.0%.These tasks require precise alignment between manipulated objects and targets.
  • 4.1.2 Experiments on COLOSSEUM: 7.3% average-success-rate improvement over the state-of-the-art baseline on COLOSSEUM demonstrates robustness across 14 visual perturbations.BridgeVLA ranks best in 13 of the 14 evaluated perturbations.
  • 4.1.3 Experiments on GemBench: BridgeVLA consistently outperforms baselines across four GemBench evaluation settings, achieving state-of-the-art results in L2 and L3 but limited performance in L4.L4 tasks comprise multiple sub-tasks; future work proposes language-model-based long-horizon decomposition.
  • 4.2 Real-Robot Experiments: BridgeVLA outperforms RVT-2 in all seven real-robot settings and handles visual disturbances and novel instructions robustly.The settings include one basic condition and six generalization conditions.
  • 4.3 Ablation Studies: 31.4% versus 88.2% average success shows that direct position regression is substantially worse than heatmap-based action prediction.The direct-regression ablation is also harder to train and more sensitive to hyperparameters.

5 Conclusions & Future Work

BridgeVLA is a 3D VLA model built on a pre-trained VLM that aligns 3D inputs and action outputs in a unified 2D image space. The paper reports efficient, effective manipulation learning and proposes broader pre-training and expressive action decoding as future directions.

  • BridgeVLA converts 3D inputs into 2D images aligned with the pre-trained VLM's image inputs.
  • It aligns observations and actions through 2D heatmap prediction in a unified 2D image space.
  • A scalable pre-training method equips the VLM with heatmap prediction before action fine-tuning.
  • Experiments show efficient and effective 3D manipulation learning in simulation and the real world.
  • Future work includes pre-training for semantic segmentation and keypoint detection and incorporating diffusion-based action decoding.

A Training & Inference Details

Training uses frozen SigLIP vision and language embeddings, while compute varies across pre-training, simulation fine-tuning, and real-world fine-tuning. Inference on an RTX 4090 takes 0.21 seconds end-to-end from point cloud input to action output.

  • The SigLIP vision encoder and language token embeddings remain frozen during pre-training and fine-tuning.
  • Pre-training uses 8 NVIDIA A100 GPUs for 3,800 steps, taking approximately 2 hours.
  • RLBench and COLOSSEUM fine-tuning each use 48 NVIDIA H100 GPUs for 83,000 steps, taking approximately 20 hours.
  • GemBench fine-tuning uses 40 NVIDIA A100 GPUs for 50 epochs, taking approximately 2.1 hours.
  • Real-world fine-tuning uses 8 NVIDIA A100 GPUs for 300 epochs, taking approximately 1.5 hours.
  • 0.21 seconds is the average end-to-end inference time from point cloud input to action output on an NVIDIA RTX 4090.

B Simulation Experiments

Across simulation and real-robot experiments, BridgeVLA uses the same key-frame selection strategy as PerAct. A timestep qualifies when the robot is stationary, the gripper state changes, or the episode ends.

  • BridgeVLA adopts PerAct's key-frame selection strategy for all simulation and real-robot experiments.
  • A timestep is labeled a key frame when the robot is stationary, the gripper state changes, or it is the episode's final state.
  • The robot is considered stationary when every joint's absolute velocity falls below 0.1 rad/s.

B.2 Data

The experiments cover RLBench, COLOSSEUM, and GemBench, with GemBench emphasizing hierarchical generalization. Data regimes vary: RLBench and COLOSSEUM use demonstration augmentation, whereas GemBench uses only trajectory keyframes.

  • RLBench evaluation selects 18 tasks spanning non-prehensile manipulation, pick-and-place, and high-precision insertion.
  • COLOSSEUM evaluates generalization to 12 unseen perturbation types, including changes in texture, color, size, backgrounds, lighting, distractors, and camera poses.
  • GemBench trains on 16 tasks with 31 variations covering seven fundamental action primitives and tests 44 tasks with 92 variations.
  • GemBench organizes its test set into four increasingly challenging settings.
  • Unlike RLBench and COLOSSEUM, GemBench uses only trajectory keyframes without demonstration augmentation.

B.3 Detailed Results on COLOSSEUM

COLOSSEUM results use reported baselines from the original paper and independently trained BridgeVLA and RVT-2 models. Performance is summarized across tasks and perturbations using repeated evaluations.

  • BridgeVLA and RVT-2 were trained and evaluated through the authors’ own processes.
  • Results for R3M-MLP, MVP-MLP, RVT, and PerAct were taken from the original COLOSSEUM paper.
  • BridgeVLA and RVT-2 results report average success rates and variance across three test repetitions for each task and perturbation.

B.4 Detailed Results on GemBench

GemBench evaluation reports per-task success rates across four settings, using baseline results sourced from prior work. Five random seeds and repeated trials reduce statistical variance.

  • Per-task success rates are reported for GemBench’s four evaluation settings.
  • Baseline GemBench results are sourced from the original GemBench study.
  • The evaluation uses 5 random seeds and 20 trials per task variation for every seed.

C.2 Basic Setting

In the basic setting, BridgeVLA is compared with representative manipulation policies under scenes resembling training data but with modified object layouts. The section also examines spatial and generalization behavior.

  • Basic Setting: BridgeVLA is compared with four representative manipulation methods in a scene similar to training data, with object layouts modified.
  • Basic Setting: SpatialVLA failed on nearly all tasks with 10 trajectories per task and remained behind BridgeVLA after adding 40 more trajectories per task.
  • Basic Setting: π0 failed with only 10 trajectories per task and often failed during online testing despite performing well on the training set.
  • Basic Setting: ACT performs well in densely covered regions but often fails near workspace boundaries, consistent with its Gaussian action prior.
  • Basic Setting: RVT-2 performs best among the baselines but is less robust than BridgeVLA, with failures in precise pickup and placement.
  • Generalization Settings: The evaluation covers six generalization settings, including visual disturbances and novel object-skill combinations.
  • Pre-training Retention: BridgeVLA retains object grounding after fine-tuning and does not forget its pre-training knowledge during 3D action fine-tuning.

C.5 Per-task Success Rate

The per-task results combine success-rate reporting across BridgeVLA’s basic and perturbed evaluations with visualizations of tasks, data, settings, and predictions. These materials document both evaluation coverage and model behavior.

  • Basic Setting: BridgeVLA’s basic-setting per-task success rates are reported in Table 12, with exceptionally high success rates using only 3 trajectories per task.
  • COLOSSEUM: COLOSSEUM success rates are reported for BridgeVLA and RVT-2 under different perturbations.
  • GemBench: GemBench per-task success rates are reported separately for Levels 1, 2, 3, and 4.
  • Task Coverage: The visual materials include 18 RLBench tasks and two sets of real-robot rollouts.
  • Pre-training Data: Pre-training examples show original images, object bounding boxes, and ground-truth heatmaps.
  • Generalization Settings: The generalization visualizations cover distractors, lighting, backgrounds, height changes, unseen combinations, and novel categories.
  • Prediction Visualization: After fine-tuning, prediction visualizations place input images, predicted heatmaps, and ground truth in separate rows.
Loading 2506.07961v2…