PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenVLA: An Open-Source Vision-Language-Action Model

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, Chelsea Finn

arXiv:2406.09246v3cs.ROcs.LG

TL;DR

Robotic VLAs have been difficult to adopt because existing systems are largely closed and efficient fine-tuning remains underexplored. OpenVLA addresses these gaps with a 7B-parameter open-source model trained on diverse robot demonstrations and adapted through parameter-efficient methods. It outperforms RT-2-X by 16.5% absolute success rate across 29 tasks while supporting efficient adaptation and deployment.

  • Problem

    Existing VLAs are largely closed, while efficient fine-tuning methods for adapting them to new robotic tasks remain underexplored.

  • Method

    OpenVLA fine-tunes a pretrained vision-language model on 970k diverse robot demonstrations, representing robot actions as language-model output tokens.

  • Results

    16.5% absolute success rate is OpenVLA’s improvement over 55B-parameter RT-2-X across 29 evaluation tasks and multiple robot embodiments.

  • Takeaways & Limitations

    OpenVLA supports open research and practical adaptation through released code, checkpoints, notebooks, LoRA fine-tuning, and quantized inference.

  • Takeaways & Limitations

    OpenVLA currently supports only single-image observations, excluding multiple images, proprioceptive inputs, and observation history.

Abstract

from arXiv · show

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

1 Introduction

OpenVLA addresses the limited generalization and accessibility of robotic policies by introducing an open-source VLA designed for broad manipulation and efficient adaptation. It combines diverse robot demonstrations with pretrained vision-language components and releases resources for training and deployment.

  • Existing learned manipulation policies struggle with scene distractors, novel objects, and unseen task instructions despite generalizing to some changed initial conditions.
  • Open-source, generalist VLAs are needed to support effective fine-tuning and adaptation in robotics.
  • 16.5% absolute success rate separates OpenVLA from RT-2-X across 29 evaluation tasks on WidowX and Google Robot embodiments.OpenVLA is a 7B-parameter model, whereas RT-2-X has 55B parameters.
  • 970k robot manipulation trajectories from Open-X Embodiment provide OpenVLA with diverse training across embodiments, tasks, and scenes.
  • LoRA and quantization enable compute-efficient OpenVLA fine-tuning and deployment on consumer-grade GPUs without compromising performance.The project also releases models, deployment and fine-tuning notebooks, and a scalable PyTorch codebase.

2 Related Work

Related work establishes generalist robot policies and VLAs as routes to applying pretrained vision-language models to manipulation. OpenVLA differs by directly fine-tuning a VLM to generate robot actions and by making the resulting generalist VLA open-source.

  • Visually-conditioned language models connect pretrained vision encoders with language models for tasks including visual question answering and object localization.
  • Generalist robot policies train across diverse datasets and robot embodiments, with Octo supporting control of multiple robots and fine-tuning to new setups.
  • OpenVLA directly fine-tunes a VLM to generate robot actions by treating actions as tokens, rather than stitching pretrained components to newly initialized policy modules.
  • Some VLM-based visuomotor policies require substantial architectural structure or calibrated cameras, limiting their applicability.
  • OpenVLA contributes an open-source generalist VLA, systematic fine-tuning investigation, and the first demonstration of parameter-efficient fine-tuning and quantization for VLAs.

3 The OpenVLA Model

OpenVLA fine-tunes a pretrained vision-language model for robot action prediction using diverse Open-X data and tokenized actions. Its design choices target broad embodiment, task, and scene coverage while leaving heterogeneous sensory inputs for future work.

  • OpenVLA is a 7B-parameter VLA trained on 970k robot demonstrations from the Open X-Embodiment dataset.
  • The model uses a visual encoder, projector, and 7B-parameter Llama 2 language-model backbone inherited from the Prismatic-7B VLM.
  • OpenVLA maps an observation image and language instruction to robot actions by discretizing continuous action dimensions into tokenizer-compatible output tokens.
  • The training mixture is curated from Open-X to balance robot embodiments, tasks, and scenes while using manipulation data with third-person cameras and single-arm end-effector control.
  • DROID data was removed after low action-token accuracy suggested that its diversity required a larger mixture weight or model.
  • Training across heterogeneous sensory inputs and action spaces remains future work, despite prior demonstrations that heterogeneous sensory inputs are feasible.
  • OpenVLA requires 15GB of GPU memory in bfloat16 during inference and runs at approximately 6Hz on one NVIDIA RTX 4090 without speed-up tricks.

4 The OpenVLA Codebase

The OpenVLA codebase supports scalable VLA training and integrates modern efficiency and deployment methods. Its evaluation tooling covers diverse robot-policy tasks and reports comparisons using standardized rollouts.

  • The PyTorch codebase scales from single-GPU fine-tuning to billion-parameter VLA training on multi-node GPU clusters.
  • Figure 3 evaluates OpenVLA against prior generalist policies across task categories covering multiple generalization axes and language conditioning.Average success rates ± StdErr use 170 total rollouts per approach.
  • The codebase supports Open-X training, HuggingFace AutoModel integration, LoRA fine-tuning, and quantized model inference.

5 Experiments

The experiments evaluate OpenVLA across robot platforms, generalization settings, downstream adaptation, and compute-efficient training and inference. OpenVLA is compared with generalist policies and from-scratch imitation learning under matched evaluations.

  • 5.1 Direct Evaluations on Multiple Robot Platforms: OpenVLA is evaluated out-of-the-box on WidowX and Google mobile-manipulator embodiments across visual, motion, physical, and semantic generalization.The evaluation also tests language conditioning with multiple objects and uses 170 rollouts for BridgeData V2 experiments.
  • 5.1 Direct Evaluations on Multiple Robot Platforms: OpenVLA performs comparably to RT-2-X on Google robot evaluations and significantly outperforms it on BridgeData V2 despite using 7B versus 55B parameters.Both models show robust behaviors with distractors, end-effector orientation, and recovery from insecure grasps.
  • 5.2 Data-Efficient Adaptation to New Robot Setups: OpenVLA achieves the highest aggregate performance after fine-tuning across Franka setups, while Diffusion Policy is competitive on narrower single-instruction tasks.OpenVLA and Octo perform better on diverse tasks involving multiple objects, distractors, and language conditioning.
  • 5.2 Data-Efficient Adaptation to New Robot Setups: OpenVLA is the only approach achieving at least 50% success across all tested downstream tasks, although Diffusion Policy produces smoother and more precise trajectories on narrower dexterous tasks.The fine-tuning comparison uses target datasets containing 10–150 demonstrations.
  • 5.3 Parameter-Efficient Fine-Tuning: LoRA achieves the best performance-compute trade-off, matching full fine-tuning while training only 1.4% of model parameters.Fine-tuning only the last layer or freezing the vision encoder performs poorly, whereas sandwich fine-tuning improves performance with lower GPU memory use than full fine-tuning.
  • 5.4 Memory-Efficient Inference via Quantization: 4-bit quantization matches bfloat16 inference performance while reducing OpenVLA’s GPU memory footprint by more than half.The comparison covers eight representative BridgeData V2 tasks and 80 rollouts per approach.

6 Discussion and Limitations

OpenVLA achieves strong cross-embodiment control and can be adapted to new robot setups, but its current inputs and inference speed limit broader deployment.

  • Limitations: OpenVLA currently supports only single-image observations, excluding multiple images, proprioception, and observation history.The authors identify flexible-input VLA fine-tuning as an important direction for future work.
  • Limitations: OpenVLA inference throughput must improve for high-frequency control setups such as ALOHA, which runs at 50Hz.Faster inference would also support evaluation on more dexterous, bimanual manipulation tasks.

A Data Mixture Details

The appendix documents the OpenVLA training-data mixture and directs readers to detailed evaluation sections for BridgeData V2 and fine-tuning experiments.

  • Data mixture: The OpenVLA data mixture mostly follows prior work, with a few additional datasets.Table 3 lists the datasets drawn from Open X-Embodiment.
  • Evaluation details: The appendix provides further details on BridgeData V2 WidowX evaluations and Franka-Tabletop and Franka-DROID fine-tuning evaluations.These experiments are discussed in Sections 5.1 and 5.2.

B.1.1 BridgeData V2 Evaluation Tasks

The BridgeData V2 WidowX suite tests generalist policies across diverse visual, motion, physical, semantic, and language-grounding shifts, with most tasks requiring horizontal reaching.

  • Evaluation design: 17 tasks comprise 5 visual, 2 motion, 3 physical, 4 semantic, and 3 language-grounding evaluations.All tasks introduce some distribution shift because the exact original objects could not be procured.
  • Evaluation design: The suite evaluates policies on start-state changes, distractors, altered object properties, unseen objects, and modified language instructions.The four OOD categories are visual, motion, physical, and semantic; language grounding varies prompts while fixing the initial state.
  • Task categories: Representative tasks include altered appearances, unseen distractors, changed object size or shape, elevated placement, and unseen target objects.Examples include a handcrafted paper pot, cluttered scenes, a smaller battery, a raised plate, and a skull windup toy.
  • Task setup: Only the easy eggplant task starts with the end-effector above the target; the other 16 tasks require horizontal reaching from a fixed location.This initial condition is combined with the suite’s distribution shifts to challenge the policies.
  • Results: OpenVLA achieves the strongest performance on most tasks and the highest aggregate success rate among the evaluated generalist policies.RT-2-X also performs well, while RT-1-X and Octo generally struggle with these generalization tasks.

B.2.1 Google Robot Evaluation Tasks

The Google robot suite evaluates generalist policies on in-distribution and increasingly difficult OOD tasks involving unseen objects, backgrounds, relations, and semantic concepts.

  • Evaluation design: The evaluation covers 12 tasks and 60 rollouts, with five in-distribution tasks followed by seven OOD tasks.Each rollout is scored as failure or success.
  • In-distribution tasks: In-distribution tasks include grasping cans, moving objects near one another, orienting a can upright, and opening a drawer.These tasks establish performance under familiar conditions.
  • OOD tasks: OOD tasks vary backgrounds, target objects, object relations, instructions, and semantic concepts.Examples include unseen fruits and cups, novel placement relations, patterned tablecloths, and celebrity photographs.
  • Results: The full results report mean success rate with standard error across the 60 rollouts.OpenVLA and RT-2-X significantly outperform RT-1-X and Octo overall, with overlapping error bars for the two leading policies.
  • Results: OpenVLA and RT-2-X complete every task at least two times out of five trials and perform comparably on this evaluation suite.RT-1-X and Octo often achieve no successes in several tasks.

B.3.1 Franka-Tabletop and Franka-DROID Tasks

The adaptation benchmark covers seven tasks across Franka-Tabletop and Franka-DROID, testing both in-distribution and out-of-distribution conditions. Tasks range from narrow single-instruction manipulation to multi-object language-grounded behaviors.

  • Task suites: Seven tasks span six Franka-Tabletop tasks and one Franka-DROID task, with 10–150 demonstrations collected per task.The Franka-Tabletop setup uses a Franka Emika Panda mounted on a table; Franka-DROID contains the Wipe Table task.
  • Task suites: The first three Franka-Tabletop tasks use single instructions, whereas the final three require selecting and manipulating the correct object from multiple objects using language.The multi-instruction tasks are Move <object> onto Plate, Knock <object> Over, and Cover <object> with Towel.
  • Evaluation conditions: In-distribution trials match training conditions, while OOD trials vary objects, backgrounds, distractors, positions, or orientations.Each Franka-Tabletop policy is evaluated with 10–12 in-distribution trials and 5–6 OOD trials.
  • Franka-Tabletop tasks: The six Franka-Tabletop tasks include placing a carrot, pouring corn, flipping a pot, moving an instructed object, knocking an instructed object over, and covering an instructed object with a towel.Training demonstrations range from 10 for Flip Pot Upright to 150 for Move <object> onto Plate.
  • Franka-DROID task: The Franka-DROID Wipe Table task requires grasping a brush and sweeping three brown objects into a dustpan, including trials with unseen distractors.The task has 70 training demonstrations with varied object positions; trials receive up to 2 points based on completion.

B.3.2 Detailed Franka-Tabletop and Franka-DROID Evaluation Results

The detailed adaptation results show that performance depends on task diversity: Diffusion Policy is strong on narrow single-instruction tasks, while OpenVLA and Octo perform better on diverse multi-instruction tasks. OpenVLA achieves the best overall average across the evaluated suites.

  • Single-instruction tasks: Diffusion Policy outperforms other methods on strong single-instruction tasks such as Put Carrot in Bowl and Pour Corn in Pot.It is trained from scratch, while OpenVLA and Octo are generalist policies fine-tuned on the same task data.
  • Multi-instruction tasks: OpenVLA and Octo achieve higher performance on the more diverse multi-instruction tasks Move <object> onto Plate, Knock <object> Over, and Cover <object> with Towel.These tasks require language-conditioned selection among multiple objects.
  • Overall results: OpenVLA achieves the highest average performance across the Franka-Tabletop and Franka-DROID tasks.The detailed results are reported in Table 7, with parameter-efficient results additionally detailed in Table 8.
  • Franka-DROID: OpenVLA obtains the best results in the Franka-DROID environment.The Wipe Table evaluation includes both in-distribution and distractor-containing OOD conditions.
  • Parameter-efficient evaluation: Parameter-efficient results use two Franka-Tabletop tasks, pairing one narrow single-instruction task with one diverse multi-instruction task and their OOD variants.The experiments use 50 and 150 demonstrations for the respective tasks.

C RT-2-X vs. OpenVLA in BridgeData V2 Evaluations

The BridgeData V2 comparison examines OpenVLA against RT-2-X while accounting for dataset preprocessing and visual-backbone differences. OpenVLA uses more OpenX data and a fused SigLIP-DinoV2 backbone, whereas RT-2-X requires a querying workaround for freezing behavior.

  • Comparison setup: OpenVLA uses a larger OpenX pretraining subset and a fused SigLIP-DinoV2 vision backbone, unlike RT-2-X’s single visual encoder.These differences accompany OpenVLA’s reported improvement in BridgeData V2 evaluations.
  • Data preprocessing: All-zero actions in the original BridgeData V2 demonstrations caused expressive policies to freeze, so OpenVLA training filtered the first transition from every demonstration.This preprocessing was sufficient to mitigate freezing in most OpenVLA cases.
  • RT-2-X evaluation: RT-2-X often freezes when deployed out of the box because it was trained without the same preprocessing, severely reducing rollout performance.Because RT-2-X is proprietary, the evaluation queried its second-most-likely action instead of retraining it.
  • Querying strategy: Dynamic querying performed worse than always selecting RT-2-X’s second-most-likely action, which the authors associate with latency-induced movement interruptions.The reported RT-2-X results therefore use the second-most-likely action consistently.

D Additional Experiments and Ablations

Additional experiments evaluate OpenVLA’s data mixture, vision encoder, parameter-efficient adaptation, quantized inference, and simulated-domain transfer. The ablations show strong dependence on OpenX diversity, a smaller benefit from DinoV2, and comparable quantized performance under blocking control.

  • OpenX training: Ablating OpenX training reduces absolute success rate by 30 percent across generalization categories, while language grounding performance remains unaffected.The result compares OpenVLA with OpenVLA-Bridge trained only on BridgeData V2.
  • Vision encoder: Removing DinoV2 causes a 5 percent performance reduction relative to the dual-encoder model, with benefits appearing only in some generalization cases.The ablation compares OpenVLA-Bridge with OpenVLA-Bridge-SigLIP on eight BridgeData V2 tasks.
  • Vision encoder training: Fine-tuning the vision encoder produced higher-performing VLAs than freezing it across the tested pretrained VLMs and architectures.The comparison is summarized in the fine-tuned versus frozen vision encoder experiment results.
  • Quantized inference: 8-bit quantization performs comparably to bfloat16 and 4-bit quantization when blocking control removes inference-speed differences.The result supports the hypothesis that prior 8-bit rollout degradation was driven by lower inference speed.
  • Simulation transfer: OpenVLA adapts effectively to simulated domains despite pretraining exclusively on real-world robot data.The paper presents this as evidence of versatility in LIBERO simulation experiments.
  • LIBERO evaluation: LIBERO comparisons evaluate Diffusion Policy, Octo, and LoRA-fine-tuned OpenVLA independently on each task suite using 500 trials per suite and three random seeds.The reported average success rates therefore aggregate 1500 trials per statistic.

E.2 LIBERO Simulation Experimental Results

OpenVLA adapts effectively to LIBERO simulation tasks, achieving the highest average success rate and rank among tested methods, though gains are narrower than in real-world fine-tuning.

  • OpenVLA achieves the highest average success rate and rank among the tested LIBERO methods.The benchmark reports success rate and ranking across four task suites, averaged over three random seeds with 500 trials each.
  • The performance margin over other methods is tighter in LIBERO than in the real-world fine-tuning experiments.
  • OpenVLA’s purely real-world pretraining and absence of simulation data may limit transfer to simulated tasks because of the real-to-sim domain gap.
  • The results suggest that adding simulation data to pretraining could increase performance gains for pretrained and fine-tuned methods.
Loading 2406.09246v3…