PaperlayerView exact revision on arXiv

Source-linked AI summary

UniVLA: Learning to Act Anywhere with Task-centric Latent Actions

Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, Hongyang Li

arXiv:2505.06111v3cs.ROcs.AIcs.LG

TL;DR

Existing VLA policies depend on action-annotated data and struggle to transfer knowledge across embodiments and environments. UniVLA learns task-centric latent actions from heterogeneous videos, predicts them with a generalist vision-language policy, and decodes them for different robots. It reports state-of-the-art manipulation, navigation, and real-robot performance while using less pretraining compute and benefiting from heterogeneous data, including human videos.

  • Problem

    Existing approaches rely on ground-truth action labels and face heterogeneous action and observation spaces, limiting unified learning across embodiments and environments.

  • Method

    UniVLA learns task-centric latent actions from cross-embodiment videos using language-conditioned DINOv2 representations, then predicts and decodes them for robot-specific control.

  • Results

    UniVLA achieves state-of-the-art performance across manipulation and navigation benchmarks and real-robot tests, including an 18.5% success-rate increase over OpenVLA on LIBERO.

  • Takeaways & Limitations

    UniVLA supports scalable policy learning from heterogeneous robot and human videos, with transferable latent representations across embodiments and views.

  • Takeaways & Limitations

    The paper identifies future work on using human-video latent embeddings as in-context samples for zero-shot skill acquisition without fine-tuning.

Abstract

from arXiv · show

A generalist robot should perform effectively across various environments. However, most existing approaches heavily rely on scaling action-annotated data to enhance their capabilities. Consequently, they are often limited to single physical specification and struggle to learn transferable knowledge across different embodiments and environments. To confront these limitations, we propose UniVLA, a new framework for learning cross-embodiment vision-language-action (VLA) policies. Our key innovation is to derive task-centric action representations from videos with a latent action model. This enables us to exploit extensive data across a wide spectrum of embodiments and perspectives. To mitigate the effect of task-irrelevant dynamics, we incorporate language instructions and establish a latent action model within the DINO feature space. Learned from internet-scale videos, the generalist policy can be deployed to various robots through efficient latent action decoding. We obtain state-of-the-art results across multiple manipulation and navigation benchmarks, as well as real-robot deployments. UniVLA achieves superior performance over OpenVLA with less than 1/20 of pretraining compute and 1/10 of downstream data. Continuous performance improvements are observed as heterogeneous data, even including human videos, are incorporated into the training pipeline. The results underscore UniVLA's potential to facilitate scalable and efficient robot policy learning.

I. INTRODUCTION

UniVLA targets cross-embodiment policy learning by replacing action-label dependence with task-centric latent actions learned from heterogeneous videos. Its framework combines unsupervised latent-action extraction, latent-action policy prediction, and efficient decoding, achieving strong benchmark and real-robot results with substantially lower pretraining cost.

  • Motivation: Existing VLA policies rely on ground-truth action labels and face incompatible action and observation spaces across embodiments and tasks.These constraints motivate a unified action representation for transferring knowledge across environments and internet-scale videos.
  • Framework: UniVLA’s recipe comprises task-centric latent-action learning, autoregressive latent-action prediction, and latent decoding into physical behaviors.Specialized decoders adapt the pretrained generalist policy to unseen tasks and robotic systems.
  • Representation learning: Language conditioning and DINOv2 features are used to reduce task-irrelevant dynamics in latent representations learned from videos.The approach addresses distractions such as non-ego-agent motion and unpredictable camera shifts.
  • Efficiency and scalability: UniVLA requires 1/20 of OpenVLA’s pretraining cost while improving with heterogeneous robot and human-video data.A lightweight 10.8M-parameter decoder reduces downstream fine-tuning requirements, according to the introduction.
  • Contribution: UniVLA extracts task-relevant latent actions from cross-embodiment videos, decoupling task-centric dynamics from irrelevant visual changes.The framework uses unsupervised extraction and aims to support knowledge transfer across diverse embodiments and environments.
  • Results: 18.5% higher success rate over OpenVLA on LIBERO, 29.6% in navigation tasks, and 36.7% in real-world deployments are reported.The paper states state-of-the-art performance across multiple benchmarks and real-robot tests.

A. Vision-language-action Models

The paper positions UniVLA against action-space and data-labeling limitations in prior VLA and latent-action methods. Its task-centric latent-action learner uses paired video frames, vector quantization, language conditioning, and DINOv2 features to separate task-relevant from irrelevant dynamics.

  • Challenges: General-purpose robot policies must handle differing viewpoints, proprioception, joints, action spaces, and control frequencies across robotic systems.Existing approaches also require extensive diverse datasets and explicit annotations for broad state-transition coverage.
  • Prior limitations: Prior latent-action methods avoid explicit action labels but may encode task-irrelevant visual dynamics through reconstruction-based or pixel-space objectives.Examples include camera shifts, external-agent motion, textures, and lighting variability.
  • Latent-action construction: UniVLA samples consecutive video frames approximately one second apart and derives latent actions with an inverse-dynamics encoder and forward-dynamics model.The interval is calibrated to recording frequency across datasets.
  • Latent-action construction: Vector quantization compresses action tokens into a discrete codebook whose decoder predicts future frames from the quantized representation.The decoder avoids historical frames to constrain the information carried by the latent actions.
  • Task-centric decoupling: Stage 1 conditions encoder and decoder on T5 instruction embeddings, while Stage 2 introduces a new codebook for task-centric dynamics.The two-stage design separates task-irrelevant representations from task-related object manipulation and goal-directed motion.

B. Pretraining of Generalist Policy

UniVLA pretrains a Prismatic-7B-based policy to predict discrete latent actions from visual observations and task instructions. Specialized decoder heads then translate those embodiment-agnostic plans into executable controls, enabling cross-domain transfer without manually aligned action spaces.

  • Policy pretraining: The generalist policy uses a fused SigLip-DINOv2 visual encoder, projection layer, and Prismatic-7B vision-language model to predict latent action tokens.The model is trained autoregressively from visual observations and task instructions.
  • Efficiency: 960 A100-hours of pretraining achieves competitive results, compared with 21,500 A100-hours for OpenVLA.The compressed action space reduces the action vocabulary from 2567 in OpenVLA to 164 when |C| = 16.
  • Latent-action decoding: Specialized action decoder heads use visual context to translate latent actions into executable control signals for heterogeneous action spaces.This decoding preserves the policy’s embodiment-agnostic latent planning while adapting outputs to specific robotic systems.

C. Post-training for Deployment

UniVLA adapts its embodiment-agnostic latent plans to target robots through lightweight action decoders and parameter-efficient fine-tuning. Historical latent actions provide sequential context without directly processing redundant visual histories.

  • Latent action decoding: Visual and latent action embeddings are aggregated through multi-head attention pooling before projection into the target robot’s action space.Randomly initialized queries extract visual and action information from the VLM’s final-layer embeddings.
  • Latent action decoding: Latent actions represent approximately one-second behaviors and can be decoded into customizable action chunks for smoother, more precise control.Chunk size is adjusted for specific embodiments.
  • Efficient adaptation: Parameter-efficient LoRA fine-tuning adds a 12.6M-parameter action head, while end-to-end training optimizes latent-action prediction and low-level action losses.The total number of trainable parameters is approximately 123M.
  • History outputs: Historical latent actions are appended to task instructions and incorporated during inference as four tokens per timestep, providing sequential context with fewer visual tokens.The method uses latent-action annotations from historical frames instead of directly supplying multiple historical observations.

IV. EVALUATIONS

UniVLA is evaluated across manipulation, navigation, and real-world settings to test transfer, adaptability, generalizability, and scalability. The benchmark suite combines diverse datasets, baselines, and task configurations, including LIBERO’s four evaluation suites.

  • Evaluation scope: The evaluation covers LIBERO, CALVIN, SimplerEnv, R2R, real-world scenarios, latent-action analysis, and ablations of critical design choices.The experiments target performance and adaptability, generalizability to unseen scenarios, and scalable use of diverse data sources.
  • Data and protocol: Pretraining combines manipulation, navigation, and human-video data from OpenX, GNM, and Ego4D, while LIBERO samples remain excluded from pretraining.Target-task policies are evaluated through supervised fine-tuning.
  • LIBERO setup: LIBERO experiments use four task suites spanning spatial reasoning, object variation, goal-oriented behavior, and long-horizon multi-step manipulation.LIBERO-Long includes heterogeneous objects, layouts, and task sequences.
  • Comparisons: Real-world experiments compare UniVLA with representative policies including OpenVLA, LAPA, Octo, MDT, and MaIL across multiple deployment tasks.The baseline set includes methods using diverse robotic datasets, unified action representations, diffusion models, and latent actions.
  • Manipulation results: On LIBERO, UniVLA reaches 95.2% average performance with the full dataset, exceeding OpenVLA by 18.7% and LAPA by 29.5%.With Bridge-V2-only pretraining, it reaches 92.5% average performance, above MaIL’s 83.5% and MDT’s 76.1%.

2) Navigation Benchmark on Room2Room

The R2R evaluation measures language-guided navigation in continuous photorealistic environments using oracle success rate. UniVLA predicts actions from the current RGB observation and historical latent action, outperforming several RGB-only baselines while matching NaVid.

  • Benchmark setup: R2R in VLN-CE evaluates language-guided navigation in reconstructed photorealistic indoor scenes using 10,819 training samples and 1,839 unseen-validation samples.Oracle success rate is the evaluation metric.
  • Baselines: The comparison includes RGB-only Seq2Seq, CMA, LLaVA-Nav, OpenVLA, and NaVid navigation policies.NaVid encodes all historical RGB observations, whereas OpenVLA is fine-tuned with navigation-action tokens.
  • Results: UniVLA increases oracle success rate from 8.10% to 47.1% over Seq2Seq and CMA.This comparison reports the combined increase stated for those baselines.
  • Results: UniVLA surpasses LLaVA-Nav by 33.1% and OpenVLA by 29.6% in oracle success rate.Its oracle success rate is comparable to NaVid’s despite using only the current observation and historical latent action.

3) Real-world Robot Deployment

Real-world tests examine UniVLA across spatial, tool-use, deformable-object, and semantic capabilities on a 7-DoF robot. UniVLA trades some task-specific trajectory precision for stronger semantic generalization, robustness, and overall deployment performance.

  • Setup: The real-world evaluation uses a Piper arm with a 7-DoF action space and RGB input from a third-view RGB-D camera.The depth channel is not used.
  • Task axes: Four tasks test spatial awareness, tool use and nonprehensile manipulation, deformable-object manipulation, and semantic understanding.They include storing a screwdriver, cleaning a cutting board, folding a towel twice, and stacking Tower of Hanoi.
  • Task-specific trade-off: Diffusion Policy achieves 53.3% success on towel folding versus UniVLA’s 46.7%, while UniVLA obtains a higher step-wise score of 2.47 versus 2.33.The comparison reflects trajectory fidelity on a specialized single-task workflow versus intermediate-stage completion.
  • Semantic generalization: UniVLA reaches 86.7% success on Tower of Hanoi, compared with Diffusion Policy’s 6.7%, and achieves 93.3% in precise object-manipulation scenarios.These results support stronger semantic understanding and generalization than the single-task policy.
  • Deployment efficiency: UniVLA runs closed-loop inference at 10Hz and outperforms LAPA by 36.7% in success rate and 0.68 in average score.OpenVLA’s inference latency produces stuttering and 38.3% average success in the reported real-world comparison.
  • Generalizability: UniVLA achieves 66.7% success under lighting variation, while novel-object replacement reduces its success rate by only 6.6%.Visual distractors cause a relatively notable performance drop for semantic-information-based policies.

B. Discussion on Latent Action

UniVLA’s task-centric latent actions transfer across embodiments and domains while improving policy performance and data efficiency. Decoder and history-action designs further strengthen deployment, especially on long-horizon tasks.

  • Cross-domain transfer: Latent actions transfer across embodiments and data sources, representing semantically consistent behaviors and generalizing to unseen LIBERO tasks.The latent action model was trained without LIBERO data yet labeled accurate actions in that domain.
  • Task-centric representations: 13% and 9.8% improvements on LIBERO-Goal and LIBERO-Long show that task-centric latent actions outperform task-irrelevant alternatives.Task-irrelevant representations produce poor alignment, lower token-prediction accuracy, and near-zero LIBERO-Long success.
  • Data scalability: 2.0% average success-rate improvement follows the addition of cross-embodiment OpenX and Ego4D data, with further gains from real-world and human-video data.Real-world scores increase by 0.3 with OpenX and by an additional 0.28 with human data; R2R shows the same trend.
  • Data efficiency: 86.3% versus 79.2% on LIBERO-Goal shows UniVLA surpasses full-data OpenVLA using only 10% of demonstration data.UniVLA also reaches state of the art on LIBERO-Goal with 10% and LIBERO-Long with 50% of training episodes.
  • Decoder design: 42.1% improvement on LIBERO-Long results from the proposed decoder, while visual-query decoding adds 2.2% average success-rate gain.The decoder consistently outperforms the auto-regressive alternative across all test suites.
  • History latent actions: 16.5% on R2R and 3.9% on LIBERO-Long are gained by adding only four historical latent-action tokens, benefiting long-horizon decisions.The method augments instruction inputs with latent actions from previous steps.

V. CONCLUSION

UniVLA plans in a unified task-centric latent action space and adapts to novel robotic setups. The paper reports state-of-the-art benchmark performance, scalable gains from heterogeneous data, and several directions for extending the framework.

  • UniVLA plans within a unified, task-centric latent action space for efficient adaptation to novel robotic setups.
  • Limitations: Fixed latent-action granularity and codebook size may not suit every task or embodiment, while evaluation primarily covers single-arm manipulation.Extending the framework to dual-arm humanoid systems or dexterous hands may require finer-grained action modeling.
  • Limitations: The dataset mainly uses fine-grained, short-horizon instructions, although the method supports instructions of varying granularity without special handling.
  • Future work: The latent-action decoder can generate multiple visual plans and may support reference alignment with reinforcement learning and test-time planning trees.
  • Future work: Encoding human demonstrations as compact latent-action sequences is proposed as a future route to zero-shot skill acquisition without fine-tuning.

APPENDIX

The appendix documents UniVLA’s heterogeneous training data, optimization and evaluation settings, and supplementary results. It reports strong CALVIN and SimplerEnv performance alongside qualitative evidence that latent actions focus on task-critical regions.

  • Training data: UniVLA training combines OXE, GNM, and Ego4D data, including robot and open-world human videos, while pretraining uses frames and instructions rather than robot actions or proprioception.
  • Optimization: Approximately 30 hours on 32 NVIDIA A100 GPUs are reported for 20,000 optimization steps in the downstream setup.
  • CALVIN: 56.5% success on CALVIN completes all five tasks, surpassing CLOVER’s 45.4% by 11.1% and OpenVLA by 13%.Average consecutively completed tasks rise from OpenVLA’s 3.27 to 3.80.
  • SimplerEnv: 35.4% success is achieved with decoder-only adaptation on SimplerEnv, retaining pretrained knowledge while minimizing adaptation costs.The appendix reports superior grasp and task success rates under this constrained setting.
  • Latent-action visualization: Task-centric latent actions attend to the robotic end-effector and interacted objects while ignoring irrelevant background regions.

2) SimplerEnv

SimplerEnv evaluates UniVLA on four randomized WidowX-plus-Bridge tasks using repeated trials. Decoder-only adaptation achieves strong task performance while preserving pretrained knowledge and reducing adaptation costs.

  • Experiment setup: SimplerEnv tests four WidowX-plus-Bridge tasks involving placing, stacking, and object manipulation under randomized object initialization.
  • Experiment setup: Each SimplerEnv task is evaluated over 24 independent trials to obtain robust performance metrics.
  • Results: 35.4% task success is achieved with decoder-only adaptation, demonstrating retained pretrained knowledge under reduced adaptation.Full fine-tuning produces a lower grasp rate than decoder-only training, likely from overfitting to seen scenes.
  • Latent-action analysis: Latent-action visualizations concentrate attention on task-critical end-effectors and interacted objects rather than irrelevant background.

C. Real-world Robots

Real-world evaluations compare policies using both task completion and task-specific quality, revealing differences that success rate alone can miss. UniVLA’s generalization supports stronger performance across varied positional configurations, while Diffusion Policy remains competitive on smooth, structured tasks.

  • Task setup: Real-world task setups include position-generalization tests for screwdriver storage, variable sweeping requirements, and folding a 20cm × 20cm towel.These setups vary object positions, sweep complexity, and the physical task configuration.
  • Evaluation: Success rate and task-specific scoring jointly evaluate completion and performance quality, so identical success rates can conceal divergent outcomes.The detailed scoring criteria are provided in Tab. A-IV, with full results in Tab. A-V.
  • Results: UniVLA’s generalizability produces higher scores than Diffusion Policy across diverse positional configurations, including partial task completion.Diffusion Policy’s higher success rate on structured folding does not translate into a higher score when rotation angles vary.
  • Architecture: The real-world action decoder uses two multi-head attention blocks for latent-action and visual embeddings, plus an MLP for proprioceptive states.The resulting embeddings are concatenated and projected into the desired action dimensions; a proprioceptive projection module is additionally used in real-world experiments.
Loading 2505.06111v3…