PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Actions for Enhanced Embodied Foundation Models

Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang, Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, Xianyuan Zhan

arXiv:2501.10105v2cs.ROcs.AIcs.CV

TL;DR

Embodied foundation models face action-space heterogeneity from different robot embodiments and control interfaces, limiting the use of diverse cross-domain data. UniAct learns generic atomic behaviors in a Universal Action Space and decodes them into embodiment-specific commands; its 0.5B model outperforms SOTA models 14 times larger across real-world and simulation evaluations.

  • Problem

    Different robot embodiments and control interfaces create heterogeneous action spaces, complicating cross-domain training for versatile embodied foundation models.

  • Method

    UniAct learns embodiment-independent atomic behaviors in a Universal Action Space, then maps universal actions to deployable commands through heterogeneous decoding heads trained across domains.

  • Results

    UniAct-0.5B outperforms SOTA embodied models 14 times larger, surpasses baselines across LIBERO suites, and outperforms OpenVLA-7B and LAPA-7B on several real-world generalization dimensions.

  • Takeaways & Limitations

    Universal actions support cross-domain data utilization, cross-embodiment generalization, and fast adaptation to robots with distinct control interfaces and physical properties.

  • Takeaways & Limitations

    UniAct is mostly evaluated on varied control interfaces for single robotic arms, leaving open whether shared physical movements extend to other embodiments.

Abstract

from arXiv · show

Training on diverse, internet-scale data is a key factor in the success of recent large foundation models. Yet, using the same recipe for building embodied agents has faced noticeable difficulties. Despite the availability of many crowd-sourced embodied datasets, their action spaces often exhibit significant heterogeneity due to distinct physical embodiment and control interfaces for different robots, causing substantial challenges in developing embodied foundation models using cross-domain data. In this paper, we introduce UniAct, a new embodied foundation modeling framework operating in a Universal Action Space. Our learned universal actions capture the generic atomic behaviors across diverse robots by exploiting their shared structural features, and enable enhanced cross-domain data utilization and cross-embodiment generalizations by eliminating the notorious heterogeneity. The universal actions can be efficiently translated back to heterogeneous actionable commands by simply adding embodiment-specific details, from which fast adaptation to new robots becomes simple and straightforward. Our 0.5B instantiation of UniAct outperforms 14X larger SOTA embodied foundation models in extensive evaluations on various real-world and simulation robots, showcasing exceptional cross-embodiment control and adaptation capability, highlighting the crucial benefit of adopting universal actions. Project page: https://github.com/2toinf/UniAct

1. Introduction

Embodied foundation models seek cross-task, cross-environment, and cross-embodiment generalization, but heterogeneous action spaces make cross-domain data difficult to share. UniAct addresses this by learning universal atomic actions that can be decoded into embodiment-specific commands.

  • Embodied foundation models aim to generalize across tasks, environments, and robot embodiments using diverse data.
  • Action heterogeneity arises from differences in robot morphology, control interfaces, viewpoints, environments, and human manipulation behaviors.
  • Actions collected across robots and institutions occupy largely disjoint manifolds, significantly complicating data sharing across sources.
  • Prior methods either force distinct action spaces into equivalent encodings or aggregate them into a shared space requiring extensive engineering.
  • UniAct learns embodiment-independent universal actions that encode generic atomic behaviors, such as moving forward toward a target.
  • A shared VLM constructs a vector-quantized codebook whose codes represent atomic behaviors transferable across robots.
  • Heterogeneous decoders translate universal actions into precise robot commands by adding embodiment-specific features from each robot’s observations.

2. Related Work

Related work explores multimodal foundation models, generalist robot policies, and latent action spaces for embodied control. UniAct differs by targeting universal actions that encode common control behaviors rather than only visual changes or pooled heterogeneous action labels.

  • Multimodal Foundation Models: Large language and vision-language models demonstrate broad task capabilities through unified tokenized representations and multimodal instruction following.
  • Generalist Robot Policies: RT-X, Octo, and OpenVLA use data from different 7-DoF robots to improve generalization beyond single-robot training.
  • Generalist Robot Policies: CrossFormer, RDT, π0, and related work extend cross-embodiment learning to robots with distinct mechanical structures, including manipulation and navigation platforms.
  • Embodied Models with Latent Action Spaces: Latent-action methods infer representations from dynamic changes between visual states, but those changes may include factors unrelated to actual control signals.
  • Embodied Models with Latent Action Spaces: UniAct’s universal action space encodes common atomic control behaviors across robotic platforms, aligning its latent representation with actionable control.

3. The UniAct Framework

UniAct constructs a discrete Universal Action Space that distills heterogeneous robot controls into shared atomic behaviors, then decodes those behaviors into embodiment-specific commands. Its training combines a shared universal-action extractor and codebook with lightweight heterogeneous decoder heads across multiple robotic datasets.

  • 3.1. Universal Action Space: UniAct represents universal actions as shared latent atomic behaviors distilled from movements generated by heterogeneous control signals across physical embodiments.The framework models the Universal Action Space as a discrete vector-quantized codebook whose entries represent generic atomic behaviors.
  • 3.1. Universal Action Space: Prior visual-change-based latent actions can capture external factors unrelated to control and depend on observation intervals, producing noisy and difficult-to-standardize behavior representations.These limitations motivate extracting actions from task progression rather than solely from changes between visual states.
  • 3.2. Universal Action Extraction: The universal action extractor is a fine-tuned VLM that predicts p(u|o, g), selecting the action most relevant to the observed task goal rather than merely explaining visual changes.Gumbel-Softmax provides gradient estimation during training despite the non-differentiable arg max, with temperature decay supporting exploration and convergence stability.
  • 3.3. Heterogeneous Decoding: Lightweight embodiment-specific decoder heads map universal actions and shared visual features to precise control signals while limiting decoder complexity to emphasize learning shared actions.The heads are implemented as simple MLP networks and can incorporate embodiment-specific details such as control type, proprioception, and distinct observations.
  • 3.4. Training Procedure: UniAct trains on K heterogeneous robotic datasets by predicting a universal action from observations and goals, then mapping it with observations to domain-specific actions.The objective jointly learns the universal codebook, extractor, and heterogeneous decoder heads using behavior-cloning losses suited to each dataset’s action labels.

4. Experiments

UniAct is evaluated for cross-embodiment generalization, simulation performance, adaptation to unseen robots and control interfaces, and the consistency and usability of its universal actions. Across these settings, the experiments compare broad task coverage, adaptation efficiency, and semantic behavior consistency.

  • Experimental Goals: The experiments test whether universal actions improve execution across domain gaps, transfer to unseen embodiments, and form a meaningful action space.The evaluation questions explicitly cover cross-embodiment execution, transfer to unseen embodiments, and universal-action meaning.
  • Experimental Setup: UniAct-0.5B is trained on 1 million demonstrations from 28 embodiments using a 0.5B model and a 256-entry universal action space.The implementation uses the LLaVA-OneVision-0.5B VLM and trains the universal action space on heterogeneous embodied data.
  • Main Results: UniAct-0.5B outperforms 14X larger OpenVLA-7B and LAPA-7B across 19 WidowX tasks spanning visual, motion, and physical generalization.Each model is evaluated over 190 rollouts, with 10 trials per task; UniAct remains comparable on semantic generalization and language grounding tasks.
  • Main Results: UniAct-0.5B surpasses baseline models across all five LIBERO suites covering 130 simulation tasks.LIBERO-90 contains 90 tasks, while each of the other four suites contains 10 tasks; models are fine-tuned with aligned tasks, data quantity, and image quality.
  • Fast Adaptation to New Embodiment: On unseen AIRBOT control interfaces, UniAct adapts by freezing the universal action extractor and codebook while training lightweight decoding heads.The evaluation covers relative and absolute end-effector and joint positions, with easy and cluttered versions of a cube-stacking task.
  • Fast Adaptation to New Embodiment: UniAct fine-tunes 4M of its 500M parameters, or 0.8%, while maintaining strong generalization across all tested AIRBOT control signals.The newly introduced heterogeneous heads are simple MLPs conditioned on universal actions and vision features.

5. Conclusion

UniAct addresses action heterogeneity through a Universal Action Space that captures shareable atomic behaviors across embodiments. Its 0.5B model outperforms much larger SOTA models, while heterogeneous decoding supports rapid adaptation to new robots.

  • UniAct operates in a Universal Action Space to address heterogeneous embodied action spaces.
  • Universal actions encode shareable atomic behaviors that enhance cross-domain data utilization and cross-embodiment generalization.
  • Heterogeneous decoding translates universal actions into embodiment-specific actions with minimal parameters, enabling fast adaptation to robots with distinct interfaces and physical properties.
  • The universal action extractor can also serve as a universal action tokenizer for future large-scale embodied foundation models.
  • Current evaluation is mostly limited to single-arm robotics platforms, with future work targeting larger models and broader embodiments.

A. Training Details

Training uses heterogeneous embodied data and implementation settings designed to balance embodiments, tasks, and scenes while learning a universal action codebook. The training procedure uses Gumbel-Softmax because straight-through estimation caused codebook collapse.

  • Training hyper-parameters: UniAct-0.5B uses a pretrained VLM and an ImageNet-pretrained ResNet18 in its heterogeneous decoding heads.
  • Image augmentation settings during training: Table 3 specifies image augmentation settings used during training.
  • Training hyper-parameters: Table 4 reports the training hyper-parameters.
  • Data construction: Training data spans 28 distinct embodiments, with sampling rates used to balance embodiments, tasks, and scenes.
  • Categorical Reparameterization: Straight-Through Estimator training caused severe collapses in the universal action codebook.

B. Evaluation Setups

The evaluation examines three robotic embodiments and reports detailed scores to make performance comparisons clearer and more intuitive.

  • Experiments cover WidowX, Franka, and AIRBOT embodiments.
  • Detailed scores are provided for each embodiment to support clearer, more intuitive performance comparisons.

B.1. WidowX Robot in Real World

WidowX experiments evaluate generalization across five dimensions, using progress-based scores for some complex tasks rather than only success rates. The setup also examines visual and semantic generalization under unseen conditions.

  • WidowX evaluation measures generalization across five distinct dimensions.
  • Progress-based scores are used for some complex tasks to reflect task completion more intuitively than success rates alone.
  • UniAct’s universal action space is described as mitigating negative transfer despite training on diverse heterogeneous data.
  • Visual Generalization: Visual generalization varies lighting, backgrounds, and object textures across three tasks with red and green target objects.
  • Semantic generalization uses target objects absent from training to test interpretation of new instructions and environments.

B.2. Franka Robot in Simulation

UniAct-0.5B is evaluated in LIBERO simulation against fine-tuned open-source baselines, using Franka demonstrations and rendered image observations. The section also describes adaptation settings involving supplementary wrist-view and proprioceptive inputs.

  • Experimental setup: 6,500 expert demonstrations spanning 130 tasks were collected with the Franka Robot in LIBERO simulation to train UniAct-0.5B.Evaluation follows the LIBERO Benchmark, with emulator-rendered images at 128×128 resolution.
  • Baseline comparison: Open-source baselines were substantially fine-tuned because they were not initially trained with simulation data.The additional fine-tuning was intended to facilitate fair comparison with UniAct-0.5B.
  • Decoder adaptation: UniAct’s decoder adaptation incorporated wrist-mounted image observations and proprioceptive data as supplementary inputs.The implementation followed the official ACT model settings and used 200K gradient steps with batch size 256 across 8 A100 GPUs.

B.3. Fast adaptation to AIRBOT

The AIRBOT experiment evaluates fast adaptation to an unseen embodiment by fine-tuning UniAct-0.5B and baseline models on newly collected demonstrations.

  • Adaptation setup: AIRBOT provides an unseen embodiment during pretraining for assessing UniAct-0.5B’s fast adaptation capability.The comparison includes baseline models fine-tuned on newly collected demonstrations.
  • UniAct fine-tuning: UniAct-0.5B was adapted to AIRBOT by training a new MLP heterogeneous head while keeping the other modules frozen.Fine-tuning used 4 A100 GPUs with DeepSpeed and lasted 1 hour.
  • Baseline comparison: OpenVLA and Octo were also fine-tuned as baseline models for the AIRBOT comparison.The supplied materials identify the baseline comparison and its detailed scores through the AIRBOT evaluation materials.

B.4. Fast adaptation to Bi-manual AIRBOT

The bimanual AIRBOT study tests UniAct’s fast adaptation on four challenging manipulation tasks using an ACT decoder, while positioning universal actions as an intermediate skill representation.

  • Decoder design: An ACT decoder head was used because a simple MLP decoder could not fulfill the perception requirements of bimanual manipulation.The decoder choice addresses the increased complexity of the bimanual setting.
  • Evaluation tasks: Four bimanual AIRBOT tasks were evaluated: sweep plate, fold towel, put cup on plate, and transport pen.Each task contributed 250 demonstration trajectories for fine-tuning UniAct-0.5B.
  • Data collection: 250 demonstration trajectories per task were collected before fine-tuning UniAct-0.5B.The task set targets precise perception and complex bimanual manipulation.
  • Universal-action hierarchy: UniAct first infers a universal action and then translates it into actionable actions, resembling a hierarchical structure.Compared with language- or visual-subgoal hierarchies, the universal-action space is described as a more fine-grained skill library that can be adapted to physically grounded actions.

D. Limitations and Future Works

The paper identifies scope and scaling questions for UniAct, including broader embodiments, embodiment-specific decoder complexity, data scaling, and uses of universal actions beyond the evaluated policies.

  • More embodiments: UniAct is mostly evaluated on single robotic arms with varied control interfaces, leaving broader embodiment coverage as an open limitation.The paper questions whether shared physical movements extend to substantially different embodiments.
  • More embodiments: Future work targets dual arms, dexterous hands, quadrupeds, and autonomous driving cars within a more broadly universal action space.These systems differ in degrees of freedom and mechanical structures but may share fundamental movements with simpler robotic arms.
  • More flexible network design: As datasets expand across embodiments, decoder heads should be specialized to the control complexity of each embodiment.The paper also identifies proprioceptive data and different views as embodiment-specific information that could improve decoder performance.
  • Scaling law: The relationship between data quantity, embodiment diversity, task diversity, and universal-action performance remains unresolved.The paper explicitly asks whether more data, more embodiments, or more demonstrations for one embodiment consistently improve results.
  • More utilization of universal action: Universal actions may also serve as a tokenizer for world models by decomposing complex actions into standardized components for planning and simulation.The proposed use extends beyond the cross-embodiment robot policies evaluated in the work.
Loading 2501.10105v2…