PaperlayerView exact revision on arXiv

Source-linked AI summary

Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization

Hao Luo, Ye Wang, Wanpeng Zhang, Sipeng Zheng, Ziheng Xi, Chaoyi Xu, Haiweng Xu, Haoqi Yuan, Chi Zhang, Yiqing Wang, Yicheng Feng, Zongqing Lu

arXiv:2601.12993v1cs.RO

TL;DR

Being-H0.5 targets the physical gap and data scarcity that limit VLA generalization across heterogeneous robots. It combines human-centric pre-training, a Unified Action Space, unified sequence modeling, and specialized flow-based architecture with robustness mechanisms. The model reports state-of-the-art benchmark results and cross-embodiment transfer across five robotic platforms, while remaining bounded by task-alignment and backbone-scope limitations.

  • Problem

    VLAs are limited by heterogeneous embodiments, scarce robot-specific data, conflicting action spaces, and a physical gap between human and robotic control.

  • Method

    Being-H0.5 combines UniHand-2.0 human-centric pre-training, a unified state-action space and sequence model, and Mixture-of-Flow architecture with robustness-oriented control mechanisms.

  • Results

    98.9% on LIBERO and 53.9% on RoboCasa are reported as state-of-the-art results, alongside strong cross-embodiment performance across five robotic platforms and emergent zero-shot transfer.

  • Takeaways & Limitations

    Human interaction data and shared action representations provide the paper’s supported route toward cross-embodiment VLA generalization and adaptation to unseen robot morphologies.

  • Takeaways & Limitations

    Direct application of human-centric data remains constrained by embodiment gaps and a lack of task-aligned labels, while the chosen VLM backbone may affect robotic performance.

Abstract

from arXiv · show

We introduce Being-H0.5, a foundational Vision-Language-Action (VLA) model designed for robust cross-embodiment generalization across diverse robotic platforms. While existing VLAs often struggle with morphological heterogeneity and data scarcity, we propose a human-centric learning paradigm that treats human interaction traces as a universal "mother tongue" for physical interaction. To support this, we present UniHand-2.0, the largest embodied pre-training recipe to date, comprising over 35,000 hours of multimodal data across 30 distinct robotic embodiments. Our approach introduces a Unified Action Space that maps heterogeneous robot controls into semantically aligned slots, enabling low-resource robots to bootstrap skills from human data and high-resource platforms. Built upon this human-centric foundation, we design a unified sequential modeling and multi-task pre-training paradigm to bridge human demonstrations and robotic execution. Architecturally, Being-H0.5 utilizes a Mixture-of-Transformers design featuring a novel Mixture-of-Flow (MoF) framework to decouple shared motor primitives from specialized embodiment-specific experts. Finally, to make cross-embodiment policies stable in the real world, we introduce Manifold-Preserving Gating for robustness under sensory shift and Universal Async Chunking to universalize chunked control across embodiments with different latency and control profiles. We empirically demonstrate that Being-H0.5 achieves state-of-the-art results on simulated benchmarks, such as LIBERO (98.9%) and RoboCasa (53.9%), while also exhibiting strong cross-embodiment capabilities on five robotic platforms.

1 Introduction

Being-H0.5 addresses cross-embodiment generalization by treating human interaction as transferable physical supervision and aligning heterogeneous robot actions in a unified space. Its large-scale training recipe, unified modeling, and architectural mechanisms support strong simulated and real-world results across diverse platforms.

  • Motivation: Existing VLAs face a physical gap caused by heterogeneous robot morphologies, limited robot-specific data, and conflicting action spaces.These factors can produce action-space interference and negative transfer when datasets are naively combined.
  • Motivation: Human-centric learning uses human movements as dense semantic priors for causal interaction and contact physics that transfer across robotic kinematic dialects.The approach treats human interaction traces as a physical-world “mother tongue.”
  • Unified training: A Unified Action Space represents diverse robot actions in a shared physical vocabulary, allowing low-resource robots to bootstrap skills from human demonstrations and data-rich platforms.The design targets semantically aligned control across disparate morphologies.
  • Model design: Being-H0.5 serializes heterogeneous supervision into unified multimodal sequences and combines Mixture-of-Flow, Manifold-Preserving Gating, and Universal Async Chunking.These components target scalable action generation, sensory robustness, and real-time control across embodiments.
  • Results: 98.9% on LIBERO and 53.9% on RoboCasa establish state-of-the-art simulated results, while evaluations across five platforms show strong cross-embodiment generalization and emergent zero-shot transfer.A single jointly trained checkpoint achieves non-zero success on unseen task–embodiment pairs without target-robot data.
  • Training recipe: UniHand-2.0 contains 400M samples across 35,000 hours and 30 embodiments, combining human, robot, and vision-language data for large-scale pre-training.The recipe includes 16,000 human hours and 14,000 robot hours, alongside a portable human data collection system.

2 Related Work

Prior VLA research has expanded toward generalist models and heterogeneous datasets, but cross-embodiment integration remains difficult. Human-centric approaches provide interaction priors, yet embodiment gaps and weak task alignment limit direct use for precise control.

  • Vision-Language-Action Models: VLAs extend pre-trained vision-language models with robotic control data, but their action-head designs differ substantially.Autoregressive action tokenization supports transfer from VLM training but can increase inference latency and limit precision in high-DoF tasks.
  • Heterogeneous Pre-Training Datasets: Large heterogeneous datasets aggregate robot demonstrations, but lab-specific data, sim-to-real gaps, and inconsistent formats hinder generalizable unified modeling.Differences in annotation granularity and embodiment structure remain integration challenges.
  • Human-Centric Learning: Human-centric learning lowers data-collection barriers and supplies large interaction priors, while direct VLA use remains non-trivial because of embodiment gaps and missing task-aligned labels.Portable interfaces transfer in-the-wild human skills to deployable robot policies, but the supervision is not automatically aligned with VLA control.
  • Human-Centric Learning: Early human-video methods learned implicit representations without action-level supervision, limiting their utility for precise downstream control.Subsequent work therefore explored latent action abstractions and recovery of actual human actions from demonstrations.

3 UniHand-2.0: A Recipe for Large-Scale Human-Centric Learning

UniHand-2.0 addresses data scarcity, limited embodiment diversity, rare dexterous-hand data, and visual-language imbalance through a large human-centric multimodal pre-training recipe. It combines human demonstrations, robot manipulation, and vision-language data to support unified robotic learning.

  • Dataset composition: UniHand-2.0 combines human demonstrations, robot manipulation, and vision-language understanding into a human-centric pre-training recipe.The three sources provide motion and semantic information, heterogeneous robot controls, and visual-language reasoning supervision.
  • Embodiment diversity: UniHand-2.0 broadens embodiment coverage while addressing structural discrepancies that make heterogeneous robot datasets difficult to unify.Existing datasets often contain few robot types, while UniHand-2.0 includes 30 distinct robotic platforms and diverse control interfaces.
  • Human-centric learning: Human motion data serves as a scalable proxy for scarce dexterous-hand demonstrations, which comprise less than 5% of existing corpora.The recipe treats the human hand as a universal template for robot end-effectors and extracts 134 million human samples from egocentric video.
  • Dataset scale: 35,000 hours of multimodal data span 16,000 hours of human video, 14,000 hours of robot data across 30 embodiments, and 5,000 equivalent hours of VLM data.The corpus contains over 400 million samples and more than 120B training tokens.
  • Balanced supervision: 26% simulation data maintains a higher simulation share than OXE at 3.1% and AgiBot World at 3.0%, while multimodal data preserves language-based reasoning.The recipe balances visual and textual supervision to support both atomic action execution and long-horizon planning and spatial reasoning.

4 UniCraftor: A System for Portable, Extensible, and Affordable Data Collection

UniCraftor is a modular human-centric data collection system designed to improve geometric and temporal supervision in scalable multimodal recordings. It combines depth, calibrated camera poses, synchronized interaction events, and extensible sensing.

  • Motivation: UniCraftor addresses missing depth, unstable camera alignment, and coarse interaction timing in open-source human-video corpora.Existing sources often provide semantic descriptions or raw RGB but lack geometric depth and precisely aligned contact or release events.
  • Sensing: Native RGB-D capture supplies physical depth that remains usable under occlusion, rapid ego-motion, and adverse lighting.The system uses a head-mounted Intel RealSense D435 with active infrared stereo.
  • Calibration: AprilTag-based PnP calibration establishes high-precision camera extrinsics and a consistent world coordinate frame.This decouples hand-camera motion and supports integration of exocentric views.
  • Temporal supervision: A hardware-synchronized foot pedal records temporally precise interaction events across sensor streams.The timestamps identify affordance-relevant keyframes such as object contact or release.
  • Dataset construction: 43 tabletop tasks and 200+ hours of multimodal recordings demonstrate the system’s portable and extensible collection setup.Central timestamping synchronizes sensors, while the modular architecture permits sensor interchange and planned tactile extensions.

5 Being-H0.5: A Foundational VLA Unifying Cross-Embodiment Control

Being-H0.5 combines specialized semantic-understanding and action-generation experts within a shared transformer backbone, using a unified state–action representation for heterogeneous embodiments. Its training and action modeling pair multimodal sequence learning with flow-based motion generation and routed embodiment-specific capacity.

  • 5.1 Model Architecture: Being-H0.5 separates multimodal understanding from action generation while coupling both experts through shared self-attention.The understanding expert supports perceptual interpretation and planning, while the action expert translates plans into kinematic execution.
  • 5.1 Model Architecture: The architecture addresses heterogeneous-data scaling because conventional action experts have limited capacity and platform-specific heads fragment shared physical priors.The proposed sparse expert design aims to preserve common structure while specializing for embodiment and task dynamics.
  • 5.1.1 Unified State-Action Space: A physically interpretable unified state–action space maps heterogeneous embodiment controls into semantically aligned slots, including human hand motion.The representation organizes quantities such as end-effector poses, joint positions, gripper width, finger articulations, and mobile-base velocity.
  • 5.1.1 Unified State-Action Space: Strict subspace standardization preserves physical meaning by using relative Cartesian deltas, axis-angle rotations, absolute joint radians, and raw magnitudes.The design avoids statistical normalization such as scaling values to [-1, 1].
  • 5.1.2 Mixture of Flow: Mixture of Flow decouples shared embodiment primitives from specialized experts, allowing task-specific pathways without cross-task interference.During training, gradients update only the relevant expert pathway, while total model capacity can exceed active parameter usage.
  • 5.2 Unified Sequence Modeling: The unified training framework serializes heterogeneous supervision into multimodal sequences and models continuous actions with a time-conditioned flow-matching velocity field.The field transports Gaussian samples toward target actions under visual and textual conditioning.

A ]. Here, SFM

The sequence model uses separate continuous-flow and discrete-motion target segments under a shared conditioning context. The two target channels are designed to remain mutually invisible to prevent information leakage.

  • The continuous-flow and discrete-motion channels share the same conditioning context while representing separate motion targets.
  • The target segments are kept mutually invisible so the model cannot trivially copy information between continuous and discrete channels.
  • The attention design preserves shared contextual conditioning while separating the two motion-prediction pathways.

A and SMASK

The attention mask lets both motion-prediction targets attend to shared context while preventing direct visibility between the continuous-flow and discrete-motion channels. This gating structure is used to block cross-channel information leakage.

  • Both motion-prediction targets can attend to the shared context sequence.
  • The continuous-flow and discrete-motion target segments remain mutually invisible under the modified attention mask.
  • A gating matrix implements the separation between shared contextual information and target-specific information flow.

A ; SMASK

This section addresses cross-embodiment adaptation by isolating embodiment-specific action components while stabilizing flow-based policies under unreliable context. It also aligns asynchronous chunk execution with platform-specific delays.

  • Embodiment-Specific Adaptation: Embodiment-specific gradients can conflict under shared parameters, producing unstable or averaged behaviors on complex hardware.
  • Manifold-Preserving Gating: MPG compares observation and reference-action embeddings in SWD space to gate feature-conditioned residuals while retaining an ungated learned prior offset.At inference, the previous action iterate supplies the reference anchor.
  • Embodiment-Specific Adaptation: ESA updates only adapters associated with an embodiment’s active unified-action slots, while overlapping slots transfer knowledge across platforms.Non-overlapping slots remain isolated, reducing parameter conflict in a shared checkpoint.
  • Manifold-Preserving Gating: MPG suppresses feature-dependent corrections when context is unreliable, reducing sensitivity to out-of-distribution observations and stabilizing trajectories.The ungated offset remains active even when the gate is small, providing a robust fallback.
  • Universal Async Chunking: UAC adapts chunked control to embodiment-specific delays by separating committed prefixes from predicted postfixes for asynchronous execution.A dual-thread buffer supports heterogeneous latency and control profiles.

6 Infrastructure: Real-Time Cross-Embodiment Deployment

The deployment infrastructure combines efficient rectified-flow inference, MPG refinement, and delay-aware asynchronous chunking to maintain temporal consistency across heterogeneous robots. It protects already committed actions while updating only future actions in a shared execution buffer.

  • Rectified-Flow Inference: Rectified-flow action generation iteratively integrates a learned velocity field from Gaussian noise, with vision-language prefix features cached during denoising.The dynamic suffix contains state and current action-token embeddings, while only action-token embeddings evolve.
  • Manifold-Preserving Refinement: MPG refinement uses a baseline prediction as a reference anchor, then performs a small number of gated refinement rounds under shifted or noisy context.The feedback loop updates the anchor from the previous prediction at each refinement round.
  • Manifold-Preserving Refinement: MPG is especially useful with fewer than 10 denoising steps because its prior fallback stabilizes velocity-field prediction under low confidence.The method reduces sensitivity to noisy context while preserving low-latency inference.
  • Universal Async Chunking: UAC commits a delay-dependent prefix, locks it throughout denoising, and stitches only the predicted postfix into the execution buffer.This prevents the model from editing actions already executing and guarantees continuity when the latency commitment is honored.
  • Universal Async Chunking: The latency commitment is configured per platform using control period, expected inference latency, and a safety margin.Overestimation is safe, whereas underestimation can cause discontinuity because execution proceeds beyond the conditioned prefix.
  • Real-Time Deployment: A dual-thread ring buffer separates fixed-frequency control from asynchronous inference, writing only postfix actions and using safe fallback behavior during underflow.The architecture absorbs latency jitter and supports differing control frequencies and compute budgets by adjusting delay and buffer parameters.
  • Real-Time Deployment: A single Being-H0.5 checkpoint supports real-time control from 10 Hz tabletop manipulators to 50 Hz humanoids across cloud and edge deployment.

7 Experiment

Experiments show that Being-H0.5 combines strong benchmark performance with cross-embodiment generalization, while human-centric pretraining and asynchronous stability mechanisms materially support adaptation and deployment.

  • Pretraining: UniHand-2.0 pretraining is crucial for a strong generalist, whose ablation degrades substantially and fails to match the specialist counterpart.The specialist ablation remains non-trivial after real-robot fine-tuning, indicating a strong architectural and post-training inductive bias.
  • Cross-embodiment transfer: A single generalist checkpoint solves previously unseen Adam-U task–embodiment pairs without Adam-U demonstrations, albeit at low success rates.The paper attributes this to the Unified Action Space and cross-embodiment joint post-training.
  • Benchmark performance: 98.9% average success on LIBERO is achieved by the specialist checkpoint, while the generalist reaches 97.6% across joint LIBERO+RoboCasa training.Both use only 224×224 RGB inputs and a 2B backbone.
  • Benchmark performance: 53.9% success on RoboCasa is achieved by the specialist, while the generalist reaches 53.3% and preserves strong LIBERO performance.The largest gain appears in Pick & Place, reaching 36–40% success versus π0.5 at 21.5%.
  • Few-shot adaptation: +25.8% average gain over the baseline appears when Und + ViT are frozen and the projector and action expert train.In single-task LIBERO-Long, the corresponding improvement reaches +41.6%.
  • Deployment robustness: Removing UAC sharply harms long-horizon performance, while disabling MPG+UAC makes bimanual coordination brittle under noisy or shifting observations.The reported failures include drifting grasps, premature contact, failed handle engagement, oscillation, and indecisive corrections.

8 Conclusion

Being-H0.5 presents a human-centric, scalable approach to cross-embodiment robot learning by combining unified human–robot motion representations, large-scale pretraining, and modular execution mechanisms.

  • Conclusion: Being-H0.5 unifies human and robotic motion in a physically grounded state–action space using the large-scale UniHand-2.0 dataset.The model uses human interaction traces as physical priors across diverse robotic morphologies.
  • Conclusion: The MoT architecture separates high-level multimodal reasoning from low-level motor execution, while MPG and UAC target sensory shifts and hardware-specific inference latencies.These mechanisms are presented as supporting robust deployment across embodiments.
  • Conclusion: Evaluation across five robotic platforms shows that a single Being-H0.5 checkpoint can handle complex interaction dynamics, including bimanual coordination and long-horizon tasks.The conclusion frames this as progress toward general-purpose robotic intelligence across environments and embodiments.

Human Data

Figure 14 presents example rollouts from all real-robot task suites.

  • Figure 14 shows example rollouts from all real-robot task suites.
Loading 2601.12993v1…