PaperlayerView exact revision on arXiv

Source-linked AI summary

Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos

Hao Luo, Yicheng Feng, Wanpeng Zhang, Sipeng Zheng, Ye Wang, Haoqi Yuan, Jiazheng Liu, Chaoyi Xu, Qin Jin, Zongqing Lu

arXiv:2507.15597v1cs.CVcs.LGcs.RO

TL;DR

Existing VLAs lack scalable, diverse dexterous manipulation data and struggle with novel scenarios. Being-H0 addresses this gap with physical instruction tuning on large-scale human videos, explicit motion modeling, physical-space alignment, and robotic post-training, and reports robust cross-modal reasoning and transfer to robotic manipulation.

  • Problem

    Existing VLAs rely on synthetic or limited teleoperated data, leaving them weak on complex dexterous manipulation and novel scenarios.

  • Method

    Being-H0 uses physical instruction tuning with human-video pretraining, physical-space alignment, post-training adaptation, part-level motion tokenization, and heterogeneous data curation.

  • Results

    Being-H0 achieves robust cross-modal reasoning across vision, language, and fine-grained hand motions, with adaptation strategies for downstream robotic manipulation.

  • Takeaways & Limitations

    Human hands can serve as a foundation manipulator for transferring dexterity from large-scale human videos to robotic learning.

  • Takeaways & Limitations

    The approach does not yet incorporate richer physical cues such as depth or tactile feedback, which may improve grounded representations.

Abstract

from arXiv · show

We introduce Being-H0, a dexterous Vision-Language-Action model (VLA) trained on large-scale human videos. Existing VLAs struggle with complex manipulation tasks requiring high dexterity and generalize poorly to novel scenarios and tasks, primarily due to their reliance on synthetic data with significant sim-to-real gaps or teleoperated demonstrations lacking scale and diversity. To address this data bottleneck, we propose leveraging human hands as a foundation manipulator, capitalizing on the rich dexterity and scalability present in web data. Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks. Additionally, we introduce a part-level motion tokenization method which achieves millimeter-level reconstruction accuracy to model precise hand trajectories for action learning. To support our proposed paradigm, we further develop a comprehensive data curation pipeline that integrates heterogeneous sources -- including motion capture, VR, and RGB-only videos -- into a large-scale dataset with millions of motion-based instructional instances. We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes. Importantly, we observe the expected gains of Being-H0 in real-world robotic manipulation as physical instruction tuning is applied. More details are available at https://beingbeyond.github.io/Being-H0.

1 Introduction

Being-H0 addresses the limited scale, diversity, and dexterity of existing VLA training by pretraining on human hand videos and transferring that knowledge to robotic manipulation through physical instruction tuning.

  • Motivation: Existing VLAs struggle with novel scenarios and complex manipulation because synthetic and teleoperated datasets lack sufficient realism, scale, and diversity.These limitations are especially severe for dexterous hands, whose hardware costs and fine-finger requirements restrict available real-world data.
  • Motivation: Human videos provide abundant real-world demonstrations, but effective transfer requires explicit physical-space alignment between 2D observations and 3D robotic actions.Prior implicit approaches and limited-scale human-centric representations do not clearly establish the learning and transfer mechanisms needed for VLAs.
  • Approach: Physical Instruction Tuning combines VLA pretraining on human videos, physical-space alignment, and post-training adaptation for robotic tasks.The paradigm extends visual instruction tuning into the physical domain while addressing heterogeneous cameras, coordinate frames, and 3D reasoning.
  • Approach: Part-Level Motion Tokenization preserves millimeter-level precision in continuous hand motions while remaining compatible with discrete autoregressive language-model architectures.The method is designed to model fine-grained hand trajectories for action learning.
  • Data: UniHand contains over one hundred fifty million instruction-following samples collected from motion capture, VR, and RGB-only videos.Its scalable curation pipeline unifies heterogeneous sources across diverse manipulation scenarios.
  • Contributions: Being-H0 is presented as a dexterous VLA trained on motion-based human video data at scale for cross-modal reasoning and downstream robotic manipulation.The model integrates the proposed physical instruction tuning, motion tokenization, and large-scale data curation components.

2 Related Work

Prior work advances multimodal and human-motion modeling, but VLA training remains constrained by limited dexterous data and substantial sim-to-real or human-to-robot alignment challenges.

  • Large Multimodal Models: Large multimodal models combine language reasoning with specialized visual encoders, while visual instruction tuning improves multimodal instruction following.These developments motivate adapting multimodal foundations to physical action prediction.
  • Human-body Motion Generation: Human-body motion datasets have expanded from professional benchmarks to million-sequence collections supported by parametric body models.This literature provides scalable precedents for modeling annotated and expressive human motion.
  • Hand Motion Generation: Hand-motion benchmarks capture hand-object interaction and precision, but mocap and multi-camera systems limit diversity while egocentric videos often lack precise annotations.Monocular reconstruction and 3D hand modeling are emerging ways to address these complementary limitations.
  • Hand Motion Generation: Hand-object interaction methods increasingly use diffusion or autoregressive models, yet most historically overlook visual inputs.The field has moved from 2D recognition toward 3D motion generation and unified temporal modeling.
  • Learning VLAs from Human Videos: VLA systems have expanded through large-scale pretraining, but available datasets remain small lab collections and mostly emphasize end-effector control.Simulation offers scalability, but the sim-to-real gap remains substantial for real-world deployment.
  • Learning VLAs from Human Videos: Human-video approaches extract visual, perceptual, and interaction representations, but existing methods do not explicitly map human motions to robotic motions.Visual and action-space alignment methods generally remain limited to simplified grippers and neglect fine-grained finger alignment.

3 Overview of Being-H0

Being-H0 uses physical instruction tuning to train a dexterous VLA on human hand motion, align heterogeneous physical observations, and adapt the foundation model to robot control.

  • 3.1 Motivation: Physical instruction tuning consists of pretraining, physical-space alignment, and post-training adaptation.The paradigm is proposed to address the limitations of existing VLA training for dexterous manipulation.
  • Pretraining: Pretraining treats the human hand as an ideal manipulator and predicts MANO-parameterized hand motions from visual input and language instructions.The target motion includes joint angles, wrist rotation, translation, and shape parameters.
  • Pretraining: The unified formulation supports vision-to-motion generation, motion captioning, and multimodal conditioning for diverse hand-object interaction scenarios.Text and motion targets are modeled as instruction-following outputs of the foundational model.
  • Physical Space Alignment: Physical-space alignment addresses varying camera intrinsics, dynamic world coordinates, missing 3D spatial priors, and heterogeneous physical properties.The alignment stage extends standard visual instruction tuning to the physical domain.
  • 3.1 Motivation: The central questions are whether large-scale human videos can pretrain dexterous VLAs to imitate human actions and whether post-training can transfer them to robots.These questions target the pretraining-to-downstream mismatch in embodied learning.
  • Hand Quantization: Grouped residual quantization discretizes continuous hand motions into motion tokens while achieving millimeter-level reconstruction accuracy.Tokens are organized into coherent motion blocks delimited by <MOT> and </MOT> for integration with text.
  • Cross-Modal Reasoning: Unified cross-modal reasoning represents text, vision, and motion in one token sequence with shared attention across modalities.This design models dependencies between visual scenes, language instructions, and hand motions.
  • Robot Control Transfer: Post-training uses a lightweight MLP-based projection and learnable action-chunk queries to adapt the pretrained foundation VLA to downstream manipulation.The projection accommodates kinematic mismatches, differing degrees of freedom, and physical constraints.

4 Physical Instruction Tuning

Physical Instruction Tuning combines human-video VLA pretraining, physical-space alignment, and post-training adaptation for robotic manipulation. Its part-level motion tokenization represents dexterous hand motion discretely while preserving precision, and the broader pipeline integrates heterogeneous data sources.

  • 4.1 ⋆Pretraining: Being-H0 pretrains a foundation VLA on human videos to support downstream manipulators through kinematic mapping.The model processes vision, text, and hand motion as modalities and is intended to generalize from human-hand dexterity to robotic devices.
  • 4.1.2 Hand Motion Tokenization: The tokenizer converts T-frame hand features into ⌈T/α⌉ discrete token embeddings, with α controlling temporal downsampling.This makes continuous hand motion compatible with autoregressive token prediction.
  • 4.1.3 Multimodal Integration: The training framework extends vision-text parameters with motion parameters during pretraining, then adds action parameters during post-training for direct robotic control.Shared attention integrates visual, textual, and motion tokens before downstream action modeling.
  • 4.1.2 Hand Motion Tokenization: Experiments find 6D rotations better for finger-joint reconstruction and axis-angle features better for wrist poses, but the model selects MANO-D162 for training.The choice reflects stronger Being-H0 training performance despite lower overall reconstruction error with axis-angle features.
  • 4.1.2 Hand Motion Tokenization: GRQ groups feature channels and applies residual-stage quantization, while a wrist-specific reconstruction term addresses errors from wrist parameters’ broad 3D distribution.The overall objective combines reconstruction, commitment, and wrist losses; the codebook uses exponential moving-average updates.
  • 4.1.2 Hand Motion Tokenization: Part-level tokenization separates wrist motion {rrot, τ} from finger motion {θ, β} to model global positioning and fine-grained manipulation independently.Separate wrist and finger tokenizers provide explicit token semantics for structured hand dynamics.
  • 4.2 ⋆Physical Space Alignment: Physical space alignment maps videos from different cameras into a unified weak-perspective space with a shared 3D reference frame.The alignment addresses inconsistent projections and supports spatial reasoning across heterogeneous recordings.
  • 4.2.3 Further Discussion: Richer physical cues such as depth and tactile feedback are identified as future additions for more grounded representations and reliable deployment.These cues are proposed because 2D visual inputs can leave physical interactions and 3D structure ambiguous.

5 UniHand: Scaling up Hand Motion Instructional Data

UniHand scales dexterous hand-motion supervision by combining heterogeneous motion sources, standardized 3D representations, hierarchical descriptions, and multiple instruction-generation tasks. The resulting dataset contains over 165 million motion-instruction pairs, with a balanced 2.5-million-instance subset.

  • 5.1 Data Sources and Statistics: The pipeline uses motion capture, VR, and RGB-only data to scale human-video pretraining while addressing variation in camera systems and recording conditions.The dataset design targets large-scale dexterous VLA learning from heterogeneous human videos.
  • 5.1 Data Sources and Statistics: UniHand aggregates 11 sources spanning motion capture, VR recordings, and RGB-based pseudo-annotation into over 440K trajectories, 130 million frames, and 1,100 hours of video.The sources trade off annotation precision, naturalism, and diversity.
  • 5.2.1 Hand Pose Standardization: Hand pose standardization represents tracked motions with MANO parameters and learns an explicit mapping from 2D observations to 3D spatial coordinates.The procedure unifies heterogeneous motion labels across datasets.
  • 5.2.2 Task Description Labeling: Hierarchical labeling separately describes global two-handed and individual hand actions, linking high-level task objectives with fine-grained hand-object interactions.This provides multi-scale semantic grounding for vision, language, and motion.
  • 5.2.3 Instructional Data Generation: Instructional data generation includes step-by-step motion generation, motion translation, and contextual motion prediction conditioned on visual, textual, and motion context.Approximately 20 templates per task type are expanded into diverse variants with duration specifications and explicit length constraints.
  • 5.2.3 Instructional Data Generation: Over 165 million motion-instruction pairs provide supervision across temporal scales, handedness configurations, manipulation scenarios, and systematic quality checks.A 2.5-million-instance subset, UniHand-2.5M, is sampled to balance data sources and task types.

6 Experiments

The experiments evaluate Being-H0’s vision-language-motion grounding through complementary generation and translation tasks, then test transfer to real-world dexterous manipulation. The protocol measures spatial, temporal, semantic, and robotic task performance across diverse settings.

  • 6.1 Experimental Setup: The motion tokenizer downsamples sequences with α = 4 and converts each one-second motion sequence into 128 tokens using grouped residual quantization.The implementation uses an 8-layer GRQ architecture with group size n = 2.
  • 6.1 Experimental Setup: Evaluation reserves 5% of UniHand videos with paired motions and text, while augmentation broadens the wrist-translation coverage beyond the EgoDex-centered distribution.This creates a held-out evaluation set with varied source distributions.
  • 6.1 Experimental Setup: Three benchmark tasks assess visual-grounded generation, contextualized generation, and hand-motion translation from complementary generation and understanding perspectives.Together they evaluate 2D-to-3D grounding, temporal coherence, and semantic alignment.
  • 6.1 Experimental Setup: MPJPE, MWTE, and PA-MPJPE measure spatial accuracy and trajectory fidelity, while M2T R@3 and FID assess semantic alignment and distribution similarity.The translation task additionally uses T2M R@3 and valid generation rate.
  • 6.1.3 Evaluation on Dexterous Manipulation: Real-robot evaluation uses a 7-DoF Franka Research 3 arm, 6-DoF Inspire hand, and RealSense cameras across grasping, articulated-object, and deformable-object tasks.The hardware setup supports evaluation of precision, generalization, and complex manipulation.
  • 6.1.3 Evaluation on Dexterous Manipulation: Success rate is measured over 20 randomized trials per task, comparing Being-H0 with GR00T N1.5 and InternVL3 after identical post-training on teleoperation data.The comparison isolates the contribution of human-hand pretraining and physical alignment under matched post-training.

6.2 Comparisons on Hand Motion Generation and Translation

Being-H0’s hand-motion generation and translation performance improves with model scale across the evaluated tasks. Larger variants also better preserve the required motion format and strengthen bidirectional motion-language alignment.

  • 6.2 Comparisons on Hand Motion Generation and Translation: 99.8% and 100% valid generation rates are achieved by Being-H0-8B and Being-H0-14B, respectively, compared with modest structural validity for Being-H0-1B.The results use free-format visual-grounded motion generation with required <MOT>......</MOT> blocks.
  • 6.2 Comparisons on Hand Motion Generation and Translation: Larger Being-H0 models consistently achieve higher T2M R@3 scores on both head and tail splits in hand-motion translation.T2M R@3 measures whether generated text retrieves the corresponding motion sequence.
  • 6.2 Comparisons on Hand Motion Generation and Translation: Table 3 compares visual-grounded and contextualized hand-motion generation on head and tail splits using MPJPE, MWTE, PA-MPJPE, and M2T R@3.The head split is sourced from EgoDex, while the tail split uses TACO, HOI4D, H2O, and OakInk2.
  • 6.2 Comparisons on Hand Motion Generation and Translation: Larger models consistently obtain lower MPJPE, MWTE, and PA-MPJPE scores, indicating improved spatial grounding and pose plausibility.Table 3 reports these metrics in centimeters on head and tail splits for visual-grounded and contextualized generation.

6.3 Comparisons on Long-range Motion Generation

Longer generated motions show accumulated error and quality degradation, but larger Being-H0 models retain more stable spatial accuracy under soft constraints. The evaluation separates short-term and long-term sequence behavior.

  • 6.3 Comparisons on Long-range Motion Generation: Soft-formatted inference constrains generated motions within plausible ranges for evaluating quality degradation as sequence length increases.The evaluation reports MPJPE, MWTE, and PA-MPJPE for short-term and long-term ranges.
  • 6.3 Comparisons on Long-range Motion Generation: MPJPE and MWTE increase in the 6–10-second long-term range because prediction errors accumulate over longer sequences.The comparison uses 2–5 seconds as short-term and 6–10 seconds as long-term motion.
  • 6.3 Comparisons on Long-range Motion Generation: Larger models maintain more stable spatial accuracy during long-range generation under soft constraints.Partial ground-truth context helps anchor the generated trajectory.

6.4 Ablation Study

The ablations evaluate motion-tokenizer design, auxiliary supervision, view-invariant balancing, and training scale for hand reconstruction and visual-grounded motion generation. Part-level tokenization and selected data components improve generation, while performance generally scales with more training samples.

  • Hand Motion Tokenization: Part-level tokenization outperforms uniform whole-hand quantization under matched token counts and codebook dimensionality on UniHand’s held-out motion sequences.The comparison includes 4-group and 16-layer residual quantization variants with codebook size K = Kw + Kf = 8192.
  • Hand Motion Tokenization: 6D rotation with MANO-D162 is most effective for Being-H0 training, although axis-angle features provide superior overall reconstruction accuracy and 6D rotation improves PA-MPJPE.Auxiliary joint positions further enhance performance.
  • Hand Motion Tokenization: MANO-D162 plus Part-Level consistently outperforms alternative tokenizer configurations in hand motion generation despite slightly higher reconstruction MPJPE.The authors attribute this to 6D rotation and part-level decomposition supporting temporal modeling and autoregressive fine-grained motion generation.
  • Data Recipe: Removing translation supervision causes marginal changes in MPJPE and MWTE but clear degradation in PA-MPJPE, M2T R@3, and FID.This indicates translation data benefits capabilit[ies] beyond global wrist motion.
  • Data Scale: Performance improves steadily as training size increases to 2.5M samples, while M2T R@3 continues rising even though PA-MPJPE slightly drops at the largest scale.Figure 9 reports normalized MPJPE, MWTE, PA-MPJPE, M2T R@3, and FID on core and tail splits.

6.5 Hand Motion Generation Samples

Being-H0 generates physically plausible hand motions across varied task instructions, scenes, viewpoints, and interaction depths. The qualitative samples include both single-hand and dual-hand behaviors rendered in unified camera coordinates.

  • Qualitative Samples: Figure 10 presents generated hand-motion sequences for diverse task instructions, scenes, and camera viewpoints.Each sequence is rendered over the first video frame in its camera coordinate system, with yellow and blue indicating left and right hands.
  • Visualization: Black padding enforces a consistent weak-perspective projection, leaving the effective image region to reflect hand-object interaction depth.Closer interactions appear larger after transformation.
  • Qualitative Samples: The model handles both single-hand and dual-hand motions across a wide range of manipulation tasks and viewpoints.The samples are presented as qualitative evidence of generalization under varied viewpoints.

6.6 Real-World Experiments

Real-world experiments show that physical instruction tuning improves dexterous robotic manipulation and generalization beyond the non-pretrained baseline. Being-H0 also retains strong performance with substantially reduced demonstration data.

  • Success Rates: Being-H0 achieves the highest success rates across all evaluated real-world dexterous manipulation tasks over 20 randomized trials.Table 7 compares Being-H0 with baseline models; InternVL3 uses the same VLA architecture and parameters without physical instruction tuning.
  • Generalization: Being-H0 generalizes from seen to unseen objects and cluttered scenes in Pick-Place-Toy, including correctly retrieving a specified target among distractors.The examples include the seen yellow duck, unseen green duck, and white duck in clutter.
  • Baseline Comparison: InternVL3 exhibits imprecise, misaligned, unstable, and height-miscalibrated behaviors across Close-Toolbox, Close-Lid, Pour-Cup, and Unfold-Clothes.The reported failures include insufficient lid-closing force, incorrect lid placement, insecure cup grasping, and missing the cloth edge.
  • Baseline Comparison: Being-H0 accurately positions lids, pinches cloth edges, and maintains stable cup grasps, transferring hand-motion knowledge to downstream robotic tasks.These qualitative outcomes cover the fine-grained manipulation behaviors contrasted with InternVL3.
  • Data Efficiency: Being-H0 matches the baseline with 25% of Pick-Place-Toy data versus the baseline’s 100%, and matches it with 25% versus 50% on Close-Toolbox and Unfold-Clothes.On Close-Lid, Being-H0 reaches 15% with 25% data while the baseline reaches 0%.

7 Conclusion and Future Work

The paper presents Being-H0 as a scalable, sample-efficient dexterous VLA trained on human videos through physical instruction tuning. It integrates heterogeneous motion data and identifies expanded physical sensing, complex tasks, and simulation-based learning as future directions.

  • Conclusion: Being-H0 combines human-video pretraining, physical space alignment, and post-training to transfer dexterous human-hand motion knowledge to robotic manipulation.The approach addresses the pretraining–downstream data mismatch when adapting LMMs into VLAs.
  • Conclusion: UniHand integrates motion capture, VR recordings, and RGB-only videos through standardization and projection alignment to address real-robot data scarcity.The conclusion describes the dataset as unified and million-level, while the contribution passage reports over one hundred fifty million instruction-following samples.
  • Future Work: Future work includes depth and tactile feedback, tool use, multi-object interactions, long-horizon reasoning, simulation, and reinforcement learning.These directions target stronger physical plausibility, more robust policy learning, and safer deployment.
Loading 2507.15597v1…