PaperlayerView exact revision on arXiv

Source-linked AI summary

UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling

Boyu Chen, Yi Chen, Lu Qiu, Jerry Bai, Yuying Ge, Yixiao Ge

arXiv:2604.19734v1cs.ROcs.AI

TL;DR

Humanoid foundation-model development is limited by scarce robotic data and mismatched human-to-robot kinematics. UniT uses visual-anchored cross-reconstruction to create shared physical-intent tokens for policy learning and world modeling. The resulting systems support human-data transfer, OOD generalization, zero-shot task transfer, and cross-embodiment dynamics modeling, while the tokenizer’s design depends on combining vision and action with explicit alignment.

  • Problem

    Humanoid foundation models face scarce high-quality robotic data and heterogeneous human-robot state-action spaces caused by biomechanical and hardware differences.

  • Method

    UniT uses visual anchoring, tri-branch cross-reconstruction, and a fusion branch to map vision and actions into a shared discrete latent space.

  • Results

    UniT improves policy data efficiency and OOD generalization, enables zero-shot human-to-humanoid transfer, and strengthens cross-embodiment dynamics modeling and control generation.

  • Takeaways & Limitations

    Unified physical-intent tokens provide a common interface for leveraging human data in humanoid policy learning and world modeling.

  • Takeaways & Limitations

    UniT’s design assumes both vision and action are needed and that explicit cross-reconstruction is required to align the modalities.

Abstract

from arXiv · show

Scaling humanoid foundation models is bottlenecked by the scarcity of robotic data. While massive egocentric human data offers a scalable alternative, bridging the cross-embodiment chasm remains a fundamental challenge due to kinematic mismatches. We introduce UniT (Unified Latent Action Tokenizer via Visual Anchoring), a framework that establishes a unified physical language for human-to-humanoid transfer. Grounded in the philosophy that heterogeneous kinematics share universal visual consequences, UniT employs a tri-branch cross-reconstruction mechanism: actions predict vision to anchor kinematics to physical outcomes, while vision reconstructs actions to filter out irrelevant visual confounders. Concurrently, a fusion branch synergies these purified modalities into a shared discrete latent space of embodiment-agnostic physical intents. We validate UniT across two paradigms: 1) Policy Learning (VLA-UniT): By predicting these unified tokens, it effectively leverages diverse human data to achieve state-of-the-art data efficiency and robust out-of-distribution (OOD) generalization on both humanoid simulation benchmark and real-world deployments, notably demonstrating zero-shot task transfer. 2) World Modeling (WM-UniT): By aligning cross-embodiment dynamics via unified tokens as conditions, it realizes direct human-to-humanoid action transfer. This alignment ensures that human data seamlessly translates into enhanced action controllability for humanoid video generation. Ultimately, by inducing a highly aligned cross-embodiment representation (empirically verified by t-SNE visualizations revealing the convergence of human and humanoid features into a shared manifold), UniT offers a scalable path to distill vast human knowledge into general-purpose humanoid capabilities.

1 Introduction

UniT addresses scarce robotic data and cross-embodiment mismatch by anchoring heterogeneous actions to visual consequences and fusing purified vision-action information into shared physical-intent tokens. Integrated into policy learning and world modeling, it supports data-efficient, OOD, and human-to-humanoid transfer.

  • Motivation: Humanoid foundation-model scaling is constrained by scarce high-quality robotic data, while human motion offers a scalable but cross-embodiment-mismatched alternative.Biomechanical and hardware differences create heterogeneous state-action spaces, making traditional case-specific motion retargeting labor-intensive and physically inconsistent.
  • UniT: UniT uses visual anchoring and tri-branch cross-reconstruction to align kinematic actions with physical outcomes while filtering irrelevant visual and action-domain information.Its temporal-visual, kinematic, and fused visuo-motor representations are jointly trained to reconstruct both visual transitions and low-level actions.
  • UniT: UniT fuses heterogeneous data into a shared discrete latent space whose tokens represent universal physical intentions.The architecture is designed to avoid action-only distribution misalignment, vision-only appearance entanglement, and disconnected modality spaces.
  • Policy Learning: VLA-UniT predicts unified tokens before a lightweight flow head generates embodiment-specific actions, improving data efficiency and OOD generalization while enabling zero-shot task transfer.The evaluation covers the RoboCasa GR1 benchmark and a real humanoid, with emergent upper-body coordination reported in deployment.
  • World Modeling: WM-UniT conditions world models on unified tokens, aligning human and humanoid dynamics and improving downstream humanoid control generation.Rollout validation reports that large-scale human-data pre-training enables transfer of physical dynamics across embodiments.
  • Additional Benefits: UniT also induces aligned cross-embodiment representations and denoises captured actions into cleaner, more executable trajectories.These structural benefits appear in both VLA policies and world models.

2 Related Work

Related work studies human-data transfer, latent action representations, and embodied policy or world-model architectures. Existing approaches provide useful foundations but leave limitations in fine-grained motor information, external grounding, or cross-embodiment action generation.

  • Learning from Human Data: Human-data approaches use egocentric videos for scalable visual pre-training, but often omit fine-grained hand or wrist pose information.This limits their utility for dexterous upper-body control.
  • Latent Action Representations: Action-only latent representations compact raw trajectories for behavior generation but lack external grounding across embodiment-specific distributions.VQ-BeT and FAST are cited as examples of structured action tokenization.
  • VLA and World Models: Vision-language-action models and action-conditioned world models are core embodied-AI paradigms, with cross-embodiment systems commonly generating raw actions or proprioceptive action tokens.The related systems include GR00T, π0, RT-X, and Octo.

3 Methodology

UniT builds a unified discrete latent action space by jointly encoding visual transitions, actions, and fused visuo-motor features with cross-reconstruction. Its tokens support both VLA policy prediction and world-model conditioning across embodiments.

  • 3.1 Overview: UniT jointly models visual transitions, action chunks, and fused visuo-motor features to produce embodiment-agnostic physical-intent tokens.The tri-branch architecture encodes each modality before shared quantization and downstream use.
  • 3.4 WM-UniT: WM-UniT uses action-branch UniT features as universal control conditions for future-frame generation, replacing embodiment-specific raw actions.The world model projects these features into cross-attention and supports autoregressive rollout.
  • 3.2 UniT: A shared RQ-VAE codebook places all branch latents and embodiments in one discrete space, capturing coarse intent and fine-grained motion detail.Residual quantization progressively refines representations across multiple codebook levels.
  • 3.2 UniT: Cross-reconstruction decodes every token into visual transitions and embodiment-specific actions, aligning tokens with physical consequences and relative motion.The visual decoder uses current observations and cosine supervision, while the action decoder uses current state to reconstruct action chunks.
  • 3.3 VLA-UniT: VLA-UniT predicts UniT tokens from vision-language context, then a lightweight flow-matching action expert generates embodiment-specific controls.This separates cross-embodiment token prediction from continuous action generation while using shared context.
  • 3.3 VLA-UniT: The VLA objective combines token cross-entropy and flow-matching losses, with both stages conditioned on the same vision-language features.The shared context is intended to support transfer across heterogeneous human and humanoid behaviors.

4 Experimental Setups

The experiments evaluate UniT for humanoid policy learning and world modeling across simulation, real-world deployment, OOD conditions, and representation alignment.

  • Simulation Benchmark: RoboCasa GR1 evaluation uses 24 tabletop tasks, including pick-and-place rearrangement and articulated multi-step manipulation.Each task is assessed over 50 simulation episodes.
  • Data Regimes: Full-data training uses 24,000 robot trajectories, while few-shot training uses 2,400 trajectories, or 10% of the full set.The few-shot regime contains 100 trajectories per task.
  • Human Data: Human-to-humanoid transfer combines few-shot robot data with 27,419 EgoDex pick-and-place trajectories before robot-only fine-tuning.The human subset is basic_pick_place.
  • Simulation Generalization: Generalization tests cover unseen appearances, object combinations, and object types in 18, 23, and 32 tasks, respectively.These suites vary visual textures, container pairings, and object categories.
  • Real-World Evaluation: Real-world evaluation uses the IRON-R01-1.11 humanoid on Pick & Place and Pouring, with 120 robot trajectories per task.The robot has a 50-dimensional action space, and training mixes proprietary robot data with EgoDex data.
  • Real-World Generalization: Real-world OOD tests introduce geometry, target, distractor, visual, and combinational variations that human demonstrations complement beyond robot-data coverage.The setup co-trains on both sources and evaluates on human-introduced conditions.
  • Variants and Metrics: Comparisons include VLA baselines, tokenizer ablations, raw-action and latent-token world-model conditions, and PSNR, SSIM, LPIPS, FVD, and EPE metrics.EPE is End-Point Error computed from optical flow.

5 Experiments

The experiments test whether UniT aligns embodiments robustly, improves humanoid policy learning, enhances world modeling, and supports the proposed design.

  • Evaluation Questions: The evaluation is organized around unified representation, efficient policy learning, effective world modeling, and design soundness.These questions correspond to Sections 5.1 through 5.3 and the ablation analysis.
  • Evaluation Questions: The central tests ask whether UniT aligns heterogeneous embodiments, remains noise-robust, improves data efficiency and generalization, and enables cross-embodiment dynamics transfer.The design question examines the contribution of the tokenizer components.
  • Evaluation Scope: The experiments assess both downstream policy execution and action-conditioned video generation rather than evaluating representation alignment in isolation.The listed questions connect representation properties to policy and world-model outcomes.

5.1 Unified Representation: Alignment and Robustness

UniT maps separated human and humanoid action distributions into overlapping token and downstream feature spaces while remaining comparatively robust to action noise.

  • Token Alignment: Raw human and humanoid actions form separated clusters, whereas UniT token embeddings become highly overlapping in the shared manifold.The comparison uses t-SNE on raw action trajectories and UniT token embeddings.
  • Noise Robustness: At σ = 0.2, FAST degrades by 10.7×, the action-only tokenizer by 2.7×, and UniT by only 1.7× in relative reconstruction error.Degradation is MSEnoisy/MSEclean, with σ normalized by the dataset’s global action standard deviation.
  • Downstream Alignment: VLA-UniT produces more interleaved human and humanoid internal features than vanilla VLA, while WM-UniT brings world-model features into a unified distribution.The analysis uses mean-pooled vision-language features and cross-attention outputs.

5.2 Policy Learning

VLA-UniT predicts visually anchored tokens instead of raw actions, achieving strong simulation efficiency and performance while transferring human demonstrations to real-world OOD and unseen tasks.

  • Policy Learning: VLA-UniT predicts shared UniT tokens, then uses a lightweight flow head to generate embodiment-specific actions for execution.This design is evaluated on RoboCasa GR1 and a real humanoid.
  • Benchmark Performance: 66.7% overall success on full-data RoboCasa exceeds FLARE’s 55.0% by 11.7% and GR00T’s 47.8% by 18.9%.VLA-UniT reaches 67.3% on Pick & Place and 64.7% on Articulated tasks.
  • Data Efficiency: 45.5% success with 10% of training data approaches the 47.8% achieved by GR00T with full data.The few-shot setting uses 100 trajectories per task, representing approximately a 10× reduction in data.
  • Human-to-Humanoid Transfer: Human co-training raises simulation in-domain performance from 45.5% to 50.0% and OOD performance from 34.7% to 38.5%.The largest in-domain gain occurs in Pick & Place, increasing from 41.7% to 49.4%.
  • Real-World In-Domain Performance: On real-world Pick & Place and Pouring, human co-training increases VLA-UniT performance to 78% and 75%, respectively.Robot-only performance is 70% and 35% on the same tasks.
  • Real-World OOD Generalization: Real-world co-training improves Geometry from 23.3% to 63.3%, Distractor from 26.7% to 60.0%, and Combinational performance from 10% to 70%.Human demonstrations provide complementary variation beyond partial robot-data coverage.
  • Zero-Shot Task Transfer: Zero-shot stacking success rises from 10% without human data to 60% with human co-training, while the GR00T and unaligned-token baselines score 0%.The transferred behavior includes waist rotation and head turning for viewpoint adjustment.

5.3 World Modeling

WM-UniT uses unified token conditioning to improve controllability and cross-embodiment transfer in humanoid video world modeling. Human pre-training and direct cross-embodiment conditioning preserve action semantics, magnitudes, phases, and temporal structure.

  • World-model performance: WM-UniT provides a stronger conditioning interface for cross-embodiment dynamics modeling and human-to-humanoid transfer than raw actions.Its evaluations cover co-training, human pre-training, and direct transfer between human and humanoid domains.
  • World-model performance: Human pre-training on EgoDex basic_pick_place consistently improves downstream RoboCasa-GR1 world-model metrics, especially humanoid controllability.The model is pre-trained on human demonstrations and fine-tuned on humanoid pick-and-place data.
  • Cross-embodiment conditioning: UniT preserves fine-grained semantics, motion magnitude, and non-monotonic temporal structure when conditioning robot videos from human actions.It distinguishes exploratory from extended reaches and reproduces reach-then-retract trajectories better than Raw Action conditioning.
  • Cross-embodiment conditioning: UniT transfers multi-phase robot actions into human video generation, retaining grasp semantics, wrist rotation, tip-down motion, and phase ordering.Raw Action conditioning captures only the coarse trajectory and collapses atomic actions into a flat reach.
  • Cross-embodiment conditioning: UniT consistently outperforms Raw Action across semantic, temporal, and geometric consistency in both human-to-robot and robot-to-human directions.The evaluation scores action-intent preservation, sequencing fidelity, and spatial or pose precision on a 1–5 scale.

5.4 Tokenizer Design Ablation

Tokenizer ablations show that combining vision and action is insufficient without explicit bidirectional cross-reconstruction. UniT’s joint modality encoding and cross-reconstruction produce the strongest cross-embodiment transfer and in-domain performance.

  • Vision-Action Synergy: 49.9% OOD average for VLA-UniT exceeds VLA-Vision at 45.2% and VLA-Action at 42.1% across all tested OOD scenarios.The joint representation combines vision’s transferable signal with action’s fine-grained motor detail.
  • Cross-Reconstruction: 30.3% for VLA-UniT without cross-reconstruction falls below single-modality variants, while full VLA-UniT gains 19.6% over this ablation.The comparison indicates that multimodal input alone does not guarantee aligned cross-embodiment representations.
  • Bidirectional Reconstruction: 66.8% for bidirectional VLA-UniT exceeds 63.1% for unidirectional VLA-Villa on in-domain performance.The ablation compares bidirectional cross-reconstruction with unidirectional V2A reconstruction.

6 Conclusion and Discussion

UniT establishes a visual-anchored unified physical language for human-to-humanoid transfer through cross-reconstruction. Across policy learning and world modeling, it improves transfer and representation alignment, while the visual branch suggests a route to using unlabeled internet video.

  • Conclusion: UniT is a visual-anchored latent action tokenizer that establishes a unified physical language through cross-reconstruction.The framework targets human-to-humanoid transfer across policy learning and world modeling.
  • Conclusion: VLA-UniT improves policy performance and data efficiency while supporting OOD generalization and zero-shot human-to-humanoid task transfer.These results are reported across simulation and real-world humanoid settings.
  • Conclusion: WM-UniT provides a stronger conditioning interface for cross-embodiment dynamics modeling and human-to-humanoid transfer.The conclusion identifies policy and world modeling as applications of the same unified interface.
  • Discussion: UniT’s visual branch encodes physical transitions from observations alone, opening a path toward learning physical priors from unlabeled internet video.This direction does not require paired action annotations for the visual branch.
Loading 2604.19734v1…