Source-linked AI summary
HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulation
Shuanghao Bai, Meng Li, Xinyuan Lv, Jiawei Wang, Xinhua Wang, Fei Liao, Chengkai Hou, Langzhe Gu, Wanqi Zhou, Kun Wu, Ziluo Ding, Zhiyuan Xu, Lei Sun, Shanghang Zhang, Zhengping Che, Jian Tang, Badong Chen
TL;DR
Whole-body humanoid manipulation remains difficult because existing VLA approaches often do not explicitly model coupled body-part dynamics under shared balance and posture. HEX addresses this gap with humanoid-aligned state representations, predictive proprioception, temporal visual review, and adaptive multimodal action generation. On real-world tasks, it achieves superior success and generalization, particularly for fast-reaction and long-horizon scenarios.
Problem
Existing VLA approaches often lack explicit modeling of coordinated body-part interactions under shared balance and posture, limiting whole-body humanoid manipulation.
Method
HEX combines a humanoid-aligned universal state representation, predictive proprioceptive modeling, visual-history summarization, and adaptive fusion for whole-body action generation.
Results
HEX achieves superior task success and generalization over strong baselines on real-world humanoid manipulation tasks, especially in fast-reaction and long-horizon settings.
Takeaways & Limitations
HEX provides an effective framework for coordinated whole-body manipulation on full-sized bipedal humanoid robots.
Abstract
from arXiv · showhide
Humans achieve complex manipulation through coordinated whole-body control, whereas most Vision-Language-Action (VLA) models treat robot body parts largely independently, making high-DoF humanoid control challenging and often unstable. We present HEX, a state-centric framework for coordinated manipulation on full-sized bipedal humanoid robots. HEX introduces a humanoid-aligned universal state representation for scalable learning across heterogeneous embodiments, and incorporates a Mixture-of-Experts Unified Proprioceptive Predictor to model whole-body coordination and temporal motion dynamics from large-scale multi-embodiment trajectory data. To efficiently capture temporal visual context, HEX uses lightweight history tokens to summarize past observations, avoiding repeated encoding of historical images during inference. It further employs a residual-gated fusion mechanism with a flow-matching action head to adaptively integrate visual-language cues with proprioceptive dynamics for action generation. Experiments on real-world humanoid manipulation tasks show that HEX achieves state-of-the-art performance in task success rate and generalization, particularly in fast-reaction and long-horizon scenarios.
1 Introduction
HEX addresses the difficulty of whole-body humanoid manipulation by combining structured predictive dynamics with temporally grounded visual-language understanding. It reports strong real-world performance, especially in fast-reaction and long-horizon tasks.
- Motivation: Existing humanoid research largely separates locomotion from hand-centric manipulation, leaving simultaneous movement and object interaction underexplored.Whole-body control must maintain dynamic balance while coordinating high-dimensional, tightly coupled motions across multiple limbs.
- Limitations of Existing Approaches: Decomposed policies depend on manual interfaces and can accumulate errors, while VLA policies often lack explicit modeling of shared balance, posture, and body-part interactions.These limitations make tightly coupled behavior particularly difficult in fast-reaction and long-horizon scenarios.
- HEX: HEX introduces a humanoid-aligned universal state representation and predictive proprioceptive modeling to represent whole-body dynamics across heterogeneous embodiments.The framework uses structured body-part representations to support scalable modeling of cross-embodiment trajectories.
- HEX: HEX summarizes past visual-language context and adaptively fuses it with predicted proprioceptive dynamics, forming a review-and-forecast paradigm for coordinated action generation.This design provides temporal scene context and state foresight when current observations are insufficient.
- Results: HEX achieves higher task success rates and stronger generalization than strong VLA and imitation-learning baselines across real-world humanoid manipulation tasks.The improvements are particularly pronounced in fast-reaction and long-horizon scenarios, and the paper reports state-of-the-art overall performance.
2 Related Work
Prior work spans reinforcement and imitation learning, VLA-based humanoid control, and cross-embodiment transfer. HEX differentiates itself through compositional, humanoid-aligned proprioceptive modeling with dynamic expert routing.
- Whole-Body Control: Learning-based humanoid whole-body control has primarily advanced through reinforcement learning and imitation learning, including motion-tracking and motion-prior-based methods.Later work extends these paradigms toward real-world, contact-rich, and visually grounded control.
- Vision-Language-Action Models: Recent VLA systems extend visual-semantic reasoning from fixed-base manipulation toward humanoid whole-body control and generalist humanoid foundation models.Examples include Humanoid-VLA, GR00T N1, and Ψ0.
- Cross-Embodiment Learning: Cross-embodiment learning transfers shared behavior representations, aligned control spaces, or pretrained policies across agents with different morphologies.Prior directions include human-to-humanoid supervision and robot-to-humanoid or cross-humanoid transfer.
- HEX Architecture: HEX’s high-level policy combines visual-language encoding, predictive proprioceptive modeling, and adaptive action fusion.This architecture links current visual-language context with humanoid-aligned state dynamics for high-level action generation.
- HEX Positioning: HEX uses canonical body-part abstractions and a compositional Unified Proprioceptive Predictor to encode heterogeneous trajectories in a shared latent space.Reusable part-level encoders and dynamic Mixture-of-Experts routing support intra- and cross-embodiment data while modeling structured whole-body and temporal dependencies.
3 Method
HEX combines a hierarchical whole-body control architecture with temporal visual review, a cross-embodiment proprioceptive predictor, and adaptive action fusion. The framework uses structured body-part states and predicted dynamics to generate high-level actions consistent with downstream whole-body execution.
- Framework Overview: HEX couples a high-level VLA policy with a higher-frequency RL-based whole-body controller for dynamically feasible, balance-preserving motion.The high-level policy produces manipulation actions that also serve as intermediate commands for the low-level controller.
- VLM with History Query Feature Cache: A lightweight history cache preserves recent visual-language context as compact query features while the VLM encodes only the current frame.The visual history window is set to 2 in all experiments, avoiding repeated encoding of long image histories.
- Unified Proprioceptive Predictor: HEX maps heterogeneous proprioceptive observations into canonical body-part slots, using learned missing-part tokens to produce a shared latent state space.The representation includes arms, hands, legs, head, waist, and an auxiliary others slot, enabling structured inputs across embodiments.
- UPP with Morphology-based MoE: The Unified Proprioceptive Predictor uses morphology-aware MoE modules and a shared transformer to model token-specific variations, cross-part dependencies, and short-horizon dynamics.A learned top-k gate routes part-time tokens to specialized experts while a shared expert preserves a common transformation; the predictor outputs future proprioceptive latents.
- Action Expert with Adaptive Fusion: The Action Expert denoises action tokens under parallel visual-language and proprioceptive cross-attention, then adaptively injects the state branch through gated residual fusion.Self-attention and feed-forward refinement produce high-level arm and hand actions that remain consistent with downstream whole-body execution.
4 Experiments
HEX is evaluated on real-world humanoid whole-body manipulation across seen, long-horizon, generalization, ablation, routing, and latency studies. It consistently improves task success and generalization, with especially strong results in long-horizon and distribution-shift settings.
- Seen Scenarios: In seen scenarios, HEX achieves the best overall performance while balancing task success and motion quality, although ACT remains competitive on trajectory fitting and smooth reactive motions.The authors attribute HEX’s improvement to explicit future-state conditioning, which supplies dynamic cues beyond current visual observations.
- Long-Horizon Scenarios: HEX achieves the best performance across all stages of long-horizon box conveyance, including an approximately 15% advantage on final box placement.The result indicates stronger sustained execution and fewer cascading errors over long-horizon whole-body manipulation.
- Generalization Study: HEX reaches a 61.8% average success rate across eight generalization variants, exceeding π0.5 at 44.3%, GR00T N1.5 at 41.0%, and SwitchVLA at 22.4%.It performs best on nearly all variants and remains robust to interference, distractors, lighting changes, object-position changes, and dynamic scenes.
- Ablation Study: Pretraining mainly accelerates optimization and improves sample efficiency, with limited effect on final single-task success after convergence.At 5k steps, pretrained and non-pretrained models achieve 2/12 and 0/12 success, while final success reaches 11/12 versus 10/12.
- Ablation Study: Removing UPP causes the largest component-ablation degradation, while adding the history cache, UPP, and MoE progressively improves Pouring and Box Conveying success.Pouring rises from 4/12 without the components to 11/12 with full HEX; Box Conveying rises from 3/15 to 8/15.
- Other Analyses: 79.8% overall success at 73.34 ms latency makes HEX the strongest method under the reported practical inference budget.HEX outperforms all baselines in task success and is faster than π0.5.
5 Conclusion
HEX addresses a key limitation of VLA-style humanoid control by explicitly modeling interactions among body parts under shared balance and posture. Real-world experiments show superior performance over strong baselines, especially in fast-reaction and long-horizon settings.
- HEX combines a humanoid-aligned universal state representation, predictive whole-body proprioceptive dynamics, and adaptive fusion of visual-language context with future state evolution.These components support more coherent and stable whole-body action generation.
- Real-world humanoid manipulation experiments show HEX achieves superior performance over strong baselines, particularly in fast-reaction and long-horizon settings.These settings require coordinated whole-body behavior.
- HEX highlights the importance of explicitly modeling structured body-part interaction for general and scalable humanoid manipulation.