Source-linked AI summary
FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructions
Peng Li, Zihan Zhuang, Yangfan Gao, Yi Dong, Sixian Li, Changhao Jiang, Shihan Dou, Zhiheng Xi, Enyu Zhou, Jixuan Huang, Hui Li, Jingjing Gong, Xingjun Ma, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Xipeng Qiu
TL;DR
Humanoid robots need more versatile natural-language whole-body control than hardcoded or task-specific motion systems provide, but paired robot data and stable physical execution remain challenging. FRoM-W1 combines H-GPT, which generates human motions from language, with H-ACT, which retargets and controls them on robots; reinforcement-learning fine-tuning improves tracking accuracy and task success, and the framework is open-sourced.
Problem
Humanoid whole-body motions are commonly hardcoded or task-specific, while paired language-robot-motion data are scarce and direct physical execution is vulnerable to instability.
Method
FRoM-W1 trains H-GPT on human motion data for language-conditioned motion generation and uses H-ACT for robot-specific retargeting, reinforcement-learning control, and deployment.
Results
Reinforcement-learning fine-tuning improved motion-tracking success rate and Mean Per-joint Joint Position Error (MPJPE) by 15%, while FRoM-W1 achieved superior HumanML3D-X motion-generation performance.
Takeaways & Limitations
FRoM-W1 provides an open-source framework for language-guided whole-body control across Unitree H1 and G1 humanoid robots.
Takeaways & Limitations
The framework remains constrained by the need for larger, higher-quality human motion datasets and by unstable real-world deployment caused by the sim2real gap.
Abstract
from arXiv · showhide
Humanoid robots are capable of performing various actions such as greeting, dancing and even backflipping. However, these motions are often hard-coded or specifically trained, which limits their versatility. In this work, we present FRoM-W1, an open-source framework designed to achieve general humanoid whole-body motion control using natural language. To universally understand natural language and generate corresponding motions, as well as enable various humanoid robots to stably execute these motions in the physical world under gravity, FRoM-W1 operates in two stages: (a) H-GPT: utilizing massive human data, a large-scale language-driven human whole-body motion generation model is trained to generate diverse natural behaviors. We further leverage the Chain-of-Thought technique to improve the model's generalization in instruction understanding. (b) H-ACT: After retargeting generated human whole-body motions into robot-specific actions, a motion controller that is pretrained and further fine-tuned through reinforcement learning in physical simulation enables humanoid robots to accurately and stably perform corresponding actions. It is then deployed on real robots via a modular simulation-to-reality module. We extensively evaluate FRoM-W1 on Unitree H1 and G1 robots. Results demonstrate superior performance on the HumanML3D-X benchmark for human whole-body motion generation, and our introduced reinforcement learning fine-tuning consistently improves both motion tracking accuracy and task success rates of these humanoid robots. We open-source the entire FRoM-W1 framework and hope it will advance the development of humanoid intelligence.
1. Introduction
FRoM-W1 addresses natural-language whole-body control for humanoid robots by combining human-motion language generation with robot-specific stable execution. Its two-stage framework uses H-GPT for motion synthesis and H-ACT for retargeting, reinforcement-learning control, and physical deployment.
- Motivation: Natural-language control remains limited because existing humanoid motions are usually hardcoded or task-specific, while paired robot language-motion datasets are scarce.Direct execution on physical bipedal robots also risks instability and falls under gravity and dynamic uncertainties.
- H-GPT: FRoM-W1 leverages large-scale human motion data to train H-GPT, which generates whole-body human motions from natural-language instructions using Chain-of-Thought decomposition.CoT converts abstract or complex instructions into body-level motion primitives with explicit temporal structure.
- H-ACT: H-ACT retargets generated SMPL-X human motions to specific robot morphologies and uses reinforcement-learning pretraining and fine-tuning for stable whole-body execution.The controller is designed to support physical deployment across humanoid platforms.
- Evaluation: FRoM-W1 was evaluated on Unitree H1 and G1 robots and on the HumanML3D-X human whole-body motion-generation benchmark.The framework also includes the 𝛿HumanML3D-X generalization benchmark and evaluation of CoT and reinforcement-learning fine-tuning.
- Results: 15% improvement in Mean Per-joint Joint Position Error (MPJPE) accompanied reinforcement-learning fine-tuning, alongside improved motion-tracking success rate.The framework also supports deployment of recent motion-mimicking controllers through its sim2real framework.
- Contributions: The authors introduce H-GPT and H-ACT and open-source the complete framework, including training code, checkpoints, benchmarks, and deployment software.The stated goal is to facilitate further research in language-guided whole-body humanoid control.
2. FRoM-W1
FRoM-W1 maps open-ended language instructions to stable humanoid whole-body actions through two stages: H-GPT generates human motion, and H-ACT retargets and executes it on robots. The framework uses language-enriched human motion data, motion tokenization, retargeting, reinforcement learning, and modular deployment to bridge semantic understanding and physical control.
- Framework overview: FRoM-W1 decomposes language-driven humanoid control into H-GPT, which generates whole-body human motion, and H-ACT, which converts it into stable robot actions.The complete pipeline is Π(I) = A_H-ACT(G_H-GPT(I)).
- H-ACT: H-ACT retargets human motion to a robot morphology, then uses a whole-body control policy to track the retargeted motion while maintaining dynamic stability.Retargeting produces robot joint trajectories and root information that serve as downstream reference motion.
- H-GPT: H-GPT enriches paired human motion data with Chain-of-Thought sequences that connect abstract instructions to temporally organized motion primitives.The CoT sequence serves as an intermediate representation between high-level language and low-level human motion.
- H-GPT: A VQ-VAE tokenizer converts whole-body human motion sequences into discrete tokens aligned with linguistic tokens, allowing an LLM-based generator to model motion autoregressively.The tokenizer encodes, quantizes, and decodes motion; H-GPT predicts CoT and motion tokens conditioned on an instruction.
- H-ACT: The control policy is trained in simulation with reinforcement learning and fine-tuned for target motions, while a modular deployment layer supports stable physical execution and policy integration.The modular design supports sim-to-real transfer, runtime policy switching, and composition across robotic platforms.
3. Experiments
The experiments evaluate H-GPT for language-driven whole-body human motion generation and H-ACT for stable humanoid control in simulation and real-world deployment. Results show improved motion-generation quality, benefits from CoT and reinforcement-learning fine-tuning, and successful deployment across H1 and G1 robots.
- Benchmarks: HumanML3D-X extends HumanML3D with hand-inclusive SMPL-X motions, while δHumanML3D-X adds transformed evaluation settings for testing generalization.Both benchmarks use language–motion pairs, and δHumanML3D-X evaluates models trained on HumanML3D-X.
- Human motion generation: H-GPT reduced FID from 0.677 for T2M-GPT to 0.229 without CoT and 0.255 with CoT on HumanML3D-X.FID is the primary metric; the no-CoT model also achieved the best DIV score of 9.674.
- Data scaling: Motion-X training achieved decent performance but remained below direct HumanML3D-X training, while incorporating CoT improved overall performance.The authors attribute the gap partly to distribution and quality differences between video-estimated, VLM-labeled data and precise motion-capture data.
- Human motion generation: CoT enhancement improved H-GPT and H-GPT++ generation results on manually evaluated complex and abstract instructions, although neither model consistently outperformed the other.Because these instructions lack ground-truth motions, evaluation used comparative human observation rather than separate absolute scores.
- RL pre-training: Filtered training data improved MPJPE, velocity, and acceleration metrics for both robots, while G1 achieved higher success rates than H1 at roughly 80%–90% versus 40%–50%.The results associate G1’s stronger performance with its structural design and identify reference-pose stability and motion-change rate as influential factors.
- RL fine-tuning: Reinforcement-learning fine-tuning steadily improved MPJPE and success rate for H1 and G1, stabilizing some motions that were previously less reliable.The evaluation used 30 randomly sampled AMASS motions and compared fine-tuning with training from scratch and pre-training alone.
- RL fine-tuning and deployment: Fine-tuning produced significant improvement around 500 steps, and real-world demonstrations covered abstract instructions, hand movements, locomotion, and multiple control strategies.The deployment results demonstrate the framework’s intended combination of language understanding, whole-body motion generation, and robot execution.
4. Related Work
FRoM-W1 builds on text-conditioned human motion generation and humanoid control research by adding hand-aware, larger-scale data, language generalization, and cross-policy deployment. It positions whole-body language control on full-size humanoids as a recent task requiring integrated generation and stable execution.
- Text-conditioned motion generation: Text-conditioned human motion generation supports applications including VR, gaming, and robotics, but HumanML3D is limited by scale and lacks hand modeling.MotionMillion expands data scale, while Motion-X uses SMPL-X with hand modeling.
- Cross-policy deployment: H-ACT supports stable deployment of different control strategies, including the TWIST policy and other motion-mimicking controllers.The framework also reports real-world failure cases involving extreme squatting and fast forward-moving target motions.
- Humanoid control: Prior humanoid work explored task-specific reinforcement learning, teleoperation, and whole-body control on platforms such as Unitree H1 and G1.Examples include locomotion, backflips, upper- and lower-body control, and teleoperation-based manipulation.
- Language-conditioned humanoid control: Language-conditioned whole-body control for full-size humanoids remains a very new task, with Robot-MDM and Harmon representing earlier approaches.FRoM-W1 extends this direction with hand modeling, new benchmarks, more data, optimized reinforcement learning, and universal sim2real deployment.
5. Conclusion
FRoM-W1 unifies language understanding, motion generation, and humanoid whole-body control through H-GPT and H-ACT. The paper also identifies larger, higher-quality motion data and more robust motion tracking as important remaining needs.
- Framework: FRoM-W1 combines H-GPT language-conditioned motion synthesis with H-ACT cross-platform, policy-agnostic execution for physically stable humanoid behavior.The framework is open-source and targets multiple embodiments.
- Remaining challenges: The authors identify a need for larger and higher-quality human motion datasets to improve textual alignment, real-world suitability, and tracking of noisy generated motions.These shortcomings are presented as areas for further improvement across the field.
A.1. Training Data Construction for H-GPT
The H-GPT training data combines language, rendered motion videos, and hand-inclusive whole-body pose representations. GPT-4o-generated first-person plans provide fine-grained Chain-of-Thought supervision, while the pose format records root, joint, and contact features.
- CoT data generation: GPT-4o generates fine-grained Chain-of-Thought data from HumanML3D-X language instructions and rendered motion videos.The resulting prompt asks for a concise first-person plan describing how to perform the observed motion.
- CoT data generation: The training prompt requires a 1-to-2-sentence first-person plan that accurately describes the motion without uncertainty alternatives.Examples cover waving, elephant-like movement, and recovering after a trip.
- Motion representation: The pose representation includes root angular and linear velocities, root height, local joint positions, velocities, and rotations in root space.Joint count is denoted by j, and the representation includes hand-inclusive whole-body poses.
- Motion representation: Binary foot-contact features are obtained by thresholding heel and toe joint velocities.These features emphasize when the feet contact the ground.
A.2. More Chain-of-Thought Evaluation Cases
The evaluation cases span complex sequential actions and abstract instructions. Examples require the model to combine locomotion, posture changes, object-like interactions, expressive gestures, and pantomime.
- Complex instructions: Complex instructions combine multiple ordered actions such as running-direction changes, jumping, kneeling, pouring, drinking, and stepping backward.The examples test sequential transitions rather than isolated motions.
- Abstract instructions: Abstract instructions describe expressive or imagined behaviors, including pulling a heavy rope, showing dejection, beckoning, pinching a delicate cube, and pressing a wall.These cases emphasize semantic interpretation beyond literal locomotion descriptions.
B.1. Hardware and Deployment Setup
FRoM-W1 uses a workstation-based inference setup with software-only deployment on Unitree H1 and G1 humanoids. The system separates H-GPT inference from onboard H-ACT execution and supports interchangeable localization options on G1.
- Deployment Components: The real-world deployment combines an inference workstation with a full-size humanoid robot equipped with hands for H-ACT motion execution.The workstation performs model execution while the robot handles motion execution through H-ACT.
- Inference Server: FRoM-W1 runs H-GPT inference on a high-performance workstation that wirelessly communicates with robots’ onboard compute units.The workstation uses dual NVIDIA RTX 5090 GPUs, an AMD Threadripper 7970X CPU, and 128GB RAM.
- Robot Platforms: The framework is deployed on Unitree H1 and G1 platforms without hardware modification, relying on software integration with existing control and sensing infrastructure.This setup is used to demonstrate the approach’s generality and practicality across two humanoid platforms.
- Robot Platforms: The H1 uses onboard PC2 computation, while the G1 runs its control policy on a built-in Jetson Orin NX 16GB unit without external compute.The H1 has 19 body degrees of freedom plus a 2-DOF wrist; the G1 has 29 degrees of freedom and three-fingered hands.
- Localization: G1 global localization can use either the ZED Mini configuration used on H1 or the odometry service provided by the Unitree SDK.These are presented as interchangeable localization options for the G1 deployment.
B.2. RoboJuDo Deployment Framework
RoboJuDo provides a unified deployment layer that connects controllers, robot environments, and heterogeneous whole-body policies through standardized interfaces. Its modular design supports policy reuse, runtime composition, simulation-to-real parity, and adaptation across robot embodiments.
- Architecture: RoboJuDo unifies the Controller, Environment, and Policy modules behind consistent interfaces for motion commands, robot state, and actions.The abstraction enables modular composition across simulation and real-robot platforms.
- Controller: The Controller standardizes inputs such as keys, motion streams, and external triggers into consistent motion commands for downstream policies.It also supports hierarchical triggering and real-time motion adaptation.
- Environment: The Environment abstracts robot communication, initialization alignment, and forward kinematics while maintaining functional parity between simulation and real-robot execution.This design allows policies to transfer directly with minimal engineering overhead.
- Policy: The Policy interface standardizes data flow across diverse control strategies, while a Policy Manager coordinates multi-policy adaptation and robot-specific configuration.DoFConfig and PolicyWrapper remap policy actions and observations between internal conventions and target robot embodiments.
- Policy Composition: RoboJuDo supports runtime policy switching, output blending, and hierarchical behavior composition without changing the Environment or Controller modules.Policies share the same (s_t, m_t) ↦ a_t+1 abstraction, enabling benchmarking and composition of skills.
- System Integration: The resulting configuration-driven pipeline connects heterogeneous inputs, environments, controllers, policies, and robot embodiments with minimal engineering effort.The framework is intended to support diverse control paradigms and execution platforms through plug-and-play integration.
B.3. Sim2Real Transfer Details
FRoM-W1 addresses sim-to-real transfer by identifying low-inertia joint oscillations as a training challenge and modifying training or deployment control accordingly. The recommended strategy improves learned-policy stability and robustness.
- Transferability: These simulation-training techniques are presented as practical measures for improving policy performance and transferability to physical humanoid platforms.The discussion frames low-inertia-joint handling as part of broader sim-to-real training practice.
- Low-Inertia Joints: Low-inertia wrist and head joints can oscillate at high frequency in simulation, destabilizing training and widening the simulation-to-reality gap.The issue is linked to numerical limitations of the physics engine’s rigid-body solver.
- Mitigation: The recommended mitigation is to remove or freeze selected low-inertia degrees of freedom during training and reintroduce them at deployment.Separate low-level controllers or scripted behaviors can control the excluded wrist and head joints in the real world.