Source-linked AI summary
GigaBrain-WBC-0.5: A Behavior World Model for Robust Whole-Body Control with Environment Interaction
Ziyang Cheng, Tianshu Tang, Jinxin Lan, Xinze Chen, Yuhan Gong, Zhichao Liu, Changzhong Wu, Yahao Mao, Zongyan Deng, Mingxuan Ma, Huasen Xi, Yilong Liu, Yutong Wu, Xiaofeng Wang, Yang Wang, Yun Ye, Guan Huang, Xiaojie Jin, Zheng Zhu, Jiwen Lu
TL;DR
Existing humanoid whole-body trackers largely lack environment-aware behavior and robust responses to infeasible commands or falls. GigaBrain-WBC-0.5 models future actions, states, and behavior commands in one policy, achieving the highest success rate across four regimes and robust hardware interaction.
Problem
Existing whole-body trackers are primarily flat-ground policies that do not model how terrain and objects reshape feasible humanoid behavior.
Method
GigaBrain-WBC-0.5 trains a causal Transformer to predict its next action, proprioceptive state, and latent behavior-command distribution for environment-aware control.
Results
81.3% terrain-interaction success and 83.1% success under implausible commands accompany 99.3% fall-recovery success, the highest rates across four regimes.
Takeaways & Limitations
The unified policy remains usable under environmental interaction, infeasible commands, falls, missing supports, and physical disturbances.
Takeaways & Limitations
The filter is validated in simulation, requires per-platform recalibration, and detects unfamiliar commands rather than directly reasoning about physical risk.
Abstract
from arXiv · showhide
Whole-body motion tracking policies turn a humanoid into a robust control interface: the teleoperator---or an upstream model---only supplies a coarse movement intent, while the low-level policy keeps the robot balanced and physically feasible. Existing trackers deliver this interface only on flat ground: trained in empty scenes, they never learn how contact with terrain and objects reshapes their dynamics, and they attempt to teach the policy to balance under any command by continually enlarging the reference-motion corpus, which stops working once feasible behaviors become environment-dependent. We present GigaBrain-WBC-0.5, the first Behavior World Model (BWM) for humanoid whole-body control. Rather than a purely reactive tracker, we train a causal Transformer to jointly predict its next action, next state, and the distribution over its next latent behavior command, so the network that acts also models how the environment shapes what it can do next. An automatic terrain-annotation pipeline recovers full 3D contact geometry from retargeted motion, enabling terrain annotation at the scale of existing motion datasets. The predicted distribution is reused at deployment to detect implausible commands online and retract them onto learned behaviors, so the robot attempts tasks in a "best-effort" manner. The result is a unified policy that takes real-time command, interacts with environment, and stays robust to implausible commands, falls, and disturbances. GigaBrain-WBC-0.5 achieves the highest success rate across all four regimes among three large-scale tracker baselines: 81.3% on terrain interaction (4.3x the strongest baseline), 83.1% under implausible commands, and 99.3% fall recovery (16.8x the strongest baseline). Hardware trials show robust interaction under missing supports and disturbances; the Unitree G1 checkpoint transfers to the Maker L01 robot with simple fine-tuning.
1 Introduction
GigaBrain-WBC-0.5 addresses the limits of flat-ground whole-body trackers with a Behavior World Model that predicts actions, future state, and command feasibility while interacting with terrain and objects. Its design combines automatic 3D terrain annotation, online best-effort command handling, and integrated fall recovery.
- Motivation: Existing whole-body trackers provide online command interfaces but are predominantly trained and evaluated on empty, flat-ground scenes.They can support demonstrations and serve as execution layers beneath learned high-level policies, yet do not learn how terrain contacts or payloads change dynamics.
- Motivation: Environment-dependent feasibility makes enlarging the flat-ground reference-motion distribution an inadequate solution for command robustness.Feasible, human-like motions vary with environmental dynamics constraints, so the policy needs to recognize impossible commands rather than merely balance under increasingly broad commands.
- Robustness: An online filter uses the world model’s predictions to recognize implausible commands and produce best-effort motions instead of emergency stops or standing still.This preserves task continuity, including in situations such as staircases where stopping can be more dangerous than continuing.
- Robustness: Fallen initializations are trained into the tracker so recovery remains an integrated behavior rather than handing control to a specialist get-up controller.The goal is to keep the operator’s task driveable after a fall instead of interrupting it again.
- Approach: GigaBrain-WBC-0.5 is the first Behavior World Model for humanoid whole-body control, using a causal Transformer to predict the action, next proprioceptive state, and next latent behavior-command distribution.The same policy models its future behavior while remaining driven by live operator commands and interacting with terrain and objects.
- Approach: Automatic spatial terrain annotation recovers genuine 3D contact geometry directly from retargeted trajectories, enabling interaction training beyond flat-ground corpora.The output represents structures such as chairs, tables, boxes, and stair treads rather than only elevation maps.
2 Related Work
Related work has established physics-based whole-body motion tracking as the dominant humanoid-control paradigm, while scaling data and latent modeling. However, existing approaches remain limited by flat-ground training, narrow generalization, specialist terrain interfaces, and separate handling of infeasible commands or falls.
- Whole-body motion tracking: Physics-based motion imitation is the standard whole-body motion-tracking recipe because motion capture supplies dense per-frame supervision without per-task reward engineering.SONIC scales this paradigm to 100M frames, while other work applies latent modeling over motions, goals, and rewards at smaller data scales.
- Limitations: Existing trackers are typically trained on flat ground in empty scenes, learning expressive free-space motion that contacts only the floor.This leaves terrain and object interactions outside the behavior distribution.
- Limitations: Tracking fidelity can collapse as the motion library grows, exposing a generality barrier between motion coverage and reliable tracking.Prior work reports the same tension between expanding motion coverage and preserving tracking performance.
- Environment interaction: Terrain-focused methods span contact-aware planning, perceptive locomotion, motion matching, and scene-aware data generation, but generally specialize in one skill family.Their low-dimensional root-velocity or navigation interfaces specify base movement rather than whole-body arm, torso, and hand behavior.
- Robustness and recovery: Robustness work commonly widens training distributions or filters commands online, while standing-up and fall-protection capabilities are usually handled by dedicated skills.This separation determines whether the operator’s task continues after a fall.
3 Method
GigaBrain-WBC-0.5 combines a causal Transformer policy with next-state and next-command prediction, using terrain reconstructed automatically from retargeted motion to model environment-dependent behavior. At deployment, predicted command distributions support closed-form retraction of implausible commands, while the learned checkpoint transfers to a second humanoid through fine-tuning.
- Policy inputs and architecture: The policy maps proprioception, the previous action, and a 10-frame reference window to 29-dimensional PD joint targets at 50 Hz.The Unitree G1 controller has 29 actuated degrees of freedom and receives a 67-dimensional proprioceptive observation.
- Policy inputs and architecture: A 6-layer causal Transformer processes 160-dimensional per-frame inputs over a masked 32-frame context with single-step KV-cached rollout.The network uses 4 heads and rotary position embeddings, with masking enforcing causality, context bounds, and episode boundaries.
- Terrain annotation: The terrain pipeline converts retargeted motion into simulator-ready geometry by detecting contacts, filtering collision-inconsistent points, clustering planar supports, and fitting oriented primitives.It enables terrain-paired training data to be built at the scale of existing motion datasets rather than scene-capture sessions.
- Behavior world model: The network predicts its action, next state, and a four-component Gaussian distribution over the next 64-dimensional latent behavior command, thereby modeling contact dynamics and situation-dependent reachable behaviors.Next-state prediction represents dynamics such as stepping onto a box or carrying a load, while next-command prediction represents which behaviors the current situation admits.
- Command retraction: At deployment, the predicted command mixture defines in-distribution regions, and implausible commands are radially retracted to a trusted mode boundary so the robot continues attempting the task.The closed-form rescaling avoids per-frame Lagrange-multiplier optimization and preserves the direction from the trusted mode toward the operator’s command.
- Embodiment transfer: Fine-tuning the G1 checkpoint on a Maker L01 recovers whole-body tracking quickly, whereas training the same architecture from scratch on L01 converges slowly.The transfer recipe reuses the architecture and retargeted corpus on the second embodiment.
4 Experiments
Experiments evaluate GigaBrain-WBC-0.5 across flat-ground, terrain, implausible-command, and fall-recovery regimes against three large-scale trackers, alongside annotation audits and hardware trials. The policy preserves flat-ground performance while substantially improving terrain interaction, command robustness, and recovery.
- Benchmarks and metrics: Evaluation uses 136-clip Standard and OOD sets, a 150-clip Terrain set, and fallen-pose initialization for the 136-clip Fall set.Standard measures flat-ground tracking; OOD contains physically implausible references, Terrain contains terrain-interaction motions, and Fall reuses Standard clips from fallen poses.
- Standard: On Standard, GigaBrain-WBC-0.5 achieves 76.6 mm MPKPE and 96.3% SR, leading all four policies in flat-ground accuracy and reliability.SONIC reaches 82.3 mm MPKPE and 94.1% SR; root linear velocity is the one metric not led, at 211.1 versus HoloMotion-1’s 121.3 mm/s.
- Terrain: 81.3% SR on Terrain at 93.3 mm MPKPE outperforms the strongest baseline by 4.3× in survival and reduces tracking error 3.0×.The three baselines achieve only 14–19% SR with 283–331 mm MPKPE; GigaBrain-WBC-0.5’s MPKPE rises 22% from flat ground to terrain.
- OOD and fall recovery: On OOD, GigaBrain-WBC-0.5 survives 83.1% of physically impossible references versus 50.0–70.6% for baselines, while achieving 158.0 mm versus 208.0 mm for the best baseline.The filter retracts implausible commands toward the nearest behavior the policy can execute rather than simply discarding them.
- Terrain annotation: The terrain-annotation audit finds 92% overall reliability, with boxes and platforms at 98%, stair treads at 92%, seats at 94%, and other primitives at 84%.The audit sampled 50 annotated Bones-Seed motions per category and used human judgments of reconstructed primitive consistency.
- Hardware and deployment: Hardware trials demonstrate best-effort responses to missing supports, standing and resuming after falls, and filtered command execution that avoids falls while preserving commanded turns.The same live-command tracker coordinates locomotion with object interaction and supports demanding motions and household or industrial behaviors after fine-tuning.
5 Conclusion
GigaBrain-WBC-0.5 combines environment interaction with online filtering of implausible commands for humanoid whole-body control. It achieves strong simulated and hardware performance, while retaining calibration, physical-risk, and terrain-coverage limitations.
- Contributions: The behavior world model addresses environment interaction and commands that the environment does not support, using recovered contact geometry and a stateless single-step projection.The projection converts the model’s predicted command distribution into an online out-of-distribution filter governed by one scalar radius.
- Results: 81.3% on terrain interaction, 4.3× the strongest baseline, and 83.1% under physically implausible out-of-distribution commands establish the policy’s simulated performance.It achieves the highest success rate in all four regimes among three large-scale tracker baselines.
- Results: 99.3% recovery from fallen initializations exceeds the single-digit baseline rates reported in sim-to-sim evaluation.Hardware trials also show agile-motion tracking, real-environment interaction, and robustness to missing supports.
- Limitations: The filter is validated in simulation, requires per-checkpoint and per-platform radius recalibration, and detects training-distribution novelty rather than directly reasoning about physical risk.Consequently, interception of infeasible commands is high-probability rather than certain; contact-derived terrain annotation recovers only geometry actually touched by motion.