PaperlayerView exact revision on arXiv

Source-linked AI summary

BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning

Yitang Li, Zhengyi Luo, Tonghe Zhang, Cunxi Dai, Anssi Kanervisto, Andrea Tirinzoni, Haoyang Weng, Kris Kitani, Mateusz Guzek, Ahmed Touati, Alessandro Lazaric, Matteo Pirotta, Guanya Shi

arXiv:2511.04131v1cs.RO

TL;DR

Humanoid control approaches are often task-specific, difficult to adapt, and lack a unified interface, while behavior cloning is constrained by the scarcity of actuator-level labels and teleoperation data. BFM-Zero learns a shared latent space for prompting a whole-body policy, and demonstrates zero-shot skills and few-shot adaptation on a real Unitree G1.

  • Problem

    Humanoid whole-body control lacks a unified, adaptable policy for diverse tasks, and behavior cloning is limited by scarce actuator-level labels and large teleoperation datasets.

    Source: §1 · p. 3

  • Method

    BFM-Zero uses off-policy unsupervised RL with motion-capture regularization to learn a shared latent representation for reward, goal, and motion prompts, with domain randomization and asymmetric history-dependent training.

    Source: §1 · p. 3, §2 · p. 4, §2.2 · p. 6

  • Results

    BFM-Zero supports zero-shot reward optimization, pose reaching, and motion tracking on a real Unitree G1, with effective few-episode improvement when zero-shot performance is unsatisfactory.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The work demonstrates off-policy unsupervised RL as a viable route to a promptable behavioral foundation model for whole-body control of a real humanoid.

    Source: §4 · p. 12

    Behavior scope and performance are connected to the motions used in training; the paper identifies dataset scaling and its relationship to learned behavior as an open research direction.

    Source: §4 · p. 12

Abstract

from arXiv · show

Building Behavioral Foundation Models (BFMs) for humanoid robots has the potential to unify diverse control tasks under a single, promptable generalist policy. However, existing approaches are either exclusively deployed on simulated humanoid characters, or specialized to specific tasks such as tracking. We propose BFM-Zero, a framework that learns an effective shared latent representation that embeds motions, goals, and rewards into a common space, enabling a single policy to be prompted for multiple downstream tasks without retraining. This well-structured latent space in BFM-Zero enables versatile and robust whole-body skills on a Unitree G1 humanoid in the real world, via diverse inference methods, including zero-shot motion tracking, goal reaching, and reward optimization, and few-shot optimization-based adaptation. Unlike prior on-policy reinforcement learning (RL) frameworks, BFM-Zero builds upon recent advancements in unsupervised RL and Forward-Backward (FB) models, which offer an objective-centric, explainable, and smooth latent representation of whole-body motions. We further extend BFM-Zero with critical reward shaping, domain randomization, and history-dependent asymmetric learning to bridge the sim-to-real gap. Those key design choices are quantitatively ablated in simulation. A first-of-its-kind model, BFM-Zero establishes a step toward scalable, promptable behavioral foundation models for whole-body humanoid control.

1 Introduction

Humanoid whole-body control must support a wide range of tasks in dynamic, unstructured environments, but existing approaches leave policies task-specific and difficult to adapt or direct. BFM-Zero investigates unsupervised RL as a route to a promptable whole-body controller that can handle multiple tasks without retraining.

1 Introduction

  • Humanoid whole-body control is a fundamental, challenging step toward robots working safely in human environments, where they must perform diverse tasks in dynamic, unstructured settings.The introduction frames this as relevant to applications ranging from manufacturing and logistics to healthcare and personal assistance.Source: §1 · p. 2
  • Existing sim-to-real RL policies are often task-specific and non-adaptive, while whole-body behavior cloning lacks readily available actuator-level labels and large teleoperation datasets.These approaches also lack a unified, explainable interface for specifying goals and composing behaviors.Source: §1 · pp. 2–3
  • BFM-Zero uses online off-policy unsupervised RL, motion-capture regularization, domain randomization, and asymmetric history-dependent training to support zero-shot tasks on a real Unitree G1.The framework supports reward optimization, pose reaching, and motion tracking without retraining, with few-episode adaptation when zero-shot performance is insufficient.Source: §1 · p. 3

2 BFM-Zero for Humanoid Whole-body Control

BFM-Zero trains a shared latent representation and promptable policy for humanoid whole-body control, then uses latent prompts for zero-shot tasks or adapts them through sampling-based optimization. Its training combines forward-backward representations with motion regularization and design choices aimed at sim-to-real transfer and physical constraints.

2 BFM-Zero for Humanoid Whole-body Control

  • The framework pre-trains a shared task latent space and promptable policy from unlabeled motion trajectories, then embeds downstream tasks for zero-shot execution or latent-space adaptation.Its setup models a 29-DoF humanoid with proprioceptive observations and histories, alongside privileged simulation state.Source: §2 · pp. 3–4

2.1 Unsupervised RL with Forward-Backward Representations

  • Forward-backward representations learn mappings that approximate long-term policy dynamics, yielding task features and a latent space of linear rewards that each conditioned policy optimizes.FB-CPR trains online and off-policy, regularizing learned policies toward demonstrated motion-capture behaviors.Source: §2.1 · pp. 4–5

2.2 BFM-Zero Pre-training for Humanoid Control

  • Off-policy actor-critic pre-training fits history-based forward and privileged backward maps, with auxiliary critics for safety constraints and a discriminator that regularizes human-like style and exploration.The design includes asymmetric history-based learning, domain randomization of physical parameters and sensor conditions, and training across thousands of environments.Source: §2.2 · pp. 5–6
  • At zero-shot inference, reward prompts are estimated from sampled replay-buffer rewards, goals use the backward-map embedding of the target state, and tracking uses a sequence of latent prompts.The Figure 3 caption specifies task-averaged reward return over 500-step episodes and joint-position error for tracking and goal-reaching evaluation.Source: §2.2 · p. 7, Figure 3 · p. 7
  • Few-shot adaptation optimizes latent prompts through simulator interaction, using CEM for single poses and sampling-based trajectory optimization for motion sequences.The trajectory procedure warm-starts from tracked motion and is reported to stabilize challenging segments and reduce tracking error while retaining the discriminator's human-like prior without network fine-tuning.Source: §2.2 · p. 7

3 Experiments

Experiments assess BFM-Zero in simulation and on humanoid hardware, then test prompt-level adaptation and the structure of its latent skill space. The evaluations cover zero-shot task performance, robustness, and interpolation without retraining.

3 Experiments

  • Experiments train the model in IsaacLab at 200 Hz with 50 Hz control, using LAFAN1 motions retargeted to Unitree G1, and also demonstrate generality on Booster T1.The LAFAN1 dataset contains 40 several-minute-long motions.Source: §3 · p. 7

3.1 Zero-shot Validation in Simulation

  • Sim DR BFM-Zero is 2.47%, 25.86%, and 10.65% worse than BFM-Zero-priv across tracking, reward, and pose-reaching tasks; Mujoco performance differences across variations are all below 7%.The privileged model is trained and tested with nominal dynamics (No DR) and is not deployable on the real robot. Reward tasks suffer a larger performance drop, in part because sparse rewards are less forgiving of suboptimal behavior and amplify model error.Source: §3.1 · p. 8
  • BFM-Zero successfully generalizes to tracking and pose-reaching tasks on 175 AMASS motions and 10 poses absent from training, though direct comparison with LAFAN1 is cautioned against.For move-ego-0.0, some inference instances perform poorly; the authors conjecture that increased training-data randomness makes inference with a small subsampled dataset more brittle.Source: §3.1 · p. 8

3.2 Zero-shot Validation on the Real Robot

  • On the Unitree G1, zero-shot tracking covers dynamic dances and other motions, with recovery and continued tracking after instability or falls, including from lower-quality video-derived motion.Figure 4 shows dancing, frequently turning while walking, and recovery while continuing to track.Source: Figure 4 · p. 9, §3.2 · p. 9
  • For goal reaching, the robot approaches sampled target poses, including infeasible ones, and makes smooth transitions between discontinuous targets or from arbitrary poses to T-pose without explicit interpolation.Figure 5 marks target poses in blue/yellow, the robot pose in black, and transitions in gray.Source: Figure 5 · p. 9, §3.2 · p. 9
  • Real-world reward prompts produce locomotion, arm, and pelvis-height behaviors; linear reward combinations yield composite skills, while one reward can produce diverse behaviors.Figure 6 illustrates sitting, crouching, low movement, varied behaviors from an arm-raising reward, and combinations with movement rewards; the policy also withstands kicks, pushes, and being dragged down, recovering naturally.Source: Figure 6 · p. 10, §3.2 · p. 10, Figure 7 · p. 10

3.3 Efficient Adaptation for BFM-Zero

  • Under an additional payload, prompt-level optimization alone maintains single-leg balance for over 15s, while unadapted motion destabilizes and causes an environmental collision within 5s, without fine-tuning model parameters.For a leaping trajectory under altered ground friction, optimization reduces tracking error by ∼29.1%.Source: §3.3 · p. 11

3.4 The Latent Space Structure of BFM-Zero

  • Latent vectors cluster by motion style, and spherical interpolation produces semantically meaningful intermediate skills on simulated and real humanoids without additional training.The latent representation is described as interpretable and structured, facilitating skill interpolation without retraining.Source: §3.4 · pp. 11–12

4 Discussion

BFM-Zero demonstrates off-policy unsupervised RL as a viable approach to real humanoid whole-body control, while its behavior scope remains tied to training motions. Related work includes single-task sim-to-real policies and two-stage methods that distill motion-tracking policies into multi-task controllers.

4 Discussion

  • BFM-Zero demonstrates that off-policy unsupervised RL can train a behavioral foundation model for whole-body control of a real humanoid robot, but behavior scope remains connected to training motions.The discussion identifies improved online adaptation and better understanding of fast adaptation and fine-tuning as needs for expressing more complex movements and broadening practical applicability.Source: §4 · p. 12
  • The discussion identifies dataset size, simulated data, architecture, and model performance as factors to study when developing scaling laws for BFM-Zero.It also calls for stronger online adaptation and further study of fast adaptation and fine-tuning.Source: §4 · p. 12

A Related Work

  • Earlier humanoid-control work largely targeted simulated robots or single tasks, while multi-task sim-to-real methods commonly distilled motion-tracking policies into one controller.These approaches use two training stages and depend on motion quality; BFM-Zero instead directly learns a promptable multi-task policy with off-policy RL.Source: A Related Work · p. 19

B.1 Training Hyperparameter Settings

Training uses fixed-length episodes and motion-data sampling with tracking-based prioritization. The setup scales off-policy learning across parallel environments, with training settings reported in Table 1.

B.1 Training Hyperparameter Settings

  • Episodes last T = 500 steps, and starting states mix randomly generated falls with states from the motion dataset.The 40 LAFAN1 motions are split into 10-second chunks, prioritized by tracking ability, with states sampled uniformly within each motion.Source: §B.1 · p. 19
  • Training scales unsupervised off-policy RL with Nenv parallel environments and scales buffer size according to Nbuffer × Nenv × T.The setup also specifies batch size, update-to-data ratio, and total training steps; their values are referenced in Table 1.Source: §B.1 · p. 20
  • Table 1 reports the training settings.The supplied table caption does not state individual parameter values.Source: Table 1 · p. 20

B.2 Network Architectures

The actor and critics use residual networks, while the discriminator and backward map use MLPs. The pre-training algorithm combines online interaction with expert-motion data and updates its representation and discriminator.

B.2 Network Architectures

  • The actor and critics use residual blocks with residual connections, layer normalization, and Mish activations; the discriminator and backward map use standard MLPs with ReLU.The critics use a two-network ensemble; Figure 10 presents the network architectures, and Table 2 lists real-test architecture parameters.Source: §B.2 · p. 21, Figure 10 · p. 21, Table 2 · p. 21

B.3 BFM-Zero Algorithm Details

  • BFM-Zero pre-training initializes online and expert buffers, samples latent prompts, and executes the policy in simulated environments.The algorithm sketch describes online interaction followed by updates, and notes that it omits parallel networks for clarity.Source: §B.3 · p. 22
  • The update phase samples transitions and motion sequences, updates the discriminator and forward-backward representation, then computes discriminator rewards.The representation update targets F(s, a; z)ᵀB(s′) ≈ Mπz(ds′|s, a).Source: §B.3 · p. 22

B.4 Training Environments

The training environment uses domain randomization, additive observation noise, and regularization rewards to facilitate sim-to-real transfer. Figure 11 provides further environment details.

B.4 Training Environments

  • Domain randomization, additive observation noise, and regularization rewards are incorporated to facilitate sim-to-real transfer.Source: §B.4 · p. 22
  • Figure 11 presents details of the training environment.Source: Figure 11 · p. 23

C Tasks and Metrics

The evaluation measures goal-pose accuracy, motion tracking, and reward maximization across diverse whole-body tasks. Goal and tracking evaluations use joint-position error, while reward evaluation covers six categories of normalized rewards.

Goal-based evaluation

  • Goal evaluation averages joint error across 21 stable LAFAN1 poses and 10 AMASS poses, with episodes fixed at 500 steps.The selected LAFAN1 poses are shown in Figure 12; joint positions are 29-dimensional.Source: C Tasks and Metrics · p. 23, Figure 12 · p. 23

Tracking evaluation

Reward evaluation

  • Reward evaluation defines six categories of next-state rewards normalized to [0,1], and reports cumulative return over 500-step episodes starting from the default pose.The evaluation covers 24 rewards in total.Source: C Tasks and Metrics · p. 24
  • The reward tasks span standing, forward, lateral and backward locomotion, vertical-axis rotation, ground poses, arm raises, and combinations of locomotion, rotation, and arm movements.Reward performance is higher-is-better in the test-set comparison.Source: C Tasks and Metrics · p. 24

D.1 Data Size and Model Size

The ablations vary motion-dataset composition and model architecture to examine tracking and reward inference. Greater model capacity generally helps tracking, while reward inference shows a milder capacity trend and sensitivity to the training data source.

D.1 Data Size and Model Size

  • Tracking improves with model capacity for almost all training mocap datasets, but saturates early for LAFAN1, plausibly because held-out CMU and BMLHandball motions are closer to its AMASS-subset data.The tracking test set excludes CMU and BMLHandball motions used for training.Source: §D.1 · p. 25
  • Residual architectures outperform simple MLPs and can scale to larger sizes, whereas scaling MLPs to larger architectures made training unstable.The ablation includes feed-forward and residual architectures with varying numbers of blocks.Source: §D.1 · pp. 25–26
  • Reward inference improves mildly with model size, performs worse when trained only on AMASS subsets, and drops more with the training buffer than with the motion set.The authors suggest domain-randomized buffer samples may explain the larger drop and identify dataset selection as future work.Source: §D.1 · p. 26

D.2 Application of BFM-Zero on Booster T1

The authors evaluate BFM-Zero on the Booster T1 after retargeting LAFAN1 motions and using the G1 hyperparameters. The policy exhibits natural walking and expressive dancing on T1.

D.2 Application of BFM-Zero on Booster T1

  • On Booster T1, BFM-Zero shows strong generalization, performing natural walking and expressive dancing after LAFAN1 motions are retargeted and the G1 hyperparameters are reused.The application is shown in Figure 15.Source: §D.2 · p. 26, Figure 15 · p. 27

Related papers

Loading 2511.04131v1…