PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models

Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, Matteo Pirotta

arXiv:2504.11054v1cs.LG

TL;DR

Existing zero-shot RL and imitation methods face limits in downstream adaptation, dataset coverage, or generalization beyond demonstrated behaviors. The paper introduces FB-CPR, which embeds unlabeled trajectories into the FB latent policy space and regularizes online RL with a latent-conditional discriminator. In humanoid control, the resulting behavioral foundation model supports zero-shot motion tracking, goal reaching, and reward optimization, with human-like behaviors and strong comparative performance.

  • Problem

    Existing unsupervised RL may require downstream task training or broad datasets, while imitation models can struggle to generalize beyond demonstrated behaviors.

  • Method

    FB-CPR embeds unlabeled trajectories into the FB latent policy space and regularizes policy learning with conditional distribution matching.

  • Results

    The humanoid behavioral foundation model solves motion tracking, goal reaching, and reward optimization zero-shot, competing with task-specific methods and outperforming unsupervised RL and model-based baselines.

  • Takeaways & Limitations

    Grounding online unsupervised RL in unlabeled behaviors can produce human-like humanoid policies while retaining zero-shot capabilities across multiple task types.

  • Takeaways & Limitations

    FB-CPR struggles with tasks far from motion-capture data and is limited to proprioceptive observations, excluding environmental navigation and object interaction.

Abstract

from arXiv · show

Unsupervised reinforcement learning (RL) aims at pre-training agents that can solve a wide range of downstream tasks in complex environments. Despite recent advancements, existing approaches suffer from several limitations: they may require running an RL process on each downstream task to achieve a satisfactory performance, they may need access to datasets with good coverage or well-curated task-specific samples, or they may pre-train policies with unsupervised losses that are poorly correlated with the downstream tasks of interest. In this paper, we introduce a novel algorithm regularizing unsupervised RL towards imitating trajectories from unlabeled behavior datasets. The key technical novelty of our method, called Forward-Backward Representations with Conditional-Policy Regularization, is to train forward-backward representations to embed the unlabeled trajectories to the same latent space used to represent states, rewards, and policies, and use a latent-conditional discriminator to encourage policies to ``cover'' the states in the unlabeled behavior dataset. As a result, we can learn policies that are well aligned with the behaviors in the dataset, while retaining zero-shot generalization capabilities for reward-based and imitation tasks. We demonstrate the effectiveness of this new approach in a challenging humanoid control problem: leveraging observation-only motion capture datasets, we train Meta Motivo, the first humanoid behavioral foundation model that can be prompted to solve a variety of whole-body tasks, including motion tracking, goal reaching, and reward optimization. The resulting model is capable of expressing human-like behaviors and it achieves competitive performance with task-specific methods while outperforming state-of-the-art unsupervised RL and model-based baselines.

1 Introduction

The paper targets behavioral foundation models for humanoids that can control high-dimensional, unstable whole-body systems. It addresses limitations of existing unsupervised RL and imitation approaches by grounding zero-shot policies in unlabeled behaviors while retaining broad task generalization.

  • Motivation: Behavioral foundation models extend promptable foundation-model ideas to agents that interact with dynamic environments through actions.The paper focuses on humanoid agents and whole-body control from proprioceptive observations.
  • Limitations of prior work: Existing unsupervised RL may require downstream task training, broad exploration datasets, or losses weakly aligned with tasks of interest.These limitations motivate approaches that preserve zero-shot task solving while improving behavioral alignment.
  • Limitations of prior work: Sequence models can imitate high-quality demonstrations but often remain limited to demonstrated policies and struggle with unseen tasks.This creates a contrast between behavioral fidelity and zero-shot generalization.
  • Goal: The paper uses unlabeled trajectories to ground zero-shot RL toward useful behaviors while retaining zero-shot solutions for diverse downstream tasks.The stated goal is to combine behavioral grounding with broad task coverage.
  • Contribution: FB-CPR embeds unlabeled trajectories into the policy latent space and uses a latent-conditional discriminator to encourage policies to cover dataset states.This regularizes forward-backward policy learning toward observation-only behaviors.
  • Evaluation: The humanoid model solves motion tracking, goal reaching, and reward optimization in zero-shot, while remaining competitive with task-specific methods and outperforming unsupervised RL and model-based baselines.Experiments use the SMPL skeleton and the AMASS motion-capture dataset.

2 Preliminaries

Forward-Backward representations factor successor measures into finite-dimensional embeddings, enabling zero-shot evaluation and optimization across rewards, goals, and imitation tasks. Their performance can degrade with finite representation rank or poor transition coverage.

  • MDP setup: A discounted Markov decision process specifies states, actions, transitions, initial states, and discounting, while policies map states to action distributions.The framework is reward-free during pre-training.
  • Successor measures: Successor measures represent discounted future-state distributions under a policy after taking an action.They provide the environment-dependent component of action values.
  • Successor measures: Separating successor dynamics from rewards enables zero-shot policy evaluation for any reward once the successor measure is learned.The reward function supplies the task-specific information.
  • Forward-Backward representations: Low-rank successor-measure decompositions yield Forward-Backward representations that support both zero-shot policy evaluation and policy optimization.FB represents successor measures using forward and backward embeddings.
  • Zero-shot inference: FB models infer reward-maximizing, goal-reaching, and imitation policies from reward samples, target states, or demonstration trajectories without task-specific learning.Reward and goal information are mapped into the latent representation used by the policy.
  • Limitations: Finite representation rank and poor dataset coverage can cause FB models to collapse to few policies with poor downstream performance.The ideal guarantee requires sufficiently large rank and full coverage of the transition distribution.

3 FB with Conditional Policy Regularization

FB-CPR regularizes online FB training so latent-conditioned policies align with behaviors in observation-only trajectories. It infers trajectory latents with ERFB and uses conditional distribution matching implemented through a discriminator-derived reward.

  • Problem formulation: During pre-training, the agent combines observation-only trajectories with online environment interaction and an unsupervised transition replay buffer.The behavior dataset contains states but no actions or latent variables.
  • Problem formulation: FB-CPR pre-trains a continuous set of latent-conditioned policies whose behavior space is represented by joint state-latent distributions.The policy-induced distribution is pπ(s, z)=ν(z)ρπz(s).
  • Trajectory embedding: ERFB assigns each unlabeled trajectory a latent representation, allowing its states to define a dataset joint distribution in the same latent space.The method uses a uniform distribution over trajectories and associates each trajectory state with its inferred latent.
  • Conditional policy regularization: The actor is regularized by minimizing the discrepancy between policy-induced and dataset-induced joint distributions.The regularization strength is controlled by α.
  • Distribution matching: The distribution-matching objective becomes an RL return using the log ratio between dataset and policy distributions as the reward.This makes the regularizer tractable within policy learning.
  • Distribution matching: A latent-conditioned discriminator estimates the distribution ratio, and a critic learns action values for the resulting approximate reward.The actor loss combines this critic with the FB policy-improvement objective.
  • Online training: FB-CPR trains online by alternating latent-conditioned rollouts, replay-buffer storage, and off-policy updates to the FB representations, discriminator, critic, and actor.The latent distribution can allocate capacity to meaningful tasks while preserving broad support.
  • Design rationale: Unlike global state-marginal matching, conditional regularization can align policies with individual trajectories by directly embedding them into the latent policy space.This avoids requiring one policy to reproduce all dataset states globally.

4 Experiments on Humanoid

The experiments evaluate FB-CPR on physically constrained whole-body humanoid tasks using SMPL and AMASS motion data. Across reward optimization, goal reaching, and tracking, FB-CPR provides strong zero-shot performance, while ablations examine its conditioning and data requirements.

  • Environment and data: The evaluation uses an SMPL humanoid with 23 actuated bodies, 358-dimensional proprioceptive states, and physically plausible joint and torque limits.Simulation runs at 450 Hz with control at 30 Hz; actions specify normalized PD targets.
  • Evaluation protocol: The benchmark covers reward optimization, goal reaching, and motion tracking, comparing single-task, multi-task, and planning-based baselines.Single-task baselines are retrained per task, while multi-task methods leverage the motion dataset M.
  • Aggregate results: 73.4%: FB-CPR’s average normalized performance across all categories relative to the top-line algorithms, while performing zero-shot without additional learning or planning.FB-CPR also outperforms ASE by more than 1.4 times in every task category and matches or surpasses specialized unsupervised RL methods.
  • Reward maximization: 61%: FB-CPR’s reward-task performance relative to TD3 retrained from scratch for each reward, while reaching 177% relative to DIFFUSER.FB-CPR inference and execution take about 12 seconds per 300-step episode, compared with at least 30 minutes for ORACLEMPPI and about 5 hours for DIFFUSER.
  • Goal reaching: 48% and 118%: FB-CPR’s performance increases over ASE on goal-reaching proximity and success, respectively, while performing similarly to specialized goal-based baselines.FB-CPR has higher proximity but lower success than planning-based approaches.
  • Motion tracking: 80% and 88%: FB-CPR’s performance relative to the top-line scorers for tracking EMD and success, alongside an overall 83% success rate on the test dataset.FB-CPR outperforms zero-shot and planning-based baselines, while Goal-TD3 improves success by about 9% but drops EMD by 1%.
  • Human evaluation: 83%: human evaluators judged FB-CPR more human-like than TD3, compared with 4% judging TD3 more natural, despite TD3 achieving higher rewards.The evaluation covered all 45 rewards and 50 goals; the authors connect the difference to underspecified rewards and motion regularization.
  • Ablations: Policy-conditioned discrimination improves tracking over state-only conditioning while performing similarly on reward tasks, supporting the role of ERFB motion embeddings.The ablation compares latent-conditioned discrimination with marginal state-distribution matching.

5 Conclusions

FB-CPR combines zero-shot forward-backward models with behavior-trajectory regularization to train a humanoid behavioral foundation model. The paper reports broad zero-shot performance while identifying limitations in data coverage, observations, and theoretical understanding.

  • 5 Conclusions: FB-CPR combines forward-backward representations with unlabeled behavior regularization to ground online unsupervised policy learning.The method embeds unlabeled trajectories in the policy latent space and uses a latent-conditional discriminator to encourage state coverage.
  • 5 Conclusions: FB-CPR remains limited by incomplete theoretical understanding, dependence on motion-capture data, proprioceptive observations, and difficulty with tasks far from captured motions.The paper specifically notes challenges with ground movements, falling or standing behaviors, navigation, and object interaction.
  • 5 Conclusions: The trained humanoid behavioral foundation model achieves state-of-the-art performance across a variety of zero-shot control tasks.The implementation uses an SMPL humanoid and AMASS motion-capture data for regularization.
  • Algorithmic details: The evaluation uses AMASS train and test splits and compares against methods that may require action-labeled demonstrations or different behavioral priors.The implementation describes AMASS preprocessing and a Goal-GAIL instance for matching motions.

C.3.1 Reward-based evaluation

The reward-based evaluation spans eight reward categories and 55 representative tasks, measuring cumulative return over 300-step episodes. Tasks cover postural control, locomotion, manipulation-like reaching, rotation, jumping, and ground movement.

  • Reward categories: The reward suite includes ground poses, crawling, locomotion, standing, handstands, arm raising, rotation, and jumping.Rewards are normalized to the range [0, 1].
  • Arm raising: Arm-raising tasks independently control left and right hand heights across low, medium, and high target ranges, producing nine tasks.The category also penalizes unnecessary movements and high actions.
  • Rotation: Rotation tasks require target angular velocity around a cardinal axis while preserving body alignment, height, and low control effort.Task names encode the axis, speed, and height parameters.
  • Evaluation caveat: FB-CPR relies on state-based reward inference, so avoiding high-speed movements depends entirely on regularized pre-training.The high-action penalty is described as a common way to discourage rapid unnatural movements.
  • Evaluation setup: 55 representative tasks are evaluated using cumulative return over episodes of H = 300 steps.The test initial distribution mixes random falling positions with a subset of AMASS, differing from the training distribution.

C.3.2 Motion tracking evaluation

Motion tracking evaluates whether a humanoid can replicate motion sequences and reach manually selected goal poses, using strict state-alignment and average-distance metrics alongside specified training protocols and baselines.

  • Motion tracking: Tracking initializes the agent in a motion’s first state and simulates the motion length to assess sequence replication.Success requires matching motion states at every step under a threshold ξ = 0.5.
  • Motion tracking: The tracking distance compares joint positions and rotations with d_smpl, but exact matching can fail because motions may be discontinuous, physically unrealizable, or involve objects.The dataset curation process cannot exclude every non-realizable motion because it was performed manually.
  • Goal-based evaluation: Goal-based evaluation uses 50 manually extracted poses spanning different body heights and body-part positions, including 38 training and 12 test poses.Figure 5 shows 10 example poses.
  • Goal-based evaluation: Goal success records whether the agent reaches a sufficiently close state, while proximity averages a soft distance-based score across 300-step episodes.The per-step score is 1 within threshold β and decreases linearly to 0 beyond β + σ.
  • Training protocol: Online training uses 50 parallel environments, a 300-step episode horizon, motion-dataset initialization, and periodically updated motion priorities.Agents receive 3M gradient steps, equivalent to 30M environment steps, except PHC, which uses 300M environment steps.
  • FB-CPR configuration: FB-CPR’s discriminator takes state and latent inputs and uses cross-entropy training with a Wasserstein GAN gradient penalty coefficient of 10.The discriminator is a three-hidden-layer MLP with 1024 units per layer and a learning rate of 10^-5.

D.1 Detailed Results

Detailed results are organized across reward optimization, goal reaching, and motion tracking, with comparisons against task-specific, unsupervised, offline, and model-based baselines plus ablations.

  • Evaluation results: Results are reported separately for reward-based tasks, goal poses, and AMASS motion-tracking subsets using average return, proximity, success rate, and EMD.Tables 18–22 provide the corresponding detailed task and dataset breakdowns.
  • Baseline results: H-GAP achieved an average normalized reward-optimization performance of 0.05 on a subset of stand and locomotion tasks after extensive tuning and model scaling.The authors report difficulties training an accurate Prior Transformer and did not test other modalities.
  • Ablations: Figure 6 evaluates sampling distributions, discriminator gradient penalties, and policy regularization methods across five seeds with bootstrapped 95% confidence intervals.The ablations cover mixed sampling, gradient-penalty choices, and behavior cloning versus moment matching when action labels are available.

D.2 Ablations

The ablations examine discriminator stabilization, mixture sampling, latent dimensionality, action-free regularization, and the necessity of unlabeled-data regularization in FB-CPR.

  • Gradient penalty: λGP = 10 performed best for both tested discriminator gradient penalties, while λGP = 0 caused unstable training and subpar performance.The comparison averaged results over 5 independent seeds.
  • Mixture sampling: Mixed sampling balances trajectory embeddings from the motion dataset with uniformly sampled hypersphere embeddings, whereas either component alone degrades part of the evaluation.Trajectory-only sampling harms reward evaluation, while hypersphere-only sampling substantially degrades performance across tasks.
  • Regularization versus behavior cloning: With action labels available, the action-free conditional discriminator outperforms behavior-cloning regularization on both reward and tracking evaluation.The best sweep settings were αBC = 0.2 and αCPR = 0.01.
  • Latent dimension: Performance improves nearly monotonically through latent dimensions 128 and 256, with performance saturating there before dropping at d = 512.The authors select d = 256 as a trade-off among capacity, succinctness, and training stability.
  • Unlabeled-data regularization: The plain online FB algorithm achieves near-zero performance, showing that unlabeled-data regularization is necessary for learning useful diverse behaviors in this setting.Its limited gains came from roughly standing up without maintaining a standing position; FB-CPR directs learning toward dataset behaviors.

D.3.1 Human Evaluation

Human raters compare TD3 and FB-CPR on reward and goal tasks using success and human-likeness judgments, revealing a trade-off between reward scores and natural behavior.

  • Evaluation protocol: Human raters evaluate paired TD3 and FB-CPR videos using subjective success and human-likeness metrics.The study covers 45 reward tasks and 50 goal tasks, with 50 raters per matching video pair.
  • Evaluation protocol: Success judgments ask whether the behavior achieves the desired objective, using a target pose for goal tasks and natural-language reward descriptions for reward tasks.The task description replaces the [description] placeholder in the rater form.
  • Findings: FB-CPR can be judged more human-like even when raters consider it unsuccessful, indicating that naturalness and task completion are distinct subjective metrics.The passage reports this pattern for the evaluated behaviors.
  • Findings: FB-CPR’s human-likeness may come with lower reward scores, but it does not reduce perceived success on goal tasks and gives FB-CPR a higher goal-task success rate than TD3.The reported comparison distinguishes reward scores from goal-task success judgments.

D.3.2 Reward-based tasks

The reward-task evaluation tests whether FB-CPR composes locomotion with arm raising and handles under-specified human-like reward descriptions while avoiding unnatural high-performing behaviors.

  • Combined tasks: FB-CPR achieves 74% of TD3’s performance when combining medium-speed forward locomotion with arm-raising tasks.The composite rewards use weighted averages because locomotion and arm raising have conflicting movement and stillness objectives.
  • Combined tasks: About 98% of human evaluators rated FB-CPR as more natural than TD3 on the combined locomotion-and-arm-raising tasks.Figure 11 illustrates natural FB-CPR motion, high-performing but unnatural TD3 behavior, and misaligned ASE behavior.
  • Under-specified tasks: Under-specified reward functions can permit high-performing policies that deviate from natural behavioral biases or hack the reward through unnatural motions.The section evaluates whether FB-CPR expresses human-like behavior under such ambiguity.
  • Under-specified tasks: FB-CPR produces human-like solutions for under-specified tasks, distinguishes medium and high right-foot-standing targets, and recovers forward running when direction is not specified.The inferred behaviors reflect biases present in the motion dataset, while ASE does not solve all tasks.

D.4 Comparison to Unsupervised Skill Discovery Methods

The comparison shows that unsupervised skill discovery without suitable inductive bias struggles on high-dimensional Humanoid control, while motion-data regularization or restricted representations improves behavior discovery.

  • METRA comparison: METRA achieves near-zero performance across tasks when trained from scratch with the full Humanoid observation vector.Its learned policies fall and remain still in different positions despite optimizing the representation-learning objectives.
  • METRA comparison: The failure arises because diverse stationary floor positions satisfy METRA’s diversity objective without encouraging locomotion or more complex skills.The authors attribute successful prior Humanoid results partly to pixel observations, ground color maps, and a small latent dimension d = 2.
  • Inductive bias: Restricting METRA’s representation to pelvis linear velocities prevents skill collapse and biases discovery toward locomotion across x, y, and z.The modified representation focuses learning on those controlled variables alone.
  • Inductive bias: Adding ASE-style regularization to METRA improves performance by using unlabeled data to focus behavior on human-like motions and avoid over-actuated movements.This comparison replaces DIAYN with METRA while retaining the same policy-regularization components.
  • Latent-space structure: FB-CPR uses its latent space simultaneously as state, trajectory, and policy embeddings, linking behavior representation to zero-shot control.The roles are represented through B(s), ERFB, and πz.

E.1 Diversity, Dataset Coverage and Transitions

FB-CPR learns diverse behaviors that remain grounded in the motion dataset while extending beyond strict imitation. Its policies provide broad motion coverage, diverse trajectories, and latent alignment between motions and rewards.

  • Diversity: FB-CPR produces the highest behavioral diversity among the compared methods.Its EMD distribution is consistently more diverse than ASE and CALM.
  • Dataset Coverage: FB-CPR achieves the highest motion-dataset coverage, increasing smoothly as the EMD matching threshold grows.Coverage is measured as the fraction of training motions selected at least once; matches above EMD 4.5 are discarded.
  • Dataset Coverage: ASE concentrates generated trajectories on a small subset of motions, whereas FB-CPR distributes matches more evenly across the dataset.The comparison uses the 50 most frequently matched motions with MATCHTHRESHOLD = 0.1.
  • Transitions: All evaluated algorithms show similar overall pairwise transition coverage when stitching source and destination motions.Transition probabilities are estimated from 10,000 sampled source–destination trajectory pairs.
  • Beyond Imitation: Random FB-CPR policies often generate behaviors that are not tightly connected to training motions, unlike CALM and ASE.The comparison suggests FB-CPR uses more policy capacity for behaviors beyond direct reproduction of the dataset.
  • Latent Structure: Motion embeddings cluster similar activities, while reward embeddings appear within related motion clusters, aligning motions and rewards in one latent space.The analysis includes jumping, running, walking, crawling, headstands, and cartwheels, and interpolation can combine two reward-conditioned behaviors.

F Ablations on Bipedal Walker

Walker ablations test how forward-backward representations, conditional policy regularization, online training, and data coverage affect downstream performance. Demonstration data improves FB substantially, while online FB without useful samples performs worst.

  • Data and Evaluation: The Walker data mix includes 17,284 rollouts and 1,333,717 transitions for MTRAIN, alongside 5,000,000 transitions from RND.Evaluation uses 42 held-out reward and imitation tasks and 25 held-out goals.
  • Setup: The ablation compares FB-CPR with offline and online FB, offline FB-AWAC, and FB-MPR using an unconditional discriminator.The study is conducted in the dm_control Walker domain across run, spin, and crawl task families.
  • Results: Adding demonstration samples to RND improves FB performance by 15% on reward tasks, 30% on imitation, and 60% on goal reaching.Results are averaged over 10 seeds and use normalized task performance or goal proximity as specified.
  • Results: Online FB without demonstrations or unsupervised data achieves the worst performance among the evaluated algorithms.The authors attribute this to representation collapse from a lack of useful samples, which also prevents effective exploration.
  • Results: Offline FB trained with only RND data performs well when the dataset provides good coverage.The result supports the role of sufficiently covering data in learning a broad range of downstream policies.

G Ablations on AntMaze

AntMaze ablations examine whether conditional policy regularization and online training improve goal-reaching with stitch demonstrations. FB-CPR achieves the strongest performance among the compared FB variants.

  • Evaluation: The AntMaze tasks specify target x,y coordinates while randomly generating the remaining state components.Each goal is evaluated over 50 episodes.
  • Results: FB-CPR achieves the strongest AntMaze performance in both success rate and proximity.Performance is averaged across three models trained with different random seeds.
  • Results: Online FB fails to reach any test goals, achieving zero success rate because of insufficient exploration.The comparison includes medium and large mazes with five downstream tasks per domain.
  • Results: FB-AWAC achieves decent performance and is competitive with non-hierarchical offline goal-conditioned RL methods reported for the benchmark.It uses offline stitch datasets consisting of short goal-reaching demonstrations designed to test subgoal stitching.
Loading 2504.11054v1…