PaperlayerView exact revision on arXiv

Source-linked AI summary

Generate, Track, Improve: Perceptive Multi-Skill Humanoid Locomotion with RL-Fine-Tuned Motion Generators

Zachary Olkin, William D. Compton, Aaron D. Ames

arXiv:2609.31577v1cs.RO

TL;DR

Humanoid locomotion still faces a gap between multi-skill whole-body control and terrain-aware perception. This paper pairs a perceptive motion generator with a tracking policy and fine-tunes the generator off-policy, improving traversal success and skill selection.

  • Problem

    Terrain-aware locomotion policies may lack extensible whole-body, multi-speed control, while multi-purpose whole-body controllers lack terrain-aware perception and motions.

    Source: §I · p. 1

  • Method

    The paper fine-tunes a perceptive flow-matching generator with off-policy RL, structured rollout search, critic-based advantage labels, and advantage weighted regression.

    Source: B. Contributions · p. 3, E. RL Fine Tuning for the Generator · p. 6

  • Results

    The off-policy loop improves success rate by up to 25 percentage points and mode selection by up to 80 percentage points.

    Source: §IV · p. 10

  • Takeaways & Limitations

    A single policy pair was demonstrated walking, running, standing, jumping on and off boxes, and traversing stairs in real-world settings, including outdoors.

    Source: §IV · pp. 10–11

    Hand-designed rewards may not extend to every motion, depth input lacks semantic information, and parts of the data pipeline still require human or heuristic choices.

    Source: A. Limitations · p. 11

Abstract

from arXiv · show

General purpose humanoids require locomotion controllers that are multi-skill, perceptive, dynamic, and robust enough to go anywhere humans can. In this work, we present a two layer locomotion architecture: (1) a perceptive flow matching motion generator plans whole body trajectories from raw depth images while a (2) perceptive tracking policy trained with control-guided RL follows these motions. Both policies are trained on a library of terrain consistent motion clips created with dynamically optimized human data which yields both accurate velocity tracking and terrain consistent references. Our central contribution is a simple yet effective off-policy RL fine tuning loop that improves the motion generator. A structured search method is used with the generator to gather data for advantage weighted regression. This off-policy loop is much more sample efficient than on-policy residual fine tuning and improves terrain consistency on unseen geometries and skill compositions. We find that successful terrain traversals increased by up to 25 percentage points and skill selection improved by up to 80 percentage points. By using raw depth images to perceive the environment no odometry or height maps are needed, and outdoor deployment is easy. With two cameras, the policy can see terrain coming from further away and adjust its velocity regardless of the commanded speed so it can traverse the terrain. A single policy pair enables a Unitree G1 humanoid to walk, run, stand, jump on and off of boxes, and traverse stairs in outdoor environments. Project page: https://zolkin1.github.io/generate-track-improve/

I. INTRODUCTION

The paper addresses the gap between terrain-aware locomotion and multi-skill whole-body control by pairing perceptive motion generation with perceptive tracking. It presents a layered architecture, generator fine-tuning, and raw-depth perception for real-world humanoid locomotion.

I. INTRODUCTION

  • Terrain-aware policies often lack extensible whole-body, multi-speed control, while multi-purpose whole-body controllers lack perception and terrain-aware motions.The paper frames this split as a challenge for operating in real-world environments at commanded speeds and interacting with the environment.Source: §I · p. 1

A. Related Works

  • Prior approaches use height fields or direct sensor measurements; the paper contrasts its raw-depth approach with methods requiring odometry or height scans.The related-work discussion also identifies limits in prior generator-tracker systems, including flat-ground scope, simulation-only evaluation, or fine-tuning requirements.Source: A. Related Works · p. 2

B. Contributions

  • The proposed generator-tracker pair combines perceptive terrain-aware references with real-time tracking, and its layered design supports modularity and adding skills.The motion generator is a flow-matching transformer conditioned on raw depth and velocity commands; the data pipeline uses optimized human references and Motion Bricks [47] to create terrain-consistent clips.Source: B. Contributions · p. 2, §I · p. 2
  • Off-policy RL fine-tuning uses structured search and advantage weighted regression to improve generator success rate by up to 25 percentage points and skill selection by up to 80 percentage points.The method perturbs flow-matching initial noise and conditioning rather than adding noise to the high-dimensional trajectory outputs.Source: B. Contributions · p. 3
  • The policies use depth images from two cameras without height maps or odometry, and are demonstrated walking, running, standing, jumping, and traversing stairs in lab and outdoor settings.The forward-facing camera lets the robot see terrain ahead and adjust velocity before reaching it.Source: B. Contributions · p. 3

II. METHODS

The system separates fast tracking from slower whole-body motion generation, with both policies trained from terrain-consistent motion references. Those references are built by optimizing human motions and creating clips across skills, velocities, and terrain geometries.

A. System Overview

  • The tracker runs at 50 Hz, while the generator is queried every 0.24 seconds to produce a 1.24-second whole-body trajectory from velocity, prior action, and two depth images.The tracker uses proprioception and one depth image; the data pipeline supplies terrain-consistent references, and fine-tuning improves the already-trained generator.Source: A. System Overview · p. 3

B. Motion Clip Creation

  • Motion references begin with 252 flat-ground locomotion motions and 46 terrain motions for jumping onto and off boxes and traversing stairs.Multiple-shooting optimization enforces periodicity and average velocity; terrain motions are retargeted across geometries using contact information and then dynamically optimized for feasible, smooth motions.Source: B. Motion Clip Creation · p. 3
  • The optimized references are joined into 6–20-second clips using sampled velocity profiles and Motion Bricks [47], yielding 10,000 clips across 140 tile geometries.The method overview describes training the tracker with CLF-RL, training the generator with supervised flow-MSE loss, then fine-tuning it with off-policy RL using rollout data, a critic, advantage labels, and advantage weighted regression.Source: B. Motion Clip Creation · p. 4, Figure 2 · p. 4

C. Tracker

The tracker is an RL policy trained to follow pre-made motions while using proprioception and depth input. Its training combines tracking and constraint-related rewards with simulation randomization for transfer to hardware.

C. Tracker

  • The tracker uses a CLF-RL variant to track body positions across links, with rewards for tracking and penalties for undesired contact, torque limits, joint limits, action rate, and torque.The policy uses an additional scaling term to balance numerical properties while retaining CLF properties.Source: C. Tracker · p. 4
  • The perceptive tracker receives a 30×26 depth scan alongside proprioceptive information, reference motion, and commanded velocity through a feed-forward MLP.On hardware, the scan is formed by down-sampling the 600p camera image.Source: C. Tracker · p. 4
  • The tracker runs at 50 Hz and outputs joint-position targets for PD controllers; training randomizes physical properties, camera settings, and observation noise.It was trained with PPO and an asymmetric actor-critic in 8192 IsaacLab environments for approximately 48 hours.Source: C. Tracker · p. 4

D. Generator

The generator is an autoregressive flow-matching transformer that uses depth images, velocity commands, and trajectory history to produce reference trajectories. Its depth inputs include robot-relative position coordinates, which the authors found critical for terrain understanding.

D. Generator

  • The autoregressive flow-matching transformer uses depth images as its sensor input, alongside velocity commands and trajectory-history tokens, to output reference trajectories.History consists of a subset of nodes from the previously generated trajectory.Source: D. Generator · pp. 4–5
  • Table I lists tracker observation terms and sizes, with history-bearing terms stacked over the stated number of steps.Source: Table I · p. 5
  • The transformer separates conditioning tokens from trajectory outputs, allowing its prefix encoder to be cached and reused during flow integration.Depth tokens receive positional embeddings based on their camera-field-of-view angle.Source: D. Generator · p. 5
  • Depth inputs include raw depth, 3D positions, and a validity flag; the position channels were critical for terrain performance without generator proprioception.The coordinates are expressed in a gravity-aligned root frame, helping distinguish terrain approach from robot motion.Source: D. Generator · p. 5
  • Training data comes from tracker rollouts, then terrain-related examples are augmented by shifting robot state and recasting depth images while keeping the reference fixed.Three augmented points are generated per relevant data point.Source: D. Generator · p. 5
  • History dropout and noise train the generator to map commands and depth to trajectories and to remain robust during autoregressive deployment.Twenty percent of training histories use a learned null embedding; the deployed policy is teacher-forced in training but remains stable in autoregressive rollouts.Source: D. Generator · p. 5

E. RL Fine Tuning for the Generator

The generator fine-tuning method uses off-policy RL with structured exploration and advantage-weighted supervised updates, while keeping the tracker fixed. Its reward design prioritizes terrain consistency, and pretraining data can be retained during fine-tuning.

E. RL Fine Tuning for the Generator

  • The method fine-tunes the generator, not the tracker, using off-policy RL because Gaussian noise over nearly 3,000 trajectory outputs makes exploration inefficient.The generator acts at replanning steps while the frozen tracker is part of the robot dynamics.Source: E. RL Fine Tuning for the Generator · pp. 5–6
  • Instead of perturbing every output, structured search varies conditioning, robot positions, and initial flow noise to explore reasonable trajectories.Perturbed rollouts retain nominal conditioning for policy training, and off-policy learning reduces the amount of data needed.Source: E. RL Fine Tuning for the Generator · pp. 5–6
  • The action is a whole-body trajectory with 62 nodes of 44 dimensions each, totaling 2,728 values per replanning step.Source: E. RL Fine Tuning for the Generator · p. 6
  • Generator rewards prioritize terrain consistency, while velocity tracking and success rewards become significant only when terrain consistency is good.Success is gated by the episode's worst terrain consistency.Source: E. RL Fine Tuning for the Generator · p. 6, Table II · p. 7
  • A critic trained with fitted value iteration labels rollout advantages, which weight the supervised flow-matching update.Two critics are trained on separate data halves and label advantages on the opposite half; returns used for labels are unbootstrapped.Source: E. RL Fine Tuning for the Generator · pp. 6–7
  • Fine-tuning can include pretraining examples at unit weight to help prevent forgetting, and the reported training uses five iterations of five update epochs.Each iteration takes about six hours on one H100 GPU, with runtime varying by terrain and episode length.Source: E. RL Fine Tuning for the Generator · p. 7

F. Deployment

Deployment uses an NVIDIA Jetson Thor for policy inference, with two depth cameras processed by a Jetson Orin. The generator is queried every 0.24 seconds, making sensor-to-trajectory delay relevant for dynamic motion.

F. Deployment

  • The NVIDIA Jetson Thor runs flow-matching inference in about 11 ms and tracker inference in under 1 ms.Source: F. Deployment · p. 7
  • A Zed X and Zed X mini feed depth data to a Jetson Orin for processing and downsampling before ROS2 sends it to the Thor.Source: F. Deployment · p. 7
  • The generator is queried every 0.24 seconds, and the authors identify mitigating sensor-to-trajectory delay as critical for dynamic motions.Source: F. Deployment · p. 7

III. RESULTS

The results evaluate fine-tuning across in-distribution and varied out-of-distribution terrains, alongside deployment on the Unitree G1. Fine-tuning improves traversal success, terrain consistency, and mode selection, and the multi-terrain variant is used on hardware.

III. RESULTS

  • The evaluation studies fine-tuning, algorithm choice, velocity tracking, terrain traversal, camera count, and data pipeline, with Unitree G1 experiments indoors and outdoors.Source: §III · p. 7

A. RL Fine Tuning

  • Out-of-distribution fine-tuning configurations included boxes, up-and-down stairs, and mixed terrain with a realistic building entryway.The configurations varied task compositions, terrain dimensions, and added walls; an in-distribution terrain set was also evaluated.Source: A. RL Fine Tuning · pp. 7–8
  • Success rates increased by as much as 25 percentage points, mean contact penalty decreased, and one stairs terrain's correct mode selection rose from 0% to about 80%.Figure 4 reports success, mean contact penalty, and the fraction of sampled episodes with every mode correct; error bars show 95% confidence intervals.Source: A. RL Fine Tuning · p. 8, Figure 4 · p. 8
  • The multi-terrain fine-tuned policy was deployed on hardware and included pretraining data to help preserve previously learned motions.Source: A. RL Fine Tuning · p. 8

B. Algorithm Comparison

AWR matches or outperforms the tested generator fine-tuning alternatives in success rate and terrain consistency. The paper also evaluates velocity tracking during terrain traversal, including autonomous speed adjustment.

B. Algorithm Comparison

  • AWR matches or outperforms the other tested methods in success rate and terrain consistency across terrains and box termination settings.Plain behavior cloning performs poorly, while the selection variants generally fall short of AWR.Source: Figure 5 · p. 8, B. Algorithm Comparison · pp. 8–9
  • Removing conditioning perturbations reduces success by 46 percentage points, and the contact penalty is consistently worse on multi-terrain, especially stairs.The authors report that the learned advantage function, advantage-weighted updates, and conditioning perturbations each contribute to performance.Source: B. Algorithm Comparison · p. 9
  • After 2,000 iterations, PPO used more than 160x the data and 30x the wall-clock time, yet its success rate was about 13 percentage points below AWR’s.The authors note PPO might improve with further training but would require substantially more data and compute.Source: B. Algorithm Comparison · p. 9

C. Velocity Tracking and Terrain Traversal

  • The fine-tuned policy tracks commanded speeds and autonomously slows for obstacles before returning to the desired speed.The study used 50 MuJoCo rollouts per speed, with different flow-matching sampling seeds.Source: C. Velocity Tracking and Terrain… · p. 9

D. Hardware Deployment

The policy pair traverses varied real-world stairs, walks and runs indoors and outdoors, and jumps onto and off boxes. The section also reports data-pipeline comparisons and the importance of the forward-facing camera for anticipating terrain.

D. Hardware Deployment

  • The robot climbed 15 consecutive steps and traversed six real-world staircases with different geometries, while also walking, running, and jumping onto and off boxes.It approached stairs at 0.8, 1.5, and 2.0 m/s and ran at 2 m/s while turning; the policy was demonstrated indoors and outdoors.Source: D. Hardware Deployment · p. 9, Figure 6 · p. 9, Figure 7 · p. 10

E. Data Pipeline

  • Optimized reference trajectories paired with Motion Bricks achieve better velocity tracking than Motion Bricks alone; motion matching took more than 7x as long per clip and produced higher-jerk motions.The authors state that reference clips bound the final policies’ velocity-tracking performance because the policies are trained to track them.Source: E. Data Pipeline · pp. 9–10

F. Two Camera Ablation

  • Removing the upper camera reduces terrain success by up to 43%, with box failures largely caused by running into the obstacle.The forward-facing camera lets the policy see terrain ahead and adjust speed before reaching it.Source: Table IV · p. 10, F. Two Camera Ablation · p. 10

IV. CONCLUSION

The paper presents a perceptive generator-tracker pipeline with off-policy RL fine-tuning, demonstrated on a humanoid across multiple real-world locomotion skills. It identifies limits in rewards and data preparation and outlines work needed to scale the approach.

IV. CONCLUSION

  • The policy pair enables real-world walking, running, standing, box jumps, and stair traversal, while fine-tuning improves mode selection by up to 80 percentage points and success rate by up to 25 percentage points.The authors report demonstrations in lab and outdoor settings and describe the resulting motions as human-like, dynamic, and robust enough for deployment.Source: §IV · pp. 10–11

A. Limitations

  • Hand-designed rewards may not extend to every motion, and selecting human reference motions still relies on heuristics or human judgment.Transition hyperparameters also require hand tuning; the authors say these data-pipeline limits must be addressed before scaling to any motion.Source: A. Limitations · p. 11

B. Future Work

  • Future work includes better transition methods, terrain-aware human data, navigation and safety integration, and learning from real-world deployment data.The paper notes that its human data currently lacks ground-truth terrain information.Source: B. Future Work · p. 11

A. CLF-RL

The section rescales CLF tracking weights to temper velocity terms while preserving the CLF’s theoretical properties. Its proof uses decoupled CARE solutions to establish stability for the scaled function.

A. CLF-RL

  • Scaling S reweights CLF tracking terms because velocity can have higher variance than position, without violating theoretical CLF properties.The scaling addresses cases where velocity tracking would otherwise be overweighted.Source: A. CLF-RL · p. 12
  • Under the stated assumptions, V̄(η)=η⊤SPSη is an exponentially stabilizing CLF for positive diagonal S.The proposition requires positive entries in S and the assumptions of Proposition 1 of.Source: A. CLF-RL · p. 12
  • Block-diagonal output dynamics make the CARE decouple, with P=blkdiag(P1, . . . , Pn) and each Pi solving its output CARE.The proof reduces the claim to examining one output at a time.Source: A. CLF-RL · p. 12
  • Positive-definite Q̄ and P̄ establish a stabilizing CARE solution, allowing Proposition 1 of to apply with Q̄ and R̄.The section defines Q̄=blkdiag(Q̄i) and R̄=diag(r̄i).Source: A. CLF-RL · p. 12

Related papers

Loading 2609.31577v1…