Source-linked AI summary
Watch Your Step: Anticipated Perception for Humanoid Locomotion
Matthias Heyrman, Chenhao Li, Ioannis Dadiotis, Marco Hutter
TL;DR
Dense elevation maps expose policies to terrain features irrelevant to the next step, which can distract control and enlarge the state space, hindering generalization and increasing computational cost. A lightweight FPM predicts the next two footholds from proprioception and velocity command, and the policy samples elevation-map patches around them as the FPM is periodically refit on policy rollouts. Roughly one-third as many training iterations are needed to reach a given terrain difficulty compared with methods using the complete elevation map, and the policy transfers zero-shot to a Unitree G1.
Problem
Dense elevation maps expose policies to terrain features irrelevant to the next step, which can distract control and enlarge the state space, hindering generalization and increasing computational cost.
Source: §I · p. 1Method
A lightweight FPM predicts the next two footholds from proprioception and velocity command, and the policy samples elevation-map patches around them as the FPM is periodically refit on policy rollouts.
Source: §I · p. 2, §III · p. 3Results
Roughly one-third as many training iterations are needed to reach a given terrain difficulty compared with methods using the complete elevation map, and the policy transfers zero-shot to a Unitree G1.
Source: §I · p. 2, §VI · p. 8Takeaways & Limitations
The approach performs best on most tested sparse terrains, while full-map methods lead on stepped terrains and all methods perform similarly on continuous terrain.
Source: §V · pp. 7–8Terrain approaching from the side may enter the FPM patches late or not at all; full-map methods lead on stepped terrains, and all methods perform similarly on continuous terrain.
Source: §V · p. 8
Abstract
from arXiv · showhide
Perceptive locomotion policies typically observe a dense robot-centric elevation map, although only the terrain beneath and around the next footsteps informs the control decision. Learned encoders compress the whole visual input, leaving the policy to work out which inputs matter while learning the baseline locomotion task. Learning both at once slows training and, without computationally expensive attention mechanisms, makes generalization to new terrain harder. We instead remove redundant inputs from the elevation map before the policy sees it. A lightweight foothold prediction model (FPM) predicts future footholds from proprioception and the velocity command alone, and we sample the elevation map only in small patches around these predicted footholds. We train the prediction model alongside the locomotion policy, periodically retraining it on new policy rollouts as the robot's gait matures. This allows the perception to anticipate where the policy will step, while the policy learns to leverage terrain information around those steps. With the proposed lightweight preprocessing, the policy reaches a given terrain difficulty in roughly a third of the training iterations required with the complete map. It outperforms policies that observe the complete map where footholds are sparse, including in narrow walled corridors, and on unseen terrain, and it matches an attention-based encoder trained at several times the cost. The policy transfers zero-shot to a Unitree G1 humanoid traversing various obstacles, stairs, gaps, and step-downs beyond the training distribution in both indoor and outdoor environments.
I. INTRODUCTION
Dense elevation maps contain terrain irrelevant to the next steps, while learned encoders must discover which inputs matter. The paper predicts upcoming footholds from state and command, samples terrain around them, and periodically refits the predictor as the gait develops.
I. INTRODUCTION
- Dense maps include terrain the robot will not step on, while learned encoders must extract relevant features from the full map and retain redundant input.Irrelevant terrain may distract the policy and the larger state space can make generalization harder and increase policy computation.Source: §I · pp. 1–2
- Prior foothold-selection modules commonly hand targets to downstream controllers using handcrafted criteria, while prior learned planners do not use predictions to filter policy observations.The paper instead uses predicted footholds to determine which terrain patches the locomotion policy observes.Source: §I · p. 2
- The predictor must use robot state and command without observing terrain, yet predictions based on an early gait may become misplaced as the policy improves.A policy trained on misplaced patches may learn to distrust them.Source: §I · p. 2
- The proposed pipeline predicts the next two footholds from proprioception and velocity command, then samples elevation-map patches around them for the policy.The FPM and policy are co-trained, with the FPM periodically refit on rollouts from the current policy.Source: §I · p. 2
- Roughly one-third as many training iterations are needed to reach a given terrain difficulty compared with methods using the complete elevation map.The approach also supports indoor and outdoor zero-shot transfer to a Unitree G1 humanoid.Source: §I · p. 2
II. PRELIMINARIES
The paper formulates humanoid locomotion as an MDP and trains a PPO policy in parallel simulation environments. The robot state includes proprioception and the velocity command, while exteroception is treated separately.
II. PRELIMINARIES
- The locomotion problem is formulated as an MDP with state, action, transition dynamics, reward, and discount factor.The policy is trained with PPO across 8192 parallel IsaacLab environments on one NVIDIA RTX 3090 GPU for a 29-DoF Unitree G1.Source: §II · p. 2
- Physics runs at 200 Hz and policy control at 50 Hz.The velocity command is resampled during training, including turn-in-place and zero commands in 2% of environments each.Source: §II · pp. 2–3
- The system state includes base linear and angular velocity, projected gravity, velocity command, and joint positions and velocities relative to nominal configuration.Exteroceptive input is treated as a separate observation rather than part of the robot state.Source: §II · p. 3
III. METHOD
The framework pairs a foothold prediction model with a locomotion policy: the model predicts the next two footholds from system state, and the policy observes elevation-map samples around those predictions.
III. METHOD
- The FPM predicts the next two footholds from current system state, and the policy receives elevation-map samples around those predicted locations.PPO continuously optimizes the policy while the FPM is periodically refit in a self-supervised fashion to the current policy's rollouts.Source: §III · p. 3
- The framework co-trains its two learned components by optimizing the policy with PPO and refitting the FPM at fixed intervals on current-policy rollouts.This makes the foothold predictions track the policy whose terrain observations they determine.Source: §III · p. 3
- The policy's exteroceptive input is sampled only around the predicted footholds, rather than supplied as an unrestricted elevation map.The FPM predicts footholds from system state; the policy uses the resulting terrain samples.Source: §III · p. 3
A. Foothold Prediction Model (FPM)
The FPM predicts future foothold positions from the current system state, using labels collected from policy rollouts. Its target is the next two landings, and it is trained with Huber-loss regression.
A. Foothold Prediction Model (FPM)
- The FPM maps the current system state to planar positions of the next two footholds in the current base frame, without observing terrain or its previous output.This prediction models the closed-loop gait of the current locomotion policy.Source: A. Foothold Prediction Model (FPM) · p. 3
1) Prediction target:
- Each recorded state is labeled with the next two footholds, regardless of which foot makes them; pre-fall states are retained when their next foothold occurs before termination.The first target is the landing of the airborne or about-to-lift foot; the second is the other foot's first contact after its upcoming lift-off.Source: 1) Prediction target · p. 3
2) Loss:
- The FPM is trained by supervised regression on recorded states and labeled footholds using Huber loss with threshold δ = 0.01.The bounded gradient limits the influence of outliers near episode boundaries.Source: 2) Loss · p. 3
3) Network:
The FPM normalizes its proprioceptive input and predicts footholds, while the policy receives height patches sampled around those predictions. Randomized patches reduce reliance on exact query locations, and the critic retains the full map during training.
3) Network:
- The FPM is a two-hidden-layer MLP that normalizes its input using per-dimension training-split statistics stored in the exported model.Deployment therefore needs no external normalization.Source: 3) Network · p. 3
B. Foothold-Gated Height Sampling
- The policy receives two 7 × 7 height patches around predicted footholds, reducing its elevation-map input from 693 cells to 102 values.The patches are preceded by their predicted footholds so the policy can interpret them relative to location.Source: B. Foothold-Gated Height Sampling · p. 3, Figure 2 · p. 4
- Per-episode patch offsets and pointwise noise perturb sampled terrain, while the critic also receives the complete map during training.The complete map is not needed at deployment.Source: B. Foothold-Gated Height Sampling · pp. 3–4
C. Co-Training Schedule
The schedule alternates policy learning and self-supervised FPM refitting, beginning with a bootstrap phase before foothold predictions are trained. Each refit uses rollouts from the current policy so the predictor tracks its evolving gait.
C. Co-Training Schedule
- Every 500 PPO iterations, the schedule gathers current-policy rollouts and refits the FPM before using it to gate observations for the next interval.Each PPO iteration comprises 24 environment steps across 8192 parallel environments.Source: C. Co-Training Schedule · p. 4, 2) FPM refit · p. 4
1) Bootstrap:
- For the first 500 RL iterations, fixed footholds beneath the base provide a height scan while the policy learns to walk on easy terrain.The FPM is not initially fit because the policy has not yet learned to stand or walk.Source: 1) Bootstrap · p. 4
2) FPM refit:
- Each FPM refit uses policy-rollout foothold labels and trains from scratch for 25 epochs on a shuffled 80/20 train-validation split.The rollout dataset records state, foot contacts, and base and foot poses.Source: 2) FPM refit · p. 4
D. Training Curriculum and Implementation
Training uses a ten-level terrain curriculum, torso assistance that decreases with terrain progress, and randomized environments and observations. The RL reward combines velocity-command tracking with regularization.
D. Training Curriculum and Implementation
- The ten-level terrain curriculum is dominated by stepping stones, stairs, and boxes, with slopes, rough ground, pits, and flat terrain making up the remainder.Stepping stones account for 30% of the terrain suite; upright and inverted pyramid stairs account for 15% each.Source: D. Training Curriculum and… · p. 4
- Torso assistance decreases with mean terrain level, carrying half the robot’s weight at level 0 and none from level 4 onward.The assistance gives the policy room to adjust steps mid-air on sparse terrain.Source: D. Training Curriculum and… · p. 4
- The RL reward combines velocity-command tracking, including stand-still behavior, with regularization terms; mirrored transitions augment each batch.Table I lists the FPM learning settings and the policy parameters and rewards.Source: D. Training Curriculum and… · p. 5, Table I · p. 5
- Training randomizes environmental and robot parameters, applies external disturbances, adds proprioceptive noise, and injects occasional base-velocity spikes.These perturbations are used to account for inaccurate models, non-ideal terrain, and possible events affecting the robot.Source: D. Training Curriculum and… · p. 5
IV. EXPERIMENTS AND RESULTS
The FPM policy reaches terrain-curriculum level 5 in 7.5k iterations, faster than the full-map and attention-based comparisons. Experiments compare it with alternative map inputs and encoders on training and unseen terrains.
IV. EXPERIMENTS AND RESULTS
- Evaluation uses five trained seeds and 1000 episodes per terrain on the training set and five previously unseen OOD terrains.The OOD set includes waves, discrete obstacles, high-amplitude random terrain, concentric gaps, and sparse rectangular pallets.Source: §IV · p. 5
- 7.5k iterations to reach level 5: FPM is 3.5× faster than HM and 2.7× faster than AME2.The figure reports mean curriculum level over five seeds, with shaded bands showing one standard deviation.Source: Figure 3 · p. 5
- Comparisons include the full elevation map, scans around current feet, a velocity-shifted crop, a CNN encoder, and the attention-based AME2 encoder.The methods share the policy architecture, PPO hyperparameters, reward set, curricula, task inputs, and asymmetric critics.Source: §IV · p. 5
A. Learning speed
FPM reaches terrain difficulty faster than the compared methods and performs strongly on sparse terrain, including in narrow corridors. The Unitree G1 also traverses indoor and outdoor obstacles, including some outside the training distribution.
A. Learning speed
- FPM reaches terrain curriculum level 5 after 7.5k iterations, a 3.4–3.5× speedup over full-map methods and 2.7× over AME2.At 40k iterations, AME2 ends marginally above FPM but requires four GPUs and 6.3 s per policy update, versus 0.55 s for FPM.Source: A. Learning speed · p. 5
C. Sim-to-Real
- The Unitree G1 traversed indoor and outdoor obstacles, including OOD gaps and drops; outdoor trials included a staircase with 18 continuous steps.Tracking error was greatest on OOD obstacles, and the policy slowed before gaps and drops, lowering its body before jumping off drops.Source: Figure 4 · p. 6, C. Sim-to-Real · pp. 6–7
B. Terrain traversal performance
- 89.2% overall success in the corridor test, with FPM highest on sparse terrain (74.2%) and second on continuous terrain (99.5%).AME2 is second on sparse terrain (71.7%); the other approaches cover less than half the commanded distance.Source: Table II · p. 6, B. Terrain traversal performance · p. 6
- FPM has the highest locomotion-test success on training terrains (84.6%) and sparse stepping stones (80.2%), and the lowest OOD fall rate (5.2%).AME2 performs better on OOD terrains (83.6% against 83.1%) and has a lower training-terrain fall rate (2.4%).Source: Table II · p. 6, B. Terrain traversal performance · p. 6
D. Ablations
The ablations test how foothold prediction horizon and CNN encoding of sampled patches affect locomotion. FPM predicts the next two footholds; shorter and longer horizons expose different performance and prediction-accuracy trade-offs.
D. Ablations
- FPM predicts the next two future footholds, while FPM1 pairs the latest foothold with the next and FPM3 predicts the second and third future footholds.The ablations also test adding a CNN encoder to the sampled patches.Source: D. Ablations · p. 7
- 74.2% against 62.2% on sparse corridor terrains: FPM outperforms FPM1, while adding a CNN raises FPM1's overall corridor performance to 86.3%.FPM without an encoder still performs best overall.Source: D. Ablations · p. 7
- FPM3 performs lower overall, and its third-step prediction has roughly 22 cm of error, approximately the length of the G1 foot.FPM3's second-future-foothold prediction is less accurate than FPM's; adding a CNN leaves its success within two points and does not lower its OOD fall rate.Source: D. Ablations · p. 7
- Prediction error rises roughly 6 cm per step, and predictions remain within a G1 foot length up to the second future foothold.The standard deviation also increases with horizon.Source: D. Ablations · p. 7
V. DISCUSSION AND LIMITATIONS
The discussion attributes the approach's efficiency to co-trained, state-based sampling at predicted footholds, while noting limits on terrain coverage and prediction horizon. The conclusion summarizes the framework and identifies possible future extensions.
V. DISCUSSION AND LIMITATIONS
- Co-trained, state-based sampling lets the policy learn to use two patches chosen at predicted footholds instead of finding relevant cells across the full map.The authors conjecture that policy-dependent observations may encourage probing before committing while tracking the velocity command.Source: §V · p. 7
- Full-map methods lead on stepped terrain, and obstacles approaching from the side may enter FPM patches late or not at all.The corridor test does not evaluate side-approaching obstacles.Source: §V · p. 8
- The effective prediction horizon is limited to the second future foothold because farther predictions become less accurate and more variable, degrading locomotion performance.The authors suggest future extensions to active depth-based perception and to manipulative locomotion using predicted hand and foot contacts.Source: §V · p. 8, §VI · p. 8
VI. CONCLUSION
- The framework predicts two footholds from proprioception and the velocity command, then samples the elevation map around them; it learns several times faster than full-map policies and transfers zero-shot to a Unitree G1.It matches an attention-based encoder at a fraction of its training cost and performs best on most sparse terrains.Source: §VI · p. 8