Source-linked AI summary
Attention-Based Map Encoding for Learning Generalized Legged Locomotion
Junzhe He, Chong Zhang, Fabian Jenelten, Ruben Grandia, Moritz BÄcher, Marco Hutter
TL;DR
Sparse-terrain locomotion demands precise foothold planning, robustness to uncertainty, and generalization across terrains, while existing learning-based and model-based approaches have complementary limitations. The paper trains an attention-based map encoding within an end-to-end reinforcement-learning controller, combining terrain features with proprioception. The resulting controllers generalized across challenging terrains on quadrupedal and humanoid robots, including real-world scenarios and an unseen obstacle parkour.
Problem
Precise, robust, and generalized end-to-end locomotion on sparse terrains, including sim-to-real transfer across quadrupedal and bipedal robots, remains missing.
Method
The approach uses a CNN to embed local height-map features and multi-head attention to select point-wise terrain features conditioned on proprioception within an end-to-end controller.
Results
The framework enabled precise and generalized locomotion with robustness to uncertainty on ANYmal-D and GR-1 across diverse, including unseen, terrains.
Takeaways & Limitations
Attention-based map encoding can provide interpretable terrain representations while combining learning-based robustness with precise foothold-oriented locomotion.
Takeaways & Limitations
Training can take several days, the 2.5D height-map representation may not suit confined spaces, and arm use for manipulation was not studied.
Abstract
from arXiv · showhide
Dynamic locomotion of legged robots is a critical yet challenging topic in expanding the operational range of mobile robots. It requires precise planning when possible footholds are sparse, robustness against uncertainties and disturbances, and generalizability across diverse terrains. While traditional model-based controllers excel at planning on complex terrains, they struggle with real-world uncertainties. Learning-based controllers offer robustness to such uncertainties but often lack precision on terrains with sparse steppable areas. Hybrid methods achieve enhanced robustness on sparse terrains by combining both methods but are computationally demanding and constrained by the inherent limitations of model-based planners. To achieve generalized legged locomotion on diverse terrains while preserving the robustness of learning-based controllers, this paper proposes to learn an attention-based map encoding conditioned on robot proprioception, which is trained as part of the end-to-end controller using reinforcement learning. We show that the network learns to focus on steppable areas for future footholds when the robot dynamically navigates diverse and challenging terrains. We synthesize behaviors that exhibit robustness against uncertainties while enabling precise and agile traversal of sparse terrains. Additionally, our method offers a way to interpret the topographical perception of a neural network. We have trained two controllers for a 12-DoF quadrupedal robot and a 23-DoF humanoid robot respectively and tested the resulting controllers in the real world under various challenging indoor and outdoor scenarios, including ones unseen during training.
INTRODUCTION
Generalized legged locomotion still requires precise foothold planning on sparse terrain, robustness to uncertainty, and transfer across diverse terrains. The paper proposes an attention-based end-to-end learning framework to address these requirements for quadrupedal and humanoid robots.
- Sparse terrains remain difficult for learning-based controllers because valid footholds are hard to discover and learn from.
- Model-based controllers can predict accurate footholds but rely on assumptions about state estimation, maps, and robot models that degrade under real-world mismatch.
- Hybrid approaches combine learning and model-based planning but retain computational demands and fragility to deployment uncertainty.
- The proposed map encoding uses a CNN for local terrain features and multi-head attention to query point-wise features alongside proprioception.
- A two-stage pipeline first trains on base terrains with perfect perception, then adds challenging terrains, perception noise, and drift.
- The framework targets precise, generalized, and robust locomotion while producing graphically interpretable map representations on ANYmal-D and GR-1.
A. Precise and Generalized Locomotion
The controllers generalized from staged simulation training to diverse unseen terrains and challenging hardware scenarios. Both robots also traversed an unseen obstacle parkour with disturbances and uncertainties.
- GR-1 generalized from base terrains to unseen terrain types while maintaining precise foot placements.
- ANYmal-D adapted to unseen terrains despite different kinematics from GR-1, demonstrating generalization across embodiments.
- Stage-two fine-tuning added terrain types, disturbances, and uncertainties to improve precision and real-world robustness.
- 100% success rate was achieved by both GR-1 and ANYmal-D on an unseen obstacle parkour with disturbances and uncertainties.
- Hardware tests covered sparse gaps, stepping stones, beams, movable supports, and height differences for both robots.
- The paper presents its end-to-end DRL controller as achieving precise and generalized performance across challenging terrains.
B. Agility and Recovery Reflexes by Whole-Body Coordination
The learned controllers developed whole-body agility and recovery behaviors on real robots. These behaviors supported climbing, stabilization, and recovery from slips or difficult foothold placements.
- Whole-body policies enabled ANYmal-D to use its knee for climbing and recovery, while GR-1 used terrain-dependent arm swing and rapid stepping responses.
- GR-1 performed an in-air single-leg switch hop when a commanded foothold became difficult to land.
- These reflex behaviors enhanced system stability and are difficult to obtain from contact-state machines and handcrafted heuristics.
C. Versatile Velocity Tracking
The learned controllers tracked varied velocity commands while navigating sparse, unstable terrain. Their responses included omnidirectional maneuvering, altered stride lengths, and command-dependent stepping patterns.
- ANYmal-D maneuvered omnidirectionally across debris with movable supports and sparse terrain.
- GR-1 changed stride length when its velocity command increased from 0.7 m/s to 1.5 m/s on a shaky balance beam.
- At 1.5 m/s, GR-1 used one step per stepping stone, whereas 0.7 m/s produced two steps per stone.
- Simulation evaluations compared velocity tracking, success, stuck, and failure outcomes against other controllers on selected terrains.
D.1. Benchmark with DTC and baseline RL controller
The proposed controller is benchmarked against DTC and baseline-rl on sparse terrains using velocity tracking and task-success criteria. It achieves lower tracking errors than DTC in most tested conditions and is evaluated under matched simulated uncertainties.
- Controller comparison: The benchmark compares the proposed method, DTC, and baseline-rl on an ANYmal-D navigating sparse terrains.Evaluation uses velocity tracking, success/failure/stuck rates, and terrain-specific success rates.
- Evaluation protocol: All controllers are deployed in the same simulated environment with observation noises and drifts sampled using the same random seed.This standardizes the simulated uncertainty conditions across controllers.
- Velocity tracking: The proposed method demonstrates substantially lower velocity-tracking errors than DTC except for gaps with small velocity commands.Tracking error is computed only for surviving agents.
- Velocity tracking: DTC exhibits high tracking errors at large velocity commands because its constant gait frequency produces farther, harder-to-track footholds.The proposed method adjusts gait frequencies across terrains and velocity commands.
D.2. Ablation study on two-stage training
The ablation study examines the proposed two-stage training pipeline and shows that the full method reaches and maintains the most difficult terrain level. Its deployment success rates exceed those of both ablated variants on almost all terrains.
- Training performance: The full method converges to terrain level 6, indicating that most robots solve the most difficult training level.Perfect training is characterized by an initial overshoot followed by convergence to level 6.
- Training performance: C2 fails to reach the same terrain level, while C3 fails to converge back to level 6.These curves indicate failure to upgrade to higher levels for C2 and failure to solve the most difficult levels for C3.
- Deployment performance: The proposed method achieves substantially higher success rates than C2 and C3 on almost all tested terrains.C2 performs worst despite perfect perception during testing.
- Deployment performance: C3 performs satisfactorily on stairs, pits, and rough terrain but worse on sparse terrains such as grid stones and beams.The ablation therefore distinguishes performance on sparse terrain from performance on less foothold-constrained terrain.
D.3. Ablation study on network structure
The proposed controller combines local CNN terrain features with proprioception-conditioned multi-head attention, while its end-to-end design is motivated by precision, robustness, generalization, and interpretability. Visualizations indicate attention concentrates on feasible future footholds across mixed, commanded, and unseen terrains.
- Network structure: The map encoder uses a CNN for local terrain features and multi-head attention to query point-wise features conditioned on proprioception.The resulting representation is passed to a policy that generates joint-level actions.
- Attention behavior: Attention weights concentrate around the next steppable region on mixed terrain, indicating guidance toward feasible footholds.The visualization describes the stage 2 fine-tuned controller and links the concentration to stable navigation on discontinuous terrain.
- Attention behavior: The stage 1 controller adapts attention across base terrains and commanded forward, sideways, and turning directions.Figure 7 distinguishes terrain-specific attention patterns under different velocity commands.
- Generalization: The stage 1 controller generalizes its attention mechanism to unseen pentagon stones, narrow pallets, single-column stones, and consecutive gaps.These visualizations associate adaptation with proprioceptive information and command direction.
- Design rationale: The end-to-end architecture is intended to retain learning-based robustness while providing precise traversal of sparse terrains and reducing modular system complexity.The paper characterizes the map encoder as implicitly selecting future footholds and the subsequent policy as producing whole-body motions.
- Training pipeline: Training proceeds in two stages, beginning with base terrains under perfect perception and adding harder terrains, noise, and drift during fine-tuning.The pipeline is designed to initialize map-encoding learning before exposing the controller to more realistic conditions.
Training
The controllers were trained with largely shared network parameters on two robot platforms, using platform-specific map dimensions and hardware configurations. Training used two stages and substantial parallel simulation, with separate compute setups for ANYmal-D and GR-1.
- ANYmal-D: ANYmal-D used 4096 parallel robots for 18,000 stage-1 and 3,600 stage-2 epochs, requiring 6 days on an Nvidia Tesla A100-40GB GPU.The reported 24 seconds per epoch reduced training time by roughly 60% compared to DTC.
- GR-1: GR-1 training used 15,000 stage-1 and 3,200 stage-2 epochs, requiring 3.5 days on an Nvidia RTX 4090 GPU.Each epoch took 14 seconds.
- Hardware and architecture: The two controllers shared network parameters, but GR-1 used smaller map scans of 17 × 11 versus ANYmal-D’s 26 × 16.ANYmal-D policy inference ran on an Intel core-i7 8850H CPU with elevation mapping on an onboard Nvidia Jetson; GR-1 inference ran on an Intel core-i7 13700h CPU.
Training Details
Training used base terrains for initial learning and added fine-tuning terrains for the second stage. The supplementary setup also specifies shared PPO settings, robot joint indexing, and terrain examples used across the curriculum.
- Terrains: Base terrains included grid stones, narrow beams, pits, and pallets, while fine-tuning terrains added pentagon stones, single-column stones, narrow pallets, rough hills, rings, and beams.The listed terrain designs include randomly placed stepping stones, radiating beams, inclined rough ground, and ring-shaped steps.
- Terrains: Stage-2 fine-tuning used additional terrain types including pentagon stones, rough hills, rings, and beams radiating outward from the center.These terrains were added for stage 2 fine-tuning.
- Terrain curriculum: A supplementary movie visualizes the training terrains and their curriculum.The visualization is provided under the method section.
- Optimization: All policies used the same PPO parameters in 4096 parallelized environments.The PPO hyperparameters are listed in Table S1.
- Robot configuration: The training setup defines joint indices for 12 ANYmal-D joints and 23 GR-1 joints.The indices correspond to the robots’ leg, waist, shoulder, elbow, and related joints as listed in the supplementary methods.
Ablation Study Details
The ablation study compares three network structures for processing map scans and generating map encodings. These alternatives differ in whether map features use CNN down-sampling, patch-based vision-transformer extraction, or concatenation with proprioception before transformer encoding.
- Ablation design: Figures S1, S2, and S3 present the network structures used in the ablation study.The figures cover the transformer encoder, CNN down-sampling, and vision-transformer alternatives.
- Transformer Encoder: The transformer-encoder structure embeds map scans with a CNN, embeds proprioception linearly, concatenates both features, and sends them to a transformer encoder.The transformer encoder generates the map encoding.
- CNN down-sampling: The CNN down-sampling structure uses two layers with kernel sizes 5 and 7, zero padding, and stride 1.This configuration is presented as an ablation network structure.
- Vision Transformer: The vision-transformer structure splits map scans into 2 × 2 patches before feature extraction.The patches are sent to a vision transformer.