Source-linked AI summary
Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-expert Distillation and RL Fine-tuning
Nikita Rudin, Junzhe He, Joshua Aurand, Marco Hutter
TL;DR
The paper addresses the difficulty of controlling legged robots across diverse, unstructured terrains. It combines separately trained expert skills, distillation, and RL fine-tuning, outperforming alternative skill-combination methods as the number of skills and terrains increases.
Problem
Existing legged-robot control methods struggle to match human or animal performance in complex environments, limiting applications such as search and rescue and space exploration.
Method
The framework trains terrain-specific expert skills, distills them into a single policy, and fine-tunes that policy with RL on additional terrains, including real-world 3D scans, using depth images as the only perceptive input.
Results
The proposed approach outperforms other skill-combination methods when the number of skills and terrains increases, while fine-tuning yields an average success rate 3.1% higher than the corresponding expert.
Takeaways & Limitations
A single controller can synthesize many locomotion skills and tackle a large variety of obstacles in unstructured terrains, though performance remains imperfect in some behaviors.
Takeaways & Limitations
Fine-tuning changes the distilled policy’s behavior and remains sensitive to shared setup choices, which can favor some terrains over others.
Abstract
from arXiv · showhide
Legged robots are well-suited for navigating terrains inaccessible to wheeled robots, making them ideal for applications in search and rescue or space exploration. However, current control methods often struggle to generalize across diverse, unstructured environments. This paper introduces a novel framework for agile locomotion of legged robots by combining multi-expert distillation with reinforcement learning (RL) fine-tuning to achieve robust generalization. Initially, terrain-specific expert policies are trained to develop specialized locomotion skills. These policies are then distilled into a unified foundation policy via the DAgger algorithm. The distilled policy is subsequently fine-tuned using RL on a broader terrain set, including real-world 3D scans. The framework allows further adaptation to new terrains through repeated fine-tuning. The proposed policy leverages depth images as exteroceptive inputs, enabling robust navigation across diverse, unstructured terrains. Experimental results demonstrate significant performance improvements over existing methods in synthesizing multi-terrain skills into a single controller. Deployment on the ANYmal D robot validates the policy's ability to navigate complex environments with agility and robustness, setting a new benchmark for legged robot locomotion.
1 Introduction
Legged-robot control remains difficult to generalize across unstructured terrains, because task-specific policies do not readily combine or adapt to unseen obstacles. The paper addresses this with multi-expert distillation, RL fine-tuning, depth-based perception, and repeated skill expansion.
- Motivation: Existing RL policies often solve only the narrow terrains used for training, without reusing knowledge across skills or preserving earlier abilities.A general controller must combine individual skills, remain adaptable to new scenarios, and avoid catastrophic forgetting.
- Motivation: Agile locomotion also requires perception despite partial observability, because unseen obstacle regions may lie outside the sensors’ field of view during approach.Standard terrain reconstruction can depend on precise state estimation that is unavailable during agile motions.
- Approach: The framework trains separate expert skills, compares combination methods, distills them into a common controller, and repeatedly fine-tunes it with RL on additional terrains.The approach treats the distilled controller as a foundation that can incorporate new terrain data while retaining previously learned skills.
- Approach: Distillation provides non-trivial performance across terrains, while hierarchical and latent-encoding approaches tend to solve only subsets and generalize poorly to complex obstacles.The comparison concerns methods for combining separate motor skills into one controller.
- Results: Fine-tuning improves basic-terrain performance, enables complex terrains without directly trained experts, and can produce new behaviors such as body motions that improve depth-camera visibility.The resulting policy can also be deployed on terrains not seen during training.
- Results: Repeated fine-tuning adds new obstacle-solving skills while maintaining performance on previously seen tasks, supporting scalable expansion of the locomotion controller.The policy uses depth images as its only perceptive input and was tested in indoor and outdoor environments, including search-and-rescue facilities.
2 Method
The method trains terrain-specific experts, distills them into a depth-image foundation policy, and fine-tunes that policy with RL across simulated and scanned real-world terrains. The resulting pipeline addresses multi-terrain skill synthesis, adaptation, and depth-based perception for agile locomotion.
- Expert skill training: Nine terrain-specific locomotion skills are trained separately with RL before being combined into a single foundation policy.The expert skills use elevation maps as exteroceptive input during their individual training.
- Policy distillation: DAgger-style supervised distillation trains the foundation policy to imitate terrain-matched experts from student observations.The student collects trajectories online, pairs its observations with expert actions, and minimizes an action-imitation loss.
- RL fine-tuning: RL fine-tuning expands training from the nine expert terrains to those terrains plus 15 new terrains reconstructed from real-world rubble scans.Fine-tuning uses no expert supervision and is performed on a combined terrain set.
- Depth-based policy architecture: The student replaces elevation maps with four onboard depth images and combines CNN features, proprioception, recurrent memory, and task commands to predict actions.Each depth image is processed separately, the extracted features are passed through LSTM layers, and an MLP produces the next action.
- RL fine-tuning: Fine-tuning is introduced because distillation reduces individual-skill performance and generalizes poorly to unseen terrains; critic pretraining and controlled exploration improve RL robustness.The critic is pretrained while policy weights remain frozen, and the initial RL policy-distribution standard deviation is reduced.
- Depth observation modeling: Sim-to-real depth transfer is addressed by degrading simulated images to mimic camera artifacts and applying matching processing to real images.The simulation pipeline clips depth, adds edge noise, inserts holes, downsamples, and models temporally consistent missing data.
3 Experiments
The experiments evaluate multi-terrain skill combination, repeated fine-tuning, perception, and real-world deployment. Distillation followed by RL fine-tuning improves performance and generalization, while remaining imperfect on some out-of-distribution terrains and in precision.
- Success rate on different terrains: The evaluation compares individual experts, the distilled policy, and the fine-tuned policy across terrains seen and unseen during training.Table 4 includes nine expert terrains, terrains added during fine-tuning, and terrains reserved for generalization evaluation.
- Success rate on different terrains: After distillation, success rate drops by 10.4% on average, with larger losses on ambiguous or precision-demanding terrains.The distilled policy can mix expert behaviors sub-optimally when terrain identity is ambiguous and may fail to imitate experts after changing perceptive modality.
- Success rate on different terrains: After fine-tuning, average performance matches or surpasses all nine experts and is 3.1% higher than the corresponding expert on average.Performance increases on Low wall, Parkour line, and scanned meshes added during fine-tuning, but remains mixed on terrains unseen throughout training.
- Repeated Fine-tuning on new terrains: Repeated fine-tuning learns new capabilities without significantly changing performance on previously seen terrains, and retaining diverse training terrains improves final performance.The policy adapts to new obstacles more effectively when prior fine-tuning is retained and training includes previously used terrains, even when the new terrain contributes only 3% of samples.
- Skill combination methods: Compared with hierarchical and VAE-based skill-combination methods, distillation with fine-tuning performs best on new terrains when starting from pre-trained networks.VAE training from scratch is more efficient than the alternatives, but the hierarchical approach and standard RL lack scalability across many terrains.
- Active perception and real-world deployment: Fine-tuning enables active perception behaviors with depth images, but precision remains imperfect and knee-heavy climbs and jumps increase motor impacts and hardware wear.The policy learns to reposition and tilt the body to bring obstacles into view; footstep selection still produces avoidable missteps.
4 Discussion
The discussion finds that multi-expert distillation followed by RL fine-tuning is more scalable than alternative skill-combination methods, while perception, tuning, precision, and memory remain limitations. Fine-tuning can add expert-free tasks, but it also reintroduces some training and balancing challenges.
- Skill combination methods: Standard multi-expert distillation initially underperforms the experts and fails to generalize because contradictory expert actions make imitation ill-posed.Changing the perception modality between experts and the student further exacerbates the challenge.
- Skill combination methods: RL fine-tuning effectively modifies the distilled policy to solve expert tasks and incorporate new tasks without existing experts.The distilled policy extracts substantial expert knowledge despite lacking sufficient robustness for direct real-world deployment.
- Skill combination methods: Directly training all tasks from scratch tends to focus on a subset, produce a shared sub-optimal behavior, and require untractable joint reward, terrain, and curriculum tuning.Separate expert training keeps tuning specialized to each task.
- Skill combination methods: Fine-tuning reduces but does not eliminate tuning effort because common rewards and termination settings can favor some terrains over others.Although skills are frozen during distillation, fine-tuning changes their behavior and recreates some from-scratch training challenges.
- Skill combination methods: Hierarchical and latent-encoding approaches struggle as skills and terrains increase, whereas the proposed approach outperforms alternatives in that scaling regime.The hierarchical approach cannot blend or interpolate skills, and pre-training on expert terrains does not solve its difficulty using all skills.
- Perception and policy limitations: End-to-end depth-image control avoids interface information loss but makes errors harder to interpret and component-wise tuning impossible.The policy also shows imprecise footholds, obstacle contacts, and limited longer-term memory; higher depth resolution and improved sim-to-real transfer may improve behavior.
5 Conclusion
The paper presents a three-stage pipeline that combines separately trained skills into a general locomotion policy and extends it through repeated RL fine-tuning. Comparisons show stronger scaling with more skills and terrains, although precision, memory, noise modeling, and distillation remain improvement areas.
- Conclusion: The three-stage pipeline trains individual skills separately, distills them into one policy, and fine-tunes that policy with RL on simulated and real-world scanned terrains.Depth images are the policy’s only perceptive input, and a custom depth-noise model supports real-world transfer.
- Conclusion: Repeated fine-tuning with additional terrains continuously extends the policy’s capabilities.The training set includes 3D scans from real-world search-and-rescue training grounds.
- Conclusion: The proposed approach outperforms previously used skill-combination methods as the number of skills and terrains increases.
- Conclusion: Future improvements should address depth-noise modeling, long-term memory, and learning from contradicting expert policies.The authors also report that both qualitative and quantitative performance can improve.