PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Parkour Learning

Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, Hang Zhao

arXiv:2309.05665v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Specialized parkour policies can perform one skill but cannot autonomously switch among skills based on visual perception. This paper combines two-stage reinforcement learning with policy distillation, demonstrating autonomous parkour across challenging obstacles on two low-cost robots.

  • Problem

    Specialized policies can execute one skill but cannot autonomously switch between parkour skills based on visual perception of the environment.

    Source: §3.2 · p. 5

  • Method

    The system learns parkour skills with two-stage reinforcement learning using soft then hard dynamics constraints, and distills them into a single vision-based policy.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    A1 and Go1 autonomously selected and executed parkour skills, including 0.40m climbing, 0.60m gap leaps, 0.2m crawling, and 0.28m slit-squeezing.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The system traversed challenging open-world environments using onboard computation, visual sensing, and power.

    Source: §1 · p. 3

    Manually constructed simulation environments limit how automatically new skills can be learned, since new skills require environments with different obstacles and appearances.

    Source: §5 · p. 8

Abstract

from arXiv · show

Parkour is a grand challenge for legged locomotion that requires robots to overcome various obstacles rapidly in complex environments. Existing methods can generate either diverse but blind locomotion skills or vision-based but specialized skills by using reference animal data or complex rewards. However, autonomous parkour requires robots to learn generalizable skills that are both vision-based and diverse to perceive and react to various scenarios. In this work, we propose a system for learning a single end-to-end vision-based parkour policy of diverse parkour skills using a simple reward without any reference motion data. We develop a reinforcement learning method inspired by direct collocation to generate parkour skills, including climbing over high obstacles, leaping over large gaps, crawling beneath low barriers, squeezing through thin slits, and running. We distill these skills into a single vision-based parkour policy and transfer it to a quadrupedal robot using its egocentric depth camera. We demonstrate that our system can empower two different low-cost robots to autonomously select and execute appropriate parkour skills to traverse challenging real-world environments.

1 Introduction

The paper presents a low-cost quadrupedal robot system that learns diverse parkour skills and combines them into a vision-based policy for autonomous traversal.

1 Introduction

  • The system addresses the challenge of learning diverse skills while handling small-robot obstacles, laggy perception during fast locomotion, and motor-damage risk.The introduction notes that at 1 m/s, a 0.2-second communication delay creates a 0.2 m perception discrepancy.Source: §1 · p. 2
  • A two-stage reinforcement-learning method uses penetrable obstacles during soft-constraint pre-training, then fine-tunes with hard dynamics constraints and realistic dynamics.An automatic curriculum minimizes obstacle penetrations; both stages use a simple reward for forward movement while conserving mechanical energy.Source: §1 · p. 2, Figure 3 · p. 3
  • After individual skills are learned, DAgger distills them into one vision-based parkour policy deployable with onboard perception and computation.The policy uses the robot’s egocentric depth camera, as described in the paper context.Source: §1 · p. 2
  • The paper contributes an open-source parkour-learning system and a two-stage reinforcement-learning approach for overcoming difficult exploration problems.The approach pairs soft dynamics constraints in pre-training with hard constraints in fine-tuning.Source: §1 · p. 2
  • The policy enables two low-cost robots, A1 and Go1, to autonomously select and execute parkour skills using onboard visual sensing, computation, and power.The demonstrated skills include climbing 0.40 m obstacles, leaping 0.60 m gaps, crawling beneath 0.2 m barriers, squeezing through 0.28 m slits by tilting, and running.Source: §1 · p. 3

2 Related Work

Prior agile-locomotion methods cover many skills, while vision-based systems use modular perception pipelines or learned visual inputs; the paper targets a single end-to-end vision-based policy for diverse parkour skills.

Agile Locomotion

  • Agile locomotion research spans model-based and learning-based skills, but most methods are blind or rely on state estimation and specialize in individual skills [48] [16] [57] [35].Reported capabilities include jumping, climbing, running, and other agile tasks.Source: §2 · p. 3

Vision-Based Locomotion

  • End-to-end learning-based locomotion uses depth, elevation maps, lidar, RGB, event cameras, or learned neural spaces, but none has demonstrated effective high-speed agile locomotion [29] [61] [28] [109] [111] [112] [113] [32] [68] [30] [33].In contrast, the paper builds a single end-to-end vision-based parkour policy for varied skills.Source: §2 · p. 3

3 Robot Parkour Learning Systems

The system learns five agile parkour skills and combines them in an end-to-end policy that uses onboard depth sensing and proprioception to control a low-cost robot. Its training pipeline uses two-stage reinforcement learning, policy distillation, and sim-to-real measures.

3.1 Parkour Skills Learning via Two-Stage RL

  • A direct-collocation-inspired two-stage RL framework trains five specialized parkour skills using a shared forward, energy, and alive reward.The GRU skill policies use privileged obstacle geometry and category information and train separately on corresponding terrains. Pre-training uses soft dynamics constraints, a penetration reward, and an adaptive obstacle-difficulty curriculum; fine-tuning uses hard constraints, while running skips pre-training. The target speed is around 1 m/s, and the energy reward penalizes squared motor power.Source: §3.1 · pp. 4–5, §3 · p. 4

3.2 Learning a Single Parkour Policy by Distillation

  • Using DAgger [44] [45], the authors distill five specialized skill policies into a single vision-based parkour policy using onboard sensing, training in simulation on 40 tracks with 20 obstacles each.In simulation, the known obstacle type determines which specialized policy teaches the parkour policy; for example, the climb policy supervises it at a high obstacle.Source: §3.2 · pp. 5–6

3.3 Sim-to-Real and Deployment

  • Sim-to-real deployment aligns simulated and onboard depth images with preprocessing while running the parkour policy at 50 Hz.Simulation uses depth clipping, Gaussian noise, and random artifacts; real images use clipping, hole-filling, and spatial and temporal smoothing. The camera refreshes at 10 Hz, and target joint positions are converted to torques through a PD controller with a 25 Nm torque limit; deployment also calibrates visual delays and enforces motor-safety measures.Source: §3 · p. 4, §3.3 · p. 6

4 Experimental Results

Across simulation and real-world tests, the vision-based parkour policy learned and executed diverse skills, achieving the strongest indoor performance across skills while traversing difficult obstacles.

4 Experimental Results

  • All policies were trained in IsaacGym [119] using simulated environments of 40 tracks with 20 obstacles each, whose obstacle difficulties increased linearly based on obstacle-property ranges.Comparisons included Blind, RND [120], MLP, and RMA [8] baselines, plus No Distill, Oracles w/o Soft Dyn, and Oracles ablations.Source: §4 · pp. 6–7

4.2 Real-World Experiments

  • 0.40m climbs (80%), 0.60m leaps (80%), and 0.2m crawls (90%) were achieved; the policy also tilted through 0.28m slits and performed best across skills.Indoor success rates were averaged over 10 trials at each difficulty; slit width was less than the robot width.Source: Figure 6 · p. 7, §4.2 · p. 8

4.1 Simulation Experiments

  • Blind achieved 13% crawling success and no climbing, leaping, or tilting; MLP failed climbing and leaping and performed much worse on crawling and tilting.Climbing and leaping require short-term memory of earlier visual observations, such as obstacle dimensions needed to control the rear legs during a climb.Source: §4.1 · pp. 7–8
  • Around 95% final average success, soft-dynamics pretraining learned faster; RND and hard-dynamics oracles made no progress on climbing or leaping.Success rates were averaged over 100 trials across climbing, leaping, crawling, and tilting, with three random seeds per method.Source: Figure 7 · p. 7, §4.1 · pp. 7–8
  • No Distill failed climbing, leaping, and tilting; RMA performed similarly on tilting but could not combine multiple specialized skills into one policy.RMA copied MLP parameters from a specialized skill, preventing it from distilling multiple skills with different MLP parameters; direct visual training without privileged information hindered No Distill.Source: §4.1 · p. 8

4.2 Real-World Experiments

  • The robot spontaneously retried failed climbs by pushing away to create run-up space, and outdoor trials included leaping a 0.6m gap and climbing 0.42m stairs.Outdoor demonstrations also included crawling beneath a camping cart and moving over slippery grass.Source: §4.2 · p. 8

5 Conclusion, Limitations and Future Directions

The authors present a two-stage reinforcement learning system for parkour skills and report robust performance across challenging skills and indoor and outdoor environments. Its current reliance on manually constructed simulation environments limits how automatically new skills can be learned; future work aims to automate environment construction and train from RGB images.

5 Conclusion, Limitations and Future Directions

  • The system shows robust performance on challenging parkour skills in indoor and outdoor environments, after extensive simulation and real-world testing.The paper presents it as a system for low-cost robots.Source: §5 · p. 8
  • A two-stage reinforcement learning method addresses difficult exploration problems in learning parkour skills.Source: §5 · p. 8
  • The current system requires manually constructed simulation environments, so new skills require adding environments with different obstacles and appearances.This reduces how automatically new skills can be learned.Source: §5 · p. 8
  • Future work will investigate automatically constructing diverse simulations from large-scale real-world data using 3D vision and graphics, and training agile locomotion skills directly from semantic RGB rather than depth images.Source: §5 · p. 8

A Experiment Videos

The paper reports real-world analysis and provides indoor and outdoor experiment videos online.

A Experiment Videos

  • Indoor and outdoor experiment videos documenting real-world analysis are available on the project website.The passage gives the address https://robot-parkour.github.io.Source: A Experiment Videos · p. 16

B Details of Training in Simulation

Simulation training uses specialized-skill policies and a distilled vision-based parkour policy, with task-specific rewards, environment randomization, and a CNN–GRU–MLP architecture.

Specialized Skills

  • Specialized-skill policies flatten proprioception, prior action, GRU state, privileged visual and physics inputs into a 256-unit GRU and [512, 256, 128] MLP producing 12 tanh joint targets.Actions are scaled by 0.4 for hip joints and 0.6 for thigh and knee joints.Source: B Details of Training in Simulation · p. 16

Rewards, Environments and PPO

  • Five specialized policies use task-specific rewards, fractal-noise terrains, fixed velocity commands, and soft-dynamics RL pretraining or hard-dynamics fine-tuning.Specified obstacle dimensions include 0.8 m width and length for climbing, 0.8 m width and depth for leaping, and 0.8 m width for crawling.Source: B Details of Training in Simulation · p. 16
  • Training randomizes robot mass, base center of mass, motor strength, terrain friction, depth-perception latency, camera position and field of view, and proprioception delay.PPO hyperparameters are listed in the supplementary material.Source: B Details of Training in Simulation · p. 16

Parkour Policy

Distillation

  • Distillation uses binary cross-entropy, and both specialized-skill and parkour-policy outputs range from −1 to 1.The compared actions are the corresponding specialized skill's action and the parkour policy's action.Source: B Details of Training in Simulation · p. 19

C Details of Simulation Setup

Simulation used IsaacGym Preview 4 with a pre-generated curriculum terrain, and the specialized policies and distilled parkour policy were trained using separate compute setups.

C Details of Simulation Setup

  • The simulator used a static map of 800 tracks arranged in a 20-by-40 grid, with linearly increasing difficulty and end-to-end connections along +x.Tracks in each row shared difficulty but differed in non-essential configurations; completing a track moved the robot to a more difficult one.Source: C Details of Simulation Setup · p. 19
  • Each specialized policy trained for 12 hours in soft dynamics and 6 hours of tuning in hard dynamics; distillation used four computers, each with one Nvidia 3090 GPU.Three computers loaded the current model and collected trajectories and specialized-policy supervision, while the fourth trained the parkour policy from the latest trajectories.Source: C Details of Simulation Setup · p. 19

D Details of Robot Setup

Real-world experiments use a Unitree A1 with onboard depth sensing and computation, running a vision-based policy through a ROS-based control pipeline.

D Details of Robot Setup

  • The experimental platform is a 12-joint Unitree A1 with an onboard Nvidia Jetson NX and front-facing Intel RealSense D435 depth camera.The robot runs ROS1 on Ubuntu 18.04, and a Unitree SDK-based ROS package exchanges states and policy commands at 100Hz while providing roll/pitch and estimated-torque limits and an emergency stop.Source: D Details of Robot Setup · p. 19
  • Depth images are captured at 240 X 424, filtered, cropped, and down-sampled to 48 X 64 before asynchronous visual encoding.The encoder sends visual embeddings to the main network script via ROS at 10Hz.Source: D Details of Robot Setup · p. 20
  • The policy runs at 50Hz, combining robot proprioception and visual embeddings to compute position-control actions clipped to a 25Nm torque limit.Commands use Kp = 50.0 and Kd = 1.0; clipping is based on current joint position and velocity.Source: D Details of Robot Setup · p. 20

E Detailed Comparison Studies on RL Pre-Training with Soft Dynamics Constraints

With soft dynamics constraints, the method is the only one reported to complete climbing and leaping; RND and Oracles w/o Soft Dyn. fail at both skills.

E Detailed Comparison Studies on RL Pre-Training with Soft Dynamics Constraints

  • The soft-dynamics method alone completes climbing and leaping; RND fails at climbing, leaping, and tilting but succeeds at crawling, while Oracles w/o Soft Dyn. learn crawling and tilting but fail at climbing and leaping.The paper identifies climbing and leaping as its most difficult skills.Source: E Detailed Comparison Studies on RL… · p. 20

Related papers

Loading 2309.05665v2…