Source-linked AI summary
Extreme Parkour with Legged Robots
Xuxin Cheng, Kexin Shi, Ananye Agarwal, Deepak Pathak
TL;DR
Extreme robot parkour requires precise behavior despite noisy sensing and actuation, which challenges conventional modular systems. The paper trains a single depth-image policy with simulation reinforcement learning and distillation-based heading control. The robot performs jumps, handstands, and tilted-ramp traversal, including obstacles twice its height or length.
Problem
Extreme parkour requires tightly coupled perception and control, while low-cost robots provide noisy, laggy actuation and artifact-prone depth perception.
Method
The paper trains a single neural network with simulation reinforcement learning to output motor commands from depth images, using dual distillation and a unified inner-product reward.
Results
The robot performs high jumps over obstacles 2x its height, long jumps across gaps 2x its length, handstands, tilted-ramp traversal, and robust terrain behaviors.
Takeaways & Limitations
End-to-end data-driven learning can produce precise extreme parkour on a low-cost robot with imprecise sensing and actuation.
Abstract
from arXiv · showhide
Humans can perform parkour by traversing obstacles in a highly dynamic fashion requiring precise eye-muscle coordination and movement. Getting robots to do the same task requires overcoming similar challenges. Classically, this is done by independently engineering perception, actuation, and control systems to very low tolerances. This restricts them to tightly controlled settings such as a predetermined obstacle course in labs. In contrast, humans are able to learn parkour through practice without significantly changing their underlying biology. In this paper, we take a similar approach to developing robot parkour on a small low-cost robot with imprecise actuation and a single front-facing depth camera for perception which is low-frequency, jittery, and prone to artifacts. We show how a single neural net policy operating directly from a camera image, trained in simulation with large-scale RL, can overcome imprecise sensing and actuation to output highly precise control behavior end-to-end. We show our robot can perform a high jump on obstacles 2x its height, long jump across gaps 2x its length, do a handstand and run across tilted ramps, and generalize to novel obstacle courses with different physical properties. Parkour videos at https://extreme-parkour.github.io/
1 Introduction
The paper frames extreme robot parkour as a problem of precise, tightly coupled perception and control under noisy hardware constraints. It addresses this with end-to-end reinforcement learning, dual distillation, and a unified reward design for diverse behaviors.
- Motivation: Extreme parkour demands precise perception and control because small errors during highly dynamic maneuvers can cause failure.The robot must anticipate obstacles and execute correct movements at the right time while operating near hardware limits.
- Motivation: Low-cost hardware introduces noisy, laggy actuation and perception artifacts, latency, and jitter that undermine terrain mapping and action planning.Even correct actions may fail when actuators execute them inaccurately or belatedly.
- Approach: A single neural network is trained with simulation RL to map depth images directly to motor commands without explicit mapping or planning.The framework uses an end-to-end data-driven policy operating from a single front-facing depth camera.
- Approach: Dual distillation first trains with privileged heading information, then teaches the policy to predict and rapidly adjust its own heading at deployment.This allows obstacle-dependent heading adjustments while retaining agile motor control from depth images.
- Approach: An inner-product reward design and automatic terrain curriculum enable one neural network to represent diverse parkour behaviors.The design targets general robot base motion acquisition while helping reinforcement learning overcome exploration challenges.
- Results: The system performs high jumps and long jumps at 2x the robot’s height and length, respectively, alongside handstands and tilted-ramp traversal.These behaviors are produced directly from a single front-facing egocentric depth camera.
2 Related Work
Prior work combines model-based controllers, elevation maps, or learned policies for locomotion and parkour. The paper positions its approach against methods that rely on task-specific composition, elevation maps, complex curricula, or privileged obstacle information.
- Legged locomotion: Classical locomotion systems combine model-based walking controllers with elevation maps built from point clouds and odometry.Such controllers can struggle when physical properties vary widely, including ice or deformable material.
- Learned locomotion: Reinforcement-learning locomotion methods adapt to changing dynamics and may also use elevation maps for perceptive walking.Related results include blind dynamic running and jumping on quadrupeds and bipeds.
- Comparison: The paper reports that its low-cost A1 reaches scenarios with obstacles twice the robot’s height and length.Table 1 compares parkour setups, with Climb and Gap expressing obstacle size relative to quadruped height and length.
- Concurrent parkour work: Concurrent parkour work composes task-specific policies with a high-level module but still relies on elevation maps.This differs from the paper’s direct depth-based policy framing.
- Concurrent parkour work: Another concurrent end-to-end depth policy uses a complex curriculum, hard-constraint distillation, and privileged obstacle abstractions.Its privileged visual information includes obstacle type, width, height, and robot distance to the obstacle.
3 Method
The method trains a unified parkour policy in simulation, then distills privileged perception and heading information into a deployable depth-based policy that selects its own direction. Unified rewards and staged training support diverse behaviors across challenging terrains.
- Training framework: A two-phase framework first trains locomotion with privileged environment, scandot, and waypoint-heading information, then distills it into a depth-based policy that predicts its own yaw.Regularized Online Adaptation estimates environmental properties from observation history during training.
- Unified reward: Waypoints placed on simulated terrains provide target directions, while world-frame velocity tracking prevents the robot from exploiting the reward by turning around obstacles.The desired direction is computed from the next waypoint and the robot’s world-frame position.
- Unified reward: A foot-clearance penalty discourages contacts within 5cm of terrain edges because edge-stepping is risky and transfers poorly to real settings.The penalty activates when a foot touches the ground near an edge.
- Unified reward: An optional forward-vector reward enables stylistic behaviors such as handstands, with its activation randomly sampled during training and remotely controlled at deployment.The handstand uses the desired direction ˆc = [0,0,−1]T.
- Terrain curriculum: The curriculum trains on tilted ramps, gaps, hurdles, and high steps arranged by difficulty, promoting or demoting episodes according to traversal distance.Robots begin in easy levels and move between levels based on whether they traverse more than half the expected distance.
- Emergent behaviors: The resulting robot behavior includes a high jump twice its height, using rear-leg propulsion, front-foot clearance and pulling, rear-leg tucking, and stabilization.The behavior emerges from the learned policy rather than manually specified motion rules.
4.1 Experimental Setup
The experiments use a 12-joint Unitree A1 with a front-facing RealSense depth camera, running depth processing at 10±2Hz and the base policy at 50Hz.
- Robot platform: The platform is a 12-joint Unitree A1 whose thigh joint is 26cm high when standing and whose body length is 40cm.These dimensions define the robot’s physical scale for the parkour experiments.
- Perception and computation: A front-facing Intel RealSense D435 provides exteroception at 10±2Hz, while depth processing runs at 10Hz and the base policy at 50Hz on a Jetson NX.The depth server passes latent features and target direction to the base policy over UDP.
4.2 Emergent results
The robot exhibits emergent parkour behaviors, including optimized high and long jumps and transitions between quadrupedal and handstand locomotion. These behaviors arise from learned control rather than manually specified movement sequences.
- High jump: 0.5m: The robot jumps onto a gym box twice the height of its hip joint.The maneuver requires highly optimized behavior.
- High jump: The high-jump sequence reduces stride length, aligns the feet, propels upward with the rear legs, clears the obstacle with the front feet, and stabilizes into walking.The robot pulls itself up with its front feet before tucking its rear legs and shifting to a stable pose.
- Long jump: Figure 5 presents keyframes of a long jump spanning 2x the robot length.The keyframes correspond to the long-jump behavior described in the accompanying results.
- Long jump: 0.8m: The robot jumps across a gap twice the separation between its front and rear feet.It moves both foot pairs near the edge, kicks forward and upward, and extends its legs to maximize force application and reach.
- Handstand: The robot transitions between quadrupedal walking and handstand walking on its front two legs.It shifts weight forward, kicks its rear legs upward, and makes small adjustments to maintain the vertical pose.
4.3 Comparison to Baselines
The method outperforms the evaluated baselines in simulation and real-world trials, while its learned direction prediction approaches oracle performance. It also shows robust handstand walking across varied terrain using proprioception alone.
- Simulation results: Our method outperforms the evaluated baselines on both mean displacement and mean edge violation in simulation.NoInner, NoClear, and Noisy exhibit specific failure modes involving obstacle collisions, unstable foot placement, or noisy terrain sensing.
- Simulation results: Ours approaches the oracle upper bound and achieves greater x displacement than Both and Mask in the distillation comparison.Both and Mask fail to converge to low imitation losses because their training data distribution drifts from the teacher.
- Real-world results: NoClear places feet near gap or cliff edges to minimize energy usage, producing unstable behavior that sometimes causes falls.The clearance reward is intended to discourage this risky foot placement.
- Real-world results: The handstand policy remains robust across indoor and outdoor terrain and walks down stairs using proprioception alone.It is trained without exteroception.
5 Discussion
The paper demonstrates that an end-to-end data-driven approach can perform precise extreme parkour despite imprecise sensing and actuation. The authors identify object manipulation with mobile manipulators as a future direction.
- Discussion: An end-to-end data-driven approach scales to precise extreme parkour on a robot with imprecise sensing and actuation.The authors attribute this possibility to a unified, general reward structure that allows emergent behavior.
- Discussion: Extending the approach to mobile manipulators is identified as a promising direction for enabling object manipulation.The current robot is described as expert at moving around the world but not at manipulating objects.