Source-linked AI summary
Deep Whole-body Parkour
Ziwen Zhuang, Shaoting Zhu, Mengjie Zhao, Hang Zhao
TL;DR
Humanoid control traditionally separates terrain-aware locomotion from general motion tracking that lacks environmental adaptation. This paper unifies them by integrating exteroceptive depth into whole-body tracking, producing a single policy that performs multiple dynamic motions across varied terrains and tolerates substantial initialization variation. The framework supports robust multi-contact behaviors beyond simple walking or running, while the authors identify data volume and autonomous skill selection as remaining challenges.
Problem
Existing perceptive locomotion handles terrain but is limited to pedal gaits, whereas general motion tracking reproduces complex skills without reacting to terrain geometry.
Method
The framework integrates exteroceptive depth perception into whole-body motion tracking and trains a single policy on motion-terrain pairs across varied environments.
Results
The policy performs dynamic multi-contact motions, including vaulting and dive-rolling, across multiple terrains and tolerates varied initialization through visual feedback.
Takeaways & Limitations
Perceptive whole-body control provides a baseline for humanoids to perform contact-rich behaviors in unstructured environments beyond simple pedal locomotion.
Takeaways & Limitations
The framework still faces challenges in data volume and autonomous skill selection.
Abstract
from arXiv · showhide
Current approaches to humanoid control generally fall into two paradigms: perceptive locomotion, which handles terrain well but is limited to pedal gaits, and general motion tracking, which reproduces complex skills but ignores environmental capabilities. This work unites these paradigms to achieve perceptive general motion control. We present a framework where exteroceptive sensing is integrated into whole-body motion tracking, permitting a humanoid to perform highly dynamic, non-locomotion tasks on uneven terrain. By training a single policy to perform multiple distinct motions across varied terrestrial features, we demonstrate the non-trivial benefit of integrating perception into the control loop. Our results show that this framework enables robust, highly dynamic multi-contact motions, such as vaulting and dive-rolling, on unstructured terrain, significantly expanding the robot's traversability beyond simple walking or running. https://project-instinct.github.io/deep-whole-body-parkour
I. INTRODUCTION
Humanoid control is split between terrain-adaptive locomotion and environment-agnostic motion tracking. The proposed framework integrates exteroceptive depth perception with whole-body tracking so robots can adapt agile, contact-rich motions to terrain and initialization.
- Motivation: Perceptive locomotion adapts gait to unstructured terrain but remains limited to foot-contact pedal motions.These systems use proprioceptive and exteroceptive observations for terrain adaptation, while the upper body is used mainly for balance.
- Framework: The data-driven framework aligns environment scans with human demonstrations to generate motion-terrain pairs for reinforcement learning.The resulting policy uses exteroceptive observations to replicate agile behaviors in the real world.
- Motivation: General motion tracking reproduces skills such as backflips, martial arts, and vaults but primarily follows pre-recorded trajectories without terrain awareness.This environment-agnostic behavior can fail when obstacle geometry differs from the reference setting.
- Proposed approach: The framework integrates exteroceptive depth perception directly into whole-body motion tracking.The policy adapts reference motion using visual occupancy rather than merely replaying an animation.
- Proposed approach: Visual feedback lets the policy tolerate varied initial distances and angles by adjusting its approach for accurate hand placement and vaulting.The resulting closed-loop behavior replaces fragile fixed-trajectory tracking with spatially aware interaction.
III. METHOD
The method builds aligned motion-terrain training data from human parkour demonstrations and reconstructed scenes, then trains perceptive whole-body control in massively parallel simulation. A grouped ray-casting system isolates each agent’s dynamic objects while sharing static terrain, accelerating depth rendering.
- Dataset curation: Human parkour motion is captured with optical motion capture while physical obstacles are digitized with LiDAR to align trajectories and reconstructed terrain.The paired motion and mesh preserve spatial correspondence between demonstrations and contact geometry.
- Dataset curation: Captured motions are retargeted to the Unitree G1 with kinematic filtering and manual keyframe adjustment to enforce feasible contacts.The process also removes high-frequency capture noise and respects the robot’s morphology.
- Environment generation: Scanned scenes are segmented into canonical obstacle meshes and procedurally randomized as motion-terrain pairs in NVIDIA Isaac Lab.Removing laboratory context encourages conditioning on local obstacle geometry rather than fixed room layouts.
- Parallel sensing: The training framework requires high-throughput depth simulation across thousands of isolated environments, which standard sensor rendering does not adequately provide.The implementation must prevent robots from perceiving other agents’ objects in shared physics scenes.
- Parallel sensing: 10× rendering speedup comes from precomputing collision-group mappings so each ray checks only shared static meshes and its agent-specific dynamic subset.The grouped strategy avoids the naive O(N) scan over all scene meshes.
B. Training settings
Training combines depth images, proprioception, future reference motion, relative-frame tracking rewards, and adaptive failure-focused sampling. A relative frame separates planar and heading alignment from reference height and tilt, while stuck detection removes unsolvable rollouts.
- Rewards: Tracking rewards combine local link position and rotation terms with global link velocity terms, while penalties constrain actions, joints, contacts, and torque.The reward design tracks whole-body motion in relative and world frames.
- Relative Frame: The relative frame uses the robot’s x-y position and yaw with the reference motion’s height, roll, and pitch.This frame supports local pose tracking while preserving the robot’s current planar alignment and heading.
- Network inputs: The policy receives future joint and base references, one noised depth image, and eight frames of historical proprioception.The critic additionally receives privileged simulator information, including actual key-link states and height scans.
- Adaptive Sampling: Adaptive sampling divides variable-length motions into temporal bins and increases sampling weight for bins associated with failures.Failure counts are smoothed before updating the sampling distribution, creating a failure-based curriculum.
- Stuck Detection: Stuck detection truncates episodes when randomized initialization places the robot in unsolvable terrain configurations for too long.This prevents large portions of rollout steps from becoming useless during training.
C. Bridging sim-to-real gap in depth perception
Real-world deployment uses 50Hz RealSense depth, but onboard filtering is replaced with GPU-based inpainting because the built-in Jetson CPU filtering does not meet the required rate. Noise and artifacts are added during training to address real-world sensing irregularities.
- Depth deployment: 50Hz RealSense depth is used at deployment, while GPU-based OpenCV inpainting replaces the built-in filter that cannot meet the Jetson CPU rate requirement.Training adds Gaussian noise and patched artifacts to account for reflection, motion blur, and stereo errors.
D. Deployment
Deployment uses an onboard depth-sensing inference system with separate processing for perception, policy execution, and logging. The policy receives motion references and relative orientation information without requiring global odometry.
- D. Deployment: ONNX accelerates the neural network onboard, while a state machine selects the motion reference during deployment.Depth sensing runs at 50Hz, and the policy is trained as a single depth-based neural network.
- D. Deployment: ROS bag recording runs in a separate process with CPU binding to reduce interference with neural-network execution.
- D. Deployment: The policy uses joint references and IMU-based relative rotation without requiring global robot-position odometry.The motion reference is provided relative to a reference frame rather than through global position error.
- D. Deployment: GPU-based OpenCV inpainting and simulated depth-image noise address deployment artifacts and help bridge the sim-to-real gap.
IV. EXPERIMENTS
The experiments evaluate whether exteroception improves agile whole-body control across multiple motions and terrains, while also examining drawbacks, distractor robustness, and sim-to-real deployment factors.
- IV. EXPERIMENTS: Four motions—kneel climb, roll vault, dive roll, and jump sit—are evaluated across a triangular barrier, a wooden box, and flat terrain.The wooden box measures 0.5m × 0.6m × 0.4m.
- IV. EXPERIMENTS: The study first tests whether the complete perceptive parkour pipeline can be deployed successfully on the real robot.
- IV. EXPERIMENTS: The experiments ask what depth vision contributes, whether it introduces drawbacks or handles unseen trivial distractors, and what supports odometry-free sim-to-real deployment.
A. Real world experiment
The real-world evaluation deploys the perceptive parkour system on a Unitree G1 using onboard depth sensing and CPU inference. Tests cover indoor and outdoor environments, with outdoor starts requiring no further calibration.
- A. Real world experiment: The system runs on a 29-DOF Unitree G1 with an Intel RealSense D435i, ROS2, and onboard CPU-based ONNX inference.A simple walking process maintains balance when the perceptive parkour policy is inactive.
- A. Real world experiment: Indoor tests record running logs over a network cable, whereas outdoor tests use only the onboard computer.
- A. Real world experiment: Outdoor experiments place the robot in front of the obstacle without determining the motion start point or performing additional calibration.
B. Non-trivial benefit of depth vision
Depth-guided tracking is tested for convergence and initialization robustness. The evaluation combines position-variance analysis with success-rate mapping over a broad starting region.
- B. Non-trivial benefit of depth vision: Training variants are compared across four motions using MPJPEg and MPJPEb.
- B. Non-trivial benefit of depth vision: Robots initialized across a 0.5m × 0.5m region converge toward the target obstacle and finish the motion-tracking task.Training randomizes starts within 0.3m × 0.3m around the reference motion.
- B. Non-trivial benefit of depth vision: Position variance in x-y coordinates drops sharply during convergence, then increases near the final stage because of common visual features from the depth camera.
- B. Non-trivial benefit of depth vision: 100% success is maintained across a 1.2m × 1.2m initialization range despite training randomization limited to −0.15m to +0.15m in x and y.Success decreases only at the experiment boundary.
C. Vision robustness
The study evaluates depth-enabled motion tracking in unseen scenes containing distractors and finds that the policy preserves task success across distractor variants, with specific objects increasing MPJPE.
- The evaluation introduces unseen simulator scenes containing distractors while keeping the reference motions unchanged and minimizing obstruction of their dynamics.
- Depth vision supports positional convergence before scene interaction, which is important because a 0.4m misplacement can cause catastrophic failure for roll-vaulting and kneel-climbing.
- The system tests wide, plane, and wall distractors, including cases that present different visual inputs without substantially disrupting the original trajectory.
- No drop in success rate is observed across the tested distractor variants when running the same motion-tracking system.
- Plane distractors significantly increase MPJPE, while wide distractors also increase it because they alter motion dynamics or disrupt localization.
D. Ablation on the training recipe
The ablation examines depth input, initialization randomization with stuck detection, and relative-frame rewards, showing that each component contributes to robust interaction-oriented tracking.
- The training recipe combines depth input, randomized initialization with stuck detection, and local tracking rewards defined in the relative frame.
- Without exteroception, motion tracking is not robust to initial-position perturbations and requires exact placement to avoid catastrophic scene-interaction failure.
- Relative-frame rewards improve global performance in interaction-intensive cases compared with using only local-frame rewards, despite less accurate local tracking.
- Stuck detection prevents large portions of rollout steps from becoming useless data samples; removing it slightly increases MPJPE while the full trajectory still succeeds.
V. CONCLUSION
The paper presents a unified, end-to-end paradigm that combines human demonstrations with onboard sensing for agile humanoid interactions. It reports depth-enabled robustness to distractors and successful deployment across diverse indoor and outdoor scenarios, while identifying data volume and autonomous skill selection as remaining challenges.
- The framework unifies whole-body motion tracking with perceptive control so humanoids learn agile, physics-based interactions from human demonstrations and onboard sensing.
- Depth vision provides non-trivial robustness against environmental distractors, and the system is validated through successful deployment in diverse indoor and outdoor real-world scenarios.
- End-to-end exteroceptive training provides infrastructure for scaling humanoid whole-body control and building extensive libraries of motion skills with intensive scene interaction.
- Data volume and autonomous skill selection remain challenges for future research.