PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching

Zhen Wu, Xiaoyu Huang, Lujie Yang, Yuanhang Zhang, Xi Chen, Pieter Abbeel, Rocky Duan, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, C. Karen Liu

arXiv:2602.15827v2cs.ROcs.AIcs.LGeess.SY

TL;DR

Humanoid parkour requires agile, adaptive motion composition and perception beyond stable locomotion. PHP composes retargeted human skills with motion matching, trains privileged motion-tracking experts, and distills them into a depth-based student using DAgger and RL. The framework supports autonomous long-horizon parkour and zero-shot deployment, while remaining limited by sensing and hardware constraints.

  • Problem

    Humanoid robots must compose scarce, highly dynamic human skills and adapt them to complex terrain using perception.

  • Method

    PHP uses motion matching for long-horizon skill composition, then distills motion-tracking experts into a depth-based multi-skill policy with DAgger and RL.

  • Results

    PHP achieves zero-shot sim-to-real transfer and dynamic parkour over diverse obstacles, including traversal supported by dense motion-matching data and RL-augmented distillation.

  • Takeaways & Limitations

    Dense motion matching supplies varied approach conditions, while RL adds success-driven correction beyond per-step imitation for highly dynamic skills.

  • Takeaways & Limitations

    Real-world capability is constrained by short-range narrow-field-of-view sensing and hardware lacking strong hands or grippers for more extreme climbing.

Abstract

from arXiv · show

While recent advances in humanoid locomotion have achieved stable walking on varied terrains, capturing the agility and adaptivity of highly dynamic human motions remains an open challenge. In particular, agile parkour in complex environments demands not only low-level robustness, but also human-like motion expressiveness, long-horizon skill composition, and perception-driven decision-making. In this paper, we present Perceptive Humanoid Parkour (PHP), a modular framework that enables humanoid robots to autonomously perform long-horizon, vision-based parkour across challenging obstacle courses. Our approach first leverages motion matching, formulated as nearest-neighbor search in a feature space, to compose retargeted atomic human skills into long-horizon kinematic trajectories. This framework enables the flexible composition and smooth transition of complex skill chains while preserving the elegance and fluidity of dynamic human motions. Next, we train motion-tracking reinforcement learning (RL) expert policies for these composed motions, and distill them into a single depth-based, multi-skill student policy, using a combination of DAgger and RL. Crucially, the combination of perception and skill composition enables autonomous, context-aware decision-making: using only onboard depth sensing and a discrete 2D velocity command, the robot selects and executes whether to step over, climb onto, vault or roll off obstacles of varying geometries and heights. We validate our framework with extensive real-world experiments on a Unitree G1 humanoid robot, demonstrating highly dynamic parkour skills such as climbing tall obstacles up to 1.25m (96% robot height), as well as long-horizon multi-obstacle traversal with closed-loop adaptation to real-time obstacle perturbations.

I. INTRODUCTION

PHP addresses the challenge of composing scarce, highly dynamic human motions into perceptive, long-horizon humanoid parkour. It combines motion matching, expert RL, and distillation into a single depth-based policy for autonomous obstacle traversal.

  • Humanoid parkour requires dynamic whole-body control, visual adaptation to perturbations, and consolidation of many skills into one visuomotor policy.
  • Scarce human-motion datasets typically provide only one or two demonstrations per skill, each lasting a few seconds.
  • Motion matching densifies sparse motion libraries by stitching fragments into diverse, obstacle-adaptive, long-horizon reference trajectories.
  • PHP retargets human motions, composes them with motion matching, trains motion-tracking experts, and distills them into one depth-conditioned policy.
  • The framework targets autonomous traversal of complex terrain using onboard perception and simple velocity commands, including dynamic obstacle courses.

B. Humanoid Skill Chaining with Human Motion Data

Human motion data provides agile behavior but makes skill chaining difficult because heterogeneous motions occupy disjoint state-space regions. PHP uses motion matching to retrieve compatible frames and compose locomotion with dynamic parkour skills.

  • Human motion references reduce reward engineering and produce agile behaviors, but heterogeneous styles make long-horizon composition difficult.
  • Prior approaches include implicit transitions, specialist switching, learned style rewards, and generated kinematic trajectories, but their humanoid demonstrations remain limited or degrade in low-data regimes.
  • PHP applies this mechanism offline to compose locomotion and atomic parkour skills into long-horizon trajectories for policy learning.
  • Motion matching retrieves motion frames whose features best match the current pose and desired future behavior, then stitches them into references.
  • The method periodically performs nearest-neighbor search within a skill-specific window and blends transitions to avoid discontinuities.

2) Long-Horizon Parkour Trajectory Synthesis:

Long-horizon trajectories connect parkour clips through reusable locomotion segments and diversify their entry conditions, commands, and terrain geometry. This produces varied references while preserving meaningful contact-rich skill execution.

  • PHP composes trajectories as Locomotion → Parkour Skill → Locomotion, using locomotion as a reusable connector between heterogeneous skills.
  • Each skill has annotated start and end frames plus a pre-skill entry window representing the meaningful approach phase.
  • During locomotion, motion matching selects frames; during a skill, the clip plays sequentially, then matching resumes after the annotated endpoint.
  • Reference trajectories vary speed, turning direction, duration, obstacle geometry, pose, and yaw to expose policies to diverse conditions.
  • Motion matching varies approach distances and stride phases, producing distinct skill entrances such as different leading legs.

C. Learning a Highly-Dynamic Visuomotor Policy

PHP trains skill-specific motion-tracking experts with privileged observations and distills them into a unified perceptive policy. The formulation uses common training components without motion-specific tuning.

  • The target is one perceptive policy that selects parkour skills from obstacles using a commanded target velocity.
  • Skill-specific experts are trained for motion quality before distillation into a single visuomotor policy.
  • The formulation follows shared tracking rewards, termination rules, randomizations, and action scaling rather than motion-specific tuning.
  • Experts receive reference states, robot state, and terrain scans, while privileged global pelvis information supports recovery behaviors.
  • Adaptive Sampling prioritizes frequently failing regions because difficult skills such as high-wall climbing may otherwise fail to converge.

2) Distilling a Unified Student Policy with DAgger and RL:

The unified student combines DAgger imitation with PPO to execute highly dynamic skills while addressing multimodality, exploration, and sim-to-real training issues.

  • RL-Augmented Distillation: DAgger alone is insufficient for climbing and vaulting because per-step imitation does not favor brief, high-magnitude torque bursts.These skills depend on episode outcomes and decisive actions that may span only a few timesteps.
  • RL-Augmented Distillation: PPO is combined with DAgger through a curriculum, using success-driven feedback to encourage expert-like high-torque actions.The objective is L = λPPO LPPO + λD LD, with λPPO + λD = 1.
  • Perceptive Student Policy: The student uses proprioception, onboard depth images, velocity commands, and randomized camera and depth conditions for robust visuomotor training.Camera extrinsics are randomized within 2.5 cm translation and 2.5° rotation around the calibrated value.
  • Training Curriculum: A warmup curriculum shifts training from DAgger toward PPO because noisy early PPO gradients can undermine distillation.The curriculum linearly tunes down λD during the first half of training, capped at 0.1.
  • Multimodal Skill Execution: The method relaxes termination from 0.5 m for experts to 1 m so mirrored skill executions are not terminated prematurely.Left-right multimodality can produce valid mirrored executions with large tracking error relative to a single reference trajectory.

IV. EXPERIMENTS

Experiments on a Unitree G1 evaluate autonomous dynamic parkour, individual skills, and long-horizon composition using only simple velocity commands for navigation.

  • Experimental Setup: The system is evaluated on a 1.3 m-tall Unitree G1 humanoid with 29 DoFs using large-scale parallel simulation training.The policies use a 3-layer CNN, a 5-layer MLP, 16,384 parallel environments, and 20K training iterations.
  • Experimental Setup: All real-world skill execution is autonomous, with only simple 2D velocity commands provided for navigation.Tasks require highly dynamic individual skills, long-horizon composition, and adaptation to environmental changes.
  • Human-Level Agility: The robot demonstrates highly dynamic parkour skills, including a direct comparison with a human on a challenging high-wall climb.The comparison evaluates whether the robot can perform the same maneuver at a comparable pace.
  • Human-Level Agility: 3.63 s: the robot climbs onto a 1.25 m wall, equal to 96% of its height, measured from toe-off.The sequence includes toe-off, pull-up, swing, and stable stand with closely matched timing across key events.
  • Additional Parkour Skills: The robot clears a 0.4 m-high, 0.5 m-long obstacle within 0.8 s while covering more than 2 m forward.The motion reaches 3.41 m/s peak forward speed and 2.53 m/s average speed; it also demonstrates a drop landing from a 1.25 m platform.

2) Multi-Obstacle Course:

Motion-matching composition enables the policy to generalize from single-obstacle training to multi-obstacle parkour, while baseline comparisons expose the value of composition and expert distillation.

  • Multi-Obstacle Course: Motion matching synthesizes long-horizon references that chain skills through shared locomotion segments and vary approach distances and timings.This allows training on single-obstacle traversal to support complex multi-obstacle courses.
  • Multi-Obstacle Course: The policy composes stepping and low- and high-wall climbs online, switching skills smoothly during continuous runs.It adapts approach and maneuver timing when multiple obstacles are randomly displaced by approximately 0.5 m.
  • Evaluation Protocol: Evaluation measures success rate across commanded speeds of 1.0 m/s or 2.0 m/s, obstacle heights, and randomized starting distances.Each task uses 100 obstacle instances and 20° yaw randomization.
  • Baseline Comparison: The velocity-tracking baseline succeeds on a 36 cm obstacle but fails on higher obstacles, relying largely on foot-only stepping rather than arm-supported climbing.This result highlights the limitation of reward shaping alone for highly dynamic humanoid parkour.
  • Baseline Comparison: Uncomposed motion data performs poorly despite atomic skills because the policy does not experience skill transitions or prepare for upcoming obstacles during walking.Motion matching instead provides coherent transitions and diverse visual contexts during skill execution.
  • Baseline Comparison: End-to-end depth training handles low obstacles but degrades on harder tasks, whereas expert distillation achieves substantially higher success across obstacle heights.The comparison suggests that training from scratch makes exploration difficult for highly dynamic skills.

3) Ablation Study:

Ablations show that dense motion-matching coverage, scalable distillation, and RL during distillation each contribute to reliable performance on difficult parkour skills.

  • Motion Matching Density: Extreme-distance motion data reduces success across tasks because the policy fails to generalize to intermediate distances where contact timing is critical.Half-density data generally lowers success on harder skills, especially when samples are skewed toward one end of the distance range.
  • Training Scalability: Increasing parallel environments or network depth generally improves success, especially on more challenging parkour tasks.The distillation framework scales favorably with rollout throughput and model capacity.
  • RL in Distillation: RL is critical for effective distillation: the DAgger-only student shows a clear performance drop and stalls during the pull-up phase on a 76 cm obstacle.The decisive torque burst spans only a few timesteps and is weakly penalized by per-step imitation loss.
  • RL in Distillation: Root tracking or an alive reward achieves success rates comparable to whole-body tracking on difficult skills when co-trained with DAgger.RL primarily supplies a success-driven exploitation signal that compensates for DAgger’s action underestimation.
  • RL in Distillation: Dropping DAgger after the curriculum and continuing with pure RL often produces jittery, unnatural behaviors.The authors interpret behavior cloning as critical regularization for RL in the high-dimensional action space.

V. CONCLUSION

PHP combines motion-matching skill composition with a teacher-student RL pipeline to enable perception-driven, long-horizon humanoid parkour. The framework demonstrates agile, adaptive whole-body behavior but remains constrained by semantic perception and hardware limitations.

  • PHP enables humanoid robots to execute long-horizon, highly dynamic parkour autonomously using onboard perception.
  • Motion matching provides coherent long-horizon references and varied approach conditions, while RL-augmented distillation transfers capabilities to a depth-based multi-skill student.
  • Zero-shot deployment on a Unitree G1 demonstrates agile, adaptive, whole-body parkour in real-world experiments.
  • The pipeline lacks semantic scene understanding and is constrained by short-range sensing, narrow camera field of view, and insufficient hand or gripper strength.These constraints can force commitment under perceptual ambiguity and prevent testing extreme climbing or hanging maneuvers.

APPENDIX

The appendix constructs motion-matching queries from robot kinematics and commanded motion, then predicts future trajectory and heading features through critically damped spring models. These features support nearest-neighbor retrieval of motion frames in a local coordinate frame.

  • Each retargeted motion frame stores robot configuration qi and a 27-dimensional matching feature xi derived from it.The configuration includes root translation, root quaternion, and 29 joint angles.
  • The matching feature combines future root trajectory, local foot positions and velocities, and root velocity in local coordinates.
  • At runtime, the query feature combines kinematic features from the current configuration with a future trajectory generated from the 2D velocity command.
  • A critically damped spring converts commanded planar velocity and heading into future root positions and facing directions.Planar positions come from integrating the spring-generated velocity, while heading is evaluated directly at future times.
  • The future trajectory feature is evaluated at 0.33, 0.67, and 1.0 seconds and transformed into the character’s local coordinate frame.

3) Transition Smoothing via Inertialization:

The framework smooths motion switching with inertialization and uses a library spanning locomotion and atomic parkour skills. Training also specifies expert-derived rewards, randomized depth sensing, and relaxed student termination conditions.

  • Inertialization preserves continuity when switching to a newly retrieved frame by applying and then decaying a transition offset.The offset decays toward zero using the same critically damped spring model.
  • The motion library provides a shared locomotion transition manifold covering commanded speeds from 0.8 to 3.5 m/s.Most parkour skills use 1.0 or 2.0 m/s, with one 3.0 m/s cat-vault skill.
  • Expert policy reward formulations and domain-randomization settings are summarized in Tables IV and V.
  • Relaxed student termination conditions improve PPO stability but may expose the student to states outside the expert policies’ training distribution.
  • Depth training adds a random offset within ±3 cm and Gaussian noise with 3 cm standard deviation, while the onboard camera operates at 30 Hz.

C. Details for Baselines

The baseline section compares handcrafted reward shaping and AMP against the proposed setting, while documenting the associated terrain curriculum and implementation references. AMP tracks walking and commanded velocity but performs poorly on obstacle traversal.

  • 1) Velocity Tracking Baseline: The velocity-tracking baseline learns locomotion from handcrafted rewards and a terrain curriculum without motion imitation or human reference trajectories.
  • Tables III–VI provide motion-clip, reward, domain-randomization, and training-hyperparameter references.
  • 1) Velocity Tracking Baseline: The terrain curriculum increases difficulty from 0.3 m to 1.0 m over 10 levels with a 2 m run-up.
  • 2) AMP Baseline: AMP is included as a baseline for chaining skills with human reference data, following the original authors’ MimicKit implementation.
  • 2) AMP Baseline: AMP walks stably and tracks commanded velocity but fails on most obstacle-traversal tasks, especially harder ones.The text also notes that AMP performance depends on implementation details and tuning.
Loading 2602.15827v2…