PaperlayerView exact revision on arXiv

Source-linked AI summary

MulDP: Multimodal Diffusion Policy for Autonomous Quadruped Parkour Navigation across Complex Terrains

Kangmai Hu, Yueqi Zhang, Peng Zhai, Xiaoyi Wei, Jiabin Hu, Zhixiang Liu, Quancheng Qian, Lihua Zhang

arXiv:2609.03984v1cs.RO

TL;DR

Autonomous quadruped parkour navigation remains underexplored, with challenges in velocity regulation, anticipatory behavior, and perception–control coupling. MulDP combines multimodal observations and diffusion-generated command horizons, supported by QPND. Across evaluated scenarios, it reports average success rate of 89.7%, average SPL of 71.9%, and average time to reach of 4.9 s.

  • Problem

    Autonomous quadruped parkour navigation remains underexplored, while demanding fine-grained velocity control, long-horizon anticipation, and perception–control coupling.

    Source: §I · p. 1

  • Method

    MulDP fuses historical and current visual observations, proprioception, and goal information to iteratively generate temporally coherent navigation commands, supported by the QPND simulation dataset.

    Source: §I · p. 2, C. Diffusion Policy · p. 4

  • Results

    Across evaluated scenarios, MulDP achieves average SR of 89.7%, average SPL of 71.9%, and average TTR of 4.9 s.

    Source: 1) Comparison Settings · p. 5

  • Takeaways & Limitations

    MulDP demonstrates the ability to identify effective planning solutions and guide the robot through complex environments across the evaluated scenarios.

    Source: 1) Comparison Settings · p. 5

Abstract

from arXiv · show

Quadruped robots have demonstrated impressive agility in parkour locomotion across complex terrains. However, most systems still rely on human intervention for high-level planning, and autonomous parkour navigation remains underexplored. The key challenges include fine-grained velocity regulation, long-horizon anticipatory behaviors, and tight coupling between perception and embodied execution. To address these challenges, we propose a Multimodal Diffusion Policy (MulDP) that integrates visual perception with robot proprioception and goal information to generate temporally coherent and anticipatory navigation velocity commands, tightly coupling perception with embodied control to enable robust autonomous navigation. To support the training of MulDP, we construct the first Quadruped Parkour Navigation Dataset (QPND), a multimodal dataset that encompasses diverse navigation behaviors and complex terrains. Extensive simulation and real-world experiments demonstrate that MulDP enables robust long-horizon autonomous navigation and effective traversal across complex terrains.

I. INTRODUCTION

Autonomous quadruped parkour navigation remains underexplored despite advances in parkour locomotion. MulDP combines multimodal conditioning and a navigation dataset to address the demands of long-horizon, terrain-aware control, with evaluations in simulation and the real world.

I. INTRODUCTION

  • Existing quadruped parkour work largely relies on human teleoperation, leaving autonomous navigation in complex environments insufficiently explored.Source: §I · p. 1
  • Parkour navigation requires fine-grained velocity modulation and anticipatory control, since crossing a wide gap can require accelerating before reaching its edge.Visual traversability alone also does not guarantee physical feasibility given the robot’s body geometry and locomotion constraints.Source: §I · pp. 1–2
  • MulDP formulates navigation velocity generation as conditional diffusion to address multimodality, long-horizon dependencies, and perception–control coupling.It is also proposed as a local planner compatible with global planners.Source: §I · p. 2
  • QPND is a multimodal simulation dataset collected under sensing constraints, using onboard perception and data augmentation to improve diversity and facilitate sim-to-real transfer.Source: §I · p. 2
  • The authors report extensive evaluations of MulDP’s effectiveness and robustness in both simulation and real-world experiments.Source: §I · p. 2

A. Robot Diffusion Policy

Diffusion policies can generate action sequences through iterative denoising, but quadruped parkour navigation poses temporal and embodied demands that existing methods do not fully address. MulDP combines multimodal inputs with diffusion-based velocity generation to couple perception and locomotion.

A. Robot Diffusion Policy

  • Diffusion policies generate action sequences by iterative denoising conditioned on visual observations and robot states, but parkour application is constrained by scarce multimodal datasets and insufficient adaptation to temporal and embodied demands.Prior diffusion-policy work spans manipulation, trajectory planning, data generation, dexterous manipulation, and human–robot interaction [17], [21], [22], [23], [24].Source: A. Robot Diffusion Policy · p. 2
  • MulDP addresses these demands with encoders for historical perception, fine-grained spatial representation, and goal conditioning, alongside an Isaac Sim dataset using noise injection and data augmentation.Source: A. Robot Diffusion Policy · p. 2

B. Visual Navigation Models

  • Waypoint-oriented visual navigation methods often output headings or goal-relative vectors, which are insufficient for legged locomotion requiring fine-grained velocity modulation and anticipatory control.Such methods also often treat quadruped navigation as obstacle avoidance without explicitly leveraging locomotion for terrain traversal.Source: B. Visual Navigation Models · p. 2
  • MulDP fuses visual perception with proprioception and other multimodal signals, then uses generative diffusion to produce dynamically feasible, temporally coherent velocity commands.The architecture fuses historical depth images, proprioception, the current depth frame, and goal information; it generates a command horizon and replans at each 5 Hz update after sending the first command to the locomotion policy.Source: B. Visual Navigation Models · p. 2, Figure 2 · p. 3

A. Overview

MulDP uses depth, proprioception, and decision memory to drive locomotion through a generative diffusion policy. QPND is generated from onboard sensing in simulation and includes varied terrain and navigation behaviors.

A. Overview

  • The system combines depth images, proprioception, and decision memory to control locomotion using a generative diffusion policy conditioned on current and historical sensory inputs.The overview describes QPND and the system modules as supporting real-time, precise navigation in complex environments.Source: A. Overview · p. 3

B. Data Generation

  • The dataset uses forward-facing depth images, proprioception, and control commands, without requiring additional sensing such as LiDAR or privileged local 3D information.This non-privileged sensing setup is intended to preserve interface validity and facilitate sim-to-real transfer.Source: B. Data Generation · p. 3
  • The simulated scenes include traversable gaps, hurdles, and stairs as well as impassable obstacles, broadening the terrain and solution diversity.The specified ranges are gaps of 0.4–0.7 m, hurdles of 0.4–0.5 m, and stair steps of 0.15–0.2 m; impassable obstacles are 0.3–1.0 m wide and up to 1.0 m high.Source: B. Data Generation · p. 3
  • QPND contains 6,400 trajectories, approximately 250,000 depth images, and 31.0 km of navigation, including multiple solutions for the same terrain.Basic terrains use scripted collection from randomly sampled start–goal pairs; impassable obstacles use expert teleoperation to cover multimodal traversal strategies.Source: B. Data Generation · p. 3
  • Data augmentation segments trajectories into contiguous motion samples, increasing data utilization by nearly 50% while preserving expert behaviors.Source: B. Data Generation · p. 4
  • Sensor augmentation crops and rescales depth images, adds random and Gaussian depth noise, and adds noise to proprioception to address deployment sensing noise.The stated depth noise includes 1% random noise and Gaussian noise with variance 0.01.Source: B. Data Generation · p. 4

C. Diffusion Policy

MulDP encodes temporal sensory history, current spatial detail, and decision memory into context for a diffusion policy. The policy iteratively denoises a future command horizon and executes only its first command before replanning.

C. Diffusion Policy

  • The Multimodal Temporal Encoder jointly embeds historical depth-image patches and proprioception, using cross-modal attention to produce fused conditioning features.The passage specifies 154 visual tokens from 44×56 images and 45-dimensional proprioception, producing 128-dimensional fused features.Source: C. Diffusion Policy · p. 4
  • A parallel Current Spatial Encoder processes the current depth image with a shallow CNN to preserve fine-grained terrain structure for immediate perception.It complements historical features when abstract temporal features alone are insufficient for safe navigation around sudden obstacles.Source: C. Diffusion Policy · p. 4
  • The Decision Memory Encoder stores the last five planning-control and goal-vector steps, representing their geometric relationship to support smooth, goal-oriented commands.Source: C. Diffusion Policy · p. 4
  • The policy predicts a horizon of future velocity and heading commands, conditioning denoising on features from the temporal, spatial, and decision-memory encoders.The command horizon is represented as H×2 commands, each containing changes in velocity and heading.Source: C. Diffusion Policy · p. 4
  • The noise-prediction network conditions on observation context and uses 10 denoising steps to balance output accuracy and inference speed.The denoising process progressively reduces noise to produce the noise-free output horizon.Source: C. Diffusion Policy · p. 4
  • At deployment, only the first command of the predicted horizon goes to the locomotion policy; the horizon is regenerated at the next 5 Hz update.Source: C. Diffusion Policy · p. 4

D. Model Training

MulDP is trained on QPND with an end-to-end supervised diffusion loss. The training conditions on expert trajectories, depth images, relative goals, and proprioception.

D. Model Training

  • MulDP uses an end-to-end supervised learning loss to train the policy on QPND.The loss is introduced as the training objective for the model.Source: D. Model Training · p. 4
  • Training samples expert trajectories, depth images, relative goals, and proprioception from QPND, using a history size of 5 for each modality.The Transformer layers are denoted by φ and the diffusion-process parameters by θ.Source: D. Model Training · p. 4
  • The diffusion strategy uses 10 DDPM denoising steps; training runs for 1000 epochs with AdamW, a 10^-4 learning rate, batch size 512, cosine scheduling, and warm-up.The schedule and warm-up are used to stabilize training.Source: D. Model Training · p. 4

IV. EXPERIMENTS

The experiments assess MulDP through quantitative and qualitative results, focusing on component contributions, fixed-goal navigation, and dynamic target following. They also compare it with existing quadruped navigation algorithms.

IV. EXPERIMENTS

  • Q1 asks whether MulDP's individual components improve navigation and parkour capabilities and how it compares with existing quadruped navigation algorithms.Source: §IV · p. 5
  • Q2 asks whether MulDP can provide flexible and diverse fixed-goal navigation in challenging environments.Source: §IV · p. 5
  • Q3 asks whether MulDP can perform robust and generalizable dynamic target-following navigation in complex outdoor environments.Source: §IV · p. 5

A. Experimental Setup and Deployment Details

The evaluation covers fixed-goal simulation tasks and fixed- and dynamic-goal real-world tasks. It specifies the simulation benchmark and robot deployment setup.

A. Experimental Setup and Deployment Details

  • Simulation evaluates fixed-goal navigation with a Unitree Go1 in Isaac Sim across six terrain types, each with two difficulty levels.The terrains are Gap, Hurdle, Upstairs, Downstairs, Gravel, and Box.Source: A. Experimental Setup and Deployment… · p. 5
  • Simulation measures Success Rate, SPL, Traversal Rate, and Time to Reach, with 50 randomized initial positions per scenario and goals 5–8 m away.Success Rate requires reaching within 0.2 m; episodes have a 60 s maximum time limit.Source: A. Experimental Setup and Deployment… · p. 5
  • Real-world fixed- and dynamic-goal experiments use a Unitree Go1 with an Orin NX 16GB, a RealSense D435i camera, and UWB target localization.The robot operates at 5 Hz, with sensor configurations matching training.Source: A. Experimental Setup and Deployment… · p. 5

B. Comparison and Ablation Study

MulDP is compared with diffusion-based and point-goal navigation methods in comparison experiments. Across evaluated scenarios, it reports average SR of 89.7%, average SPL of 71.9%, and average TTR of 4.9 s.

B. Comparison and Ablation Study

  • The comparisons include NavDP*, ViNT*, PointNav*, and released NavDP and ViNT checkpoints, with test terrains referenced to Section IV-A.NavDP* is the critic-free NavDP variant trained on QPND; the cited figure is titled “Comparison Experiments of MulDP.”Source: 1) Comparison Settings · p. 5, Figure 3 · p. 5
  • MulDP achieves 89.7% SR versus 59.0% for NavDP*, while ViNT* has the highest non-diffusion baseline SR at 71.5%.MulDP and NavDP* both exceed 40% TR, and the authors attribute MulDP's SR advantage over NavDP* to direct depth input and end-to-end joint training.Source: 1) Comparison Settings · p. 5
  • Released NavDP and ViNT checkpoints fail in high-difficulty terrain scenarios and have SR of 38.4% and 35.8%, respectively.Their SPL scores are 27.1% and 25.5%, respectively; the authors report conservative obstacle avoidance and inability to traverse obstacles in these scenarios.Source: 1) Comparison Settings · p. 5
  • 89.7% average SR, 71.9% average SPL, and 4.9 s average TTR are reported for MulDP across all evaluated scenarios.The authors describe these results as demonstrating its ability to identify effective planning solutions and guide the robot through complex environments.Source: 1) Comparison Settings · p. 5

2) Ablation Settings:

Ablations identify the contributions of proprioception, decision memory, spatial perception, and QPND’s noise injection and trajectory augmentation. The results also show dataset noise supporting transfer to a narrow residential environment with unseen obstacles and layouts.

2) Ablation Settings:

  • The ablations remove proprioceptive inputs, the Decision Memory Encoder, the Current Spatial Encoder, depth-image and proprioceptive-signal noise injection, or trajectory augmentation.The experiments evaluate each module’s contribution.Source: 2) Ablation Settings · pp. 5–6
  • Removing proprioception lowers success rate from 89.7% to 69.5% and raises time to reach from 4.9 s to 7.5 s; removing decision memory yields 7.4% SR and 90.3% TR.Removing the current spatial encoder lowers SR from 89.7% to 82.0% and can induce erroneous decisions in challenging dynamic scenarios.Source: 2) Ablation Settings · p. 6, Table II · p. 6
  • Without trajectory augmentation, SR falls to 32.5%, SPL to 23.7%, and TR to 21.0%; without data noising, SR is 75.0%.Without augmentation, dataset size is nearly halved; the authors report that this exacerbates overfitting and degrades generalization.Source: 2) Ablation Settings · p. 6
  • In an unseen narrow residential environment, MulDP avoids obstacles while the w/o DN model collides; the authors attribute the contrast to real-world depth-sensing noise.The deployed w/o DN robot also misjudges obstacle geometry and fails to distinguish low from tall obstacles, producing inappropriate traversal behaviors.Source: 2) Ablation Settings · p. 6, Figure 4 · p. 6

C. Fixed-goal Navigation Task

MulDP generates diverse feasible routes for fixed goals in simulation and demonstrates fixed-goal navigation in real-world obstacle scenarios. Its velocity profiles show anticipatory acceleration for parkour and heading adjustments during obstacle avoidance.

C. Fixed-goal Navigation Task

  • In an unseen 16 m × 16 m Isaac Sim environment with a goal 13 m from the start, MulDP generates diverse, feasible trajectories under identical start–goal conditions.The authors contrast its distribution over feasible motion plans with deterministic regression policies that converge to averaged behavior.Source: C. Fixed-goal Navigation Task · p. 6
  • In real-world fixed-goal scenarios, MulDP generates diverse plans around static obstacles and performs real-time navigation in dynamic-obstacle scenarios.Fig. 1 depicts both settings.Source: C. Fixed-goal Navigation Task · p. 6
  • In the parkour velocity profile, the robot accelerates before gap edges, with peak velocities approaching 2 m/s, then reduces speed after landing.The authors describe this as anticipatory momentum accumulation rather than reacting only upon contact.Source: C. Fixed-goal Navigation Task · p. 6
  • During obstacle avoidance, MulDP maintains average linear velocity around 1.1 m/s while angular velocity adjusts, reaching local peaks as linear velocity remains smooth.The authors characterize this as heading adjustment rather than abrupt deceleration.Source: Figure 7 · p. 7

D. Dynamic Target-Following Navigation Task

In outdoor dynamic target-following, MulDP completes over 1 km of autonomous parkour navigation across diverse terrains, including under distribution shifts and with uncommon obstacles. The conclusion reports robust fixed-goal and target-following navigation, and identifies future work in semantic perception, higher-level planning, and lightweight safety evaluation.

D. Dynamic Target-Following Navigation Task

  • MulDP completes over 1 km of fully autonomous outdoor parkour navigation while tracking a dynamic target across stairs, grass, gaps, obstacle avoidance, and hurdle jumping.The authors report long-horizon decision consistency and adaptation despite illumination variation and motion blur causing distribution shifts between training data and onboard observations.Source: D. Dynamic Target-Following Navigation… · p. 7
  • On unseen outdoor obstacle geometries, including disc-shaped objects on grass and bicycles, MulDP recognizes non-traversable structures and chooses safe circumvention.These obstacle configurations were not explicitly encountered during training.Source: D. Dynamic Target-Following Navigation… · p. 7, Figure 8 · p. 7

V. CONCLUSIONS

  • The conclusion reports robust fixed-goal and dynamic target-following navigation, including long-distance outdoor deployment without human intervention.The stated future work includes semantic perception, higher-level planning, and lightweight safety evaluation modules.Source: §V · p. 8
  • MulDP generates temporally coherent velocity-command horizons and combines depth perception, proprioception, goal information, and a low-level locomotion policy for parkour behaviors.The conclusion names gap crossing, stair traversal, obstacle circumvention, and hurdle jumping among those behaviors.Source: §V · p. 8

Related papers

Loading 2609.03984v1…