PaperlayerView exact revision on arXiv

Source-linked AI summary

E2DRL: End-to-End Differentiable Reinforcement Learning for 3D Obstacle Avoidance in Quadruped Robots

Cheng Fan, Bingyang Zhou, Gang Yang, Zixuan Zhuang, Aoqian Zhang, Pengzhi Yang, Xin Liu, Chunzheng Wang, Yan Rui Tan, Fan Shi

arXiv:2610.19627v1cs.RO

TL;DR

Differentiable RL for legged robots has largely addressed blind walking or simulated 2D tasks, leaving deployable visual 3D obstacle avoidance unexplored. E2DRL trains a privileged teacher with a soft differentiable obstacle formulation and distills it into a depth-based student for deployment. In simulation, it reaches the avoidance-success plateau faster than PPO and outperforms comparison methods in reported tracking and obstacle traversal; the paper also reports sim-to-real validation.

  • Problem

    Differentiable RL for legged robots has been limited mainly to blind locomotion or simulated 2D tasks, while deployable visuomotor policies for 3D obstacle avoidance remain unexplored.

    Source: §I · p. 1

  • Method

    E2DRL trains a privileged teacher via SAPO with a soft differentiable obstacle-interaction surrogate, distills it into a recurrent depth-based student via DAgger, and uses terrain-aware gating for deployment.

    Source: §I · pp. 1–2, A. System Overview · p. 2

  • Results

    SAPO reaches the avoidance-success plateau after approximately 16M training steps versus PPO’s approximately 67M, while E2DRL also shows stronger velocity tracking and obstacle traversal than PPO and Duality-based DCBF NMPC.

    Source: 2) Simulation Results · p. 5

  • Takeaways & Limitations

    The paper reports the first sim-to-real validation of end-to-end differentiable RL for quadrupedal 3D obstacle avoidance in static and dynamic environments.

    Source: §I · p. 2, §VI · pp. 7–8

Abstract

from arXiv · show

Differentiable reinforcement learning shows strong promise for robot policy learning, but its application to legged locomotion has remained limited to blind walking or simulated 2D tasks. Extending it to end-to-end visual obstacle avoidance is challenging because hard collision constraints and early termination upon contact corrupt first-order gradients, trapping optimizers in local minima. We present E2DRL, which addresses this with a soft differentiable obstacle-interaction formulation: penetration penalties across multiple body keypoints with velocity-conditioned stop-gradient shaping to provide the first-order training signals for 3D obstacle avoidance. To realize a deployable perception-control system, a privileged teacher trained via Soft Analytic Policy Optimization (SAPO) in a differentiable simulator is distilled into a recurrent depth-based student via DAgger, then blended with a base locomotion policy using a terrain-aware gating module for deployment. In simulation, E2DRL achieves higher sample efficiency than PPO and demonstrates superior velocity-tracking performance and 3D obstacle-avoidance capability compared with both PPO and Duality-based DCBF NMPC. We present the first sim-to-real validation of end-to-end differentiable RL for 3D obstacle avoidance on a Unitree Go2 in static and dynamic environments.

I. INTRODUCTION

E2DRL addresses the challenge of applying differentiable reinforcement learning to command-conditioned 3D obstacle avoidance, where hard contacts and early termination disrupt first-order optimization. It combines a soft obstacle-interaction surrogate with teacher-to-student transfer and terrain-aware policy gating; the paper reports improved sample efficiency, tracking, and obstacle traversal.

I. INTRODUCTION

  • Quadruped obstacle avoidance must combine commanded-velocity tracking with 3D posture and locomotion changes, such as lateral bypassing or crawling beneath obstacles.The introduction notes that cluttered environments include obstacles with varied heights and geometries.Source: §I · p. 1
  • Optimization-based DCBF NMPC uses online safety-constrained trajectory optimization, whereas model-free RL estimates gradients from sampled returns and can require extensive interaction.The introduction identifies responsive velocity tracking amid challenging obstacles as difficult for methods relying on online constrained optimization and simplified models.Source: §I · p. 1
  • Hard collisions and early termination create gradient discontinuities that can trap differentiable RL in local minima, while prior legged-robot work focused on blind locomotion or simulated 2D tasks.The stated challenge is to keep obstacle interactions informative to first-order optimization while discouraging nonphysical penetration.Source: §I · p. 1, B. Differentiable Reinforcement Learning · p. 2
  • E2DRL combines a soft differentiable obstacle surrogate, a distilled depth-based student, and terrain-aware gating, with improved sample efficiency, velocity tracking, and 3D obstacle traversal.The paper also reports the first sim-to-real validation of end-to-end differentiable RL for overhanging and ground obstacles in static and dynamic environments.Source: §I · pp. 1–2

A. Learning-based Quadrupedal Locomotion

  • Prior learning-based quadruped work includes obstacle navigation, high-speed cluttered-environment locomotion, and parkour, while this work targets end-to-end 3D avoidance with velocity-command tracking.The related-work passage identifies Miki et al., He et al. [20], and Zhuang et al. [21] for those examples, respectively.Source: A. Learning-based Quadrupedal Locomotion · p. 2

B. Differentiable Reinforcement Learning

  • Differentiable simulation offers analytical policy gradients, but discontinuous contact can impair optimization; prior quadruped demonstrations were limited to blind locomotion or simulated 2D tasks.The cited related work describes SHAC [23] and SAPO [24] as approaches addressing optimization-landscape issues.Source: B. Differentiable Reinforcement Learning · p. 2

III. PROBLEM FORMULATION

The formulation specifies velocity-commanded vision-based navigation around overhanging and ground obstacles, then outlines a two-stage learning and deployment pipeline. A privileged teacher is trained in a differentiable simulator, distilled into a depth-and-proprioception student, and combined with a locomotion policy through terrain-aware gating.

III. PROBLEM FORMULATION

  • The robot must traverse overhanging obstacles requiring body-height adaptation and ground obstacles requiring lateral bypassing, while following desired velocity commands.The task is vision-based 3D obstacle avoidance for a quadruped.Source: §III · p. 2
  • The proprioceptive observation includes angular velocity, projected gravity, velocity commands, joint states, and previous actions.The proprioceptive state is specified as 45-dimensional.Source: §III · p. 2
  • During training, the teacher receives privileged exteroceptive state containing robot pose and velocity plus obstacle configuration and geometric parameters.The obstacle parameters include clearance height, lateral width, and longitudinal depth.Source: §III · p. 2
  • The student replaces privileged exteroception with an onboard 64×64 depth image, alongside proprioception, for deployment.The student observation is depth imagery plus proprioception.Source: §III · p. 2

A. System Overview

  • The system uses a two-stage learning framework intended to bridge sample-efficient first-order policy gradients with vision-based real-world deployment.The system overview identifies onboard perception as the basis for 3D obstacle avoidance.Source: A. System Overview · p. 2
  • A privileged teacher is trained with SAPO in a differentiable simulator, using a stable frictional contact model and soft obstacle interactions.The soft obstacle-interaction formulation is introduced to improve optimization robustness in 3D obstacle-avoidance scenarios.Source: A. System Overview · p. 2
  • DAgger distills the teacher into a vision-based student that relies on onboard depth images and proprioception.The student policy is the second stage of the learning framework.Source: A. System Overview · p. 2
  • A separately trained terrain-aware gate interpolates between the obstacle-avoidance student and the original proprioceptive locomotion policy to preserve nominal locomotion capability.The gate is also used during distillation data collection.Source: A. System Overview · p. 2

B. Privileged Teacher Policy via Differentiable RL

The privileged teacher is optimized in a differentiable simulator using SAPO and a soft obstacle-interaction surrogate. The surrogate keeps rollouts active through obstacle penetration and supplies differentiable penalties for 3D avoidance.

B. Privileged Teacher Policy via Differentiable RL

  • SAPO optimizes the privileged teacher over a k-step objective combining reward, policy entropy, discounting, and a terminal soft value.The simulator's differentiable transition enables analytical state and action gradients to be backpropagated through the rollout.Source: B. Privileged Teacher Policy via… · pp. 2–3
  • The differentiable frictional contact model uses a formulation whose gradients remain stable as contact velocities approach zero, with tangential force defined as zero at zero relative tangential velocity.The paper adopts the formulation of Geilinger et al. (2020) instead of NVIDIA Warp's standard friction model.Source: B. Privileged Teacher Policy via… · p. 3
  • The training surrogate keeps rollouts active during obstacle penetration and applies continuous differentiable penalties, providing dense first-order signals for crawling and bypassing.It is explicitly a training surrogate, not a model of physical obstacle contact.Source: B. Privileged Teacher Policy via… · p. 3
  • Penetration penalties are computed over differentiable robot keypoints, including the base center of mass, joints, feet, and front and rear base points.The penalty activates only when a keypoint penetrates an obstacle; collision-free traversal yields zero penetration depth.Source: B. Privileged Teacher Policy via… · p. 3
  • Velocity-conditioned stop-gradient shaping scales the penalty without adding a gradient along forward velocity, preserving velocity tracking while discouraging high-speed traversal during penetration.The negative constant supplies the critic with a signal to distinguish successful obstacle avoidance.Source: B. Privileged Teacher Policy via… · p. 3

C. Vision-based Student Policy Distillation

DAgger distills the privileged teacher into a vision-based student using onboard depth and proprioception. The student combines visual features with recurrent processing to produce joint targets, while training progressively shifts control from teacher to student rollouts.

C. Vision-based Student Policy Distillation

  • The student uses a CNN for depth-image features, combines them with proprioception in a GRU, and maps recurrent features to 12-dimensional joint position targets.The GRU captures spatio-temporal obstacle dynamics.Source: C. Vision-based Student Policy… · p. 4
  • DAgger trains the depth-and-proprioception student by minimizing the L2 difference between teacher and student actions.The student policy is intended for deployment on physical hardware.Source: C. Vision-based Student Policy… · pp. 3–4
  • The distillation rollout distribution mixes teacher and student control, with the predefined mixing probability decaying from one to zero as training progresses.Collected data is aggregated with previous data for Follow-The-Leader training; alternating data collection and training enables single-GPU distillation.Source: C. Vision-based Student Policy… · p. 4

D. Training Details

Training uses a two-stage curriculum that moves from basic velocity tracking to progressively harder 3D obstacle avoidance. Deployment adds terrain-aware switching and action blending between the student and nominal locomotion policies.

D. Training Details

  • The curriculum begins with target-distance velocity-tracking locomotion, then increases obstacle difficulty after five consecutive collision-free traversals.Source: D. Training Details · p. 4
  • The per-step reward is a weighted sum of training reward terms, with only the penetration reward included in that weighted reward.Source: D. Training Details · p. 4
  • The terrain-aware gate switches from the base locomotion policy to the obstacle-avoidance student when estimated obstacle-surface distance falls below 0.6 m.The gate estimates obstacle proximity from visual observations because raw depth alone cannot reliably identify obstacles requiring avoidance.Source: D. Training Details · p. 4
  • Policy outputs are interpolated over a 30-step transition window to avoid abrupt action discontinuities during switching.The gate is trained on depth observations from Isaac Gym, NVIDIA Warp, and real-world data.Source: D. Training Details · p. 4

V. EXPERIMENTS

Experiments compare differentiable RL with PPO and tuned Duality-based DCBF NMPC in simulation, then assess transfer to a Unitree Go2. E2DRL improves sample efficiency, velocity tracking, and obstacle avoidance, with ablations examining its obstacle-interaction design.

2) Simulation Results:

  • The evaluation also examines sim-to-real transfer of the vision-based policy on a Unitree Go2 in indoor and outdoor settings with static and dynamic obstacles.The static-obstacle figure reports successful avoidance of overhanging and ground obstacles across varied indoor and outdoor scenarios.Source: §V · p. 4, Figure 5 · p. 6

1) Simulation Setup:

  • In simulation, the robot avoids overhanging and ground obstacles while tracking velocity commands, with behaviors shown in top and side views.Training uses 128 parallel environments, one randomly selected obstacle per episode, and identical reward, soft-obstacle, and contact-dynamics settings across PPO, SHAC, and SAPO.Source: 1) Simulation Setup · pp. 4–5, 2) Simulation Results · p. 5, Figure 2 · p. 5

2) Simulation Results:

  • SAPO converges in about 16M training steps versus PPO's 67M, and its final discounted reward is about 1.1 times PPO's; SHAC's is about half SAPO's.SAPO and SHAC have comparable convergence rates; the authors attribute SAPO's reward advantage to entropy regularization helping differentiable RL escape poor local optima.Source: 2) Simulation Results · p. 5, Figure 3 · p. 5
  • At 1.2 m/s, E2DRL tracks commanded velocity more reliably than PPO and tuned NMPC in 0.28 m overhang and 0.8 m ground-obstacle scenarios.NMPC ultimately halts before completing traversal, PPO oscillates, and the gated policy tracks better than the ungated policy in obstacle-free segments.Source: 2) Simulation Results · p. 5, Figure 4 · p. 5
  • Across 100 environments, E2DRL passes beneath overhangs as low as 0.24 m and bypasses ground obstacles up to 1.0 m wide, while PPO and tuned NMPC fail at easier configurations.The obstacle-proximity and SDF baselines also achieve lower success rates; the distilled student handles more challenging obstacles than the policy trained directly with visual observations.Source: 2) Simulation Results · p. 5, Figure 6 · p. 6
  • Replacing the soft penetration penalty with hard contact termination reduces feasible overhang height from 0.24 m to 0.33 m and ground-obstacle width from 1.0 m to 0.58 m.Removing velocity-conditioned stop-gradient shaping further restricts feasible height and width to 0.34 m and 0.85 m; without it, the optimizer tends to suppress forward speed rather than learn avoidance.Source: 2) Simulation Results · p. 6
  • The contact model keeps reported actor-gradient magnitude below 1.0, avoiding spikes during foot strikes.The reported plots separate the whole-actor gradient norm from the norm induced by the soft obstacle-interaction formulation.Source: 2) Simulation Results · p. 6

1) Real-World Experimental Setup:

The Unitree Go2 was tested with onboard depth perception in static and dynamic obstacle settings. Across indoor and outdoor trials, E2DRL demonstrated 3D obstacle avoidance, including crawling beneath overhangs and bypassing ground obstacles.

1) Real-World Experimental Setup:

  • The real-world evaluation used onboard depth perception to test the policy in both static and dynamic 3D obstacle environments.Experiments were conducted indoors and outdoors.Source: 1) Real-World Experimental Setup · p. 6

2) Results in Static Environments:

  • Static trials included crawling beneath clearances of 0.38 m and 0.27 m, lateral ground-obstacle avoidance, and outdoor traversal around chairs, pillars, and trees.Across five trials per scenario, success was 5/5 for ground obstacles and 4/5 for overhanging obstacles; the overhang failure involved collision under the lower clearance.Source: 2) Results in Static Environments · p. 7

3) Results in Dynamic Environments:

VI. CONCLUSIONS

SUPPLEMENTARY MATERIAL

The supplement documents the student and deployment architecture alongside shared training and evaluation procedures. It specifies the network structure, curriculum transition, randomization, and termination conditions.

S1. Network Architecture and Deployment

  • The depth-based student combines visual features with proprioception, processes them through a 256-unit GRU, and outputs 12 joint-position targets.Its depth encoder has three convolutional layers; the GRU state resets at each episode.Source: S1. Network Architecture and Deployment · p. 9
  • The terrain-aware gate estimates obstacle proximity rather than using minimum raw depth, because the nearest return is typically ground at about 0.5 m.The gate is trained separately from the student and nominal policy.Source: S1. Network Architecture and Deployment · p. 9

S2. Training and Evaluation Protocol

  • PPO, SHAC, and SAPO comparisons share architectures, observation and action spaces, rewards, obstacle and contact formulations, and curriculum.Each method uses five random seeds; plotted shaded bands represent empirical variability, not confidence intervals.Source: S2. Training and Evaluation Protocol · p. 9
  • Training switches from walking to obstacle avoidance at 70% success reaching a target 4.5 m away, then raises difficulty after five consecutive collision-free traversals.Episodes are capped at 3000 simulation steps and also terminate on specified height or body-angle limits.Source: S2. Training and Evaluation Protocol · p. 9

S3. Body Keypoints and Baseline Implementation

The supplement details the obstacle geometry and baselines, analyzes how velocity stop-gradient shapes obstacle gradients and forward motion, and reports teacher–student transfer performance near feasibility limits.

S3. Body Keypoints and Baseline Implementation

  • The obstacle-interaction reward uses 19 body keypoints; the single-keypoint ablation uses the center of the robot base’s top surface.The proximity and signed-distance-field baselines use the implementations described for Miki et al. and Stanley et al., respectively.Source: S3. Body Keypoints and Baseline… · p. 9

S4. Gradient Analysis

  • Without velocity stop-gradient, obstacle-loss gradients acquire a pronounced negative longitudinal component, and traversal fails in the reported 1.2 m/s scenarios.With stop-gradient, the full formulation closely tracks the command and traverses a 0.26 m overhang and a 0.90 m ground obstacle; the mechanism removes the direct derivative through forward velocity while retaining velocity-dependent scaling and geometric feedback.Source: S4. Gradient Analysis · p. 10, Figure S3 · p. 10, Figure S4 · p. 10

S5. Teacher–Student Transfer and Gating

  • The student matches the teacher on ground obstacles through 0.9 m widths, but loses performance near the feasibility boundary for overhangs.Both reliably traverse clearances from 0.33 m to 0.28 m; student performance decreases from 0.27 m to 0.24 m while the teacher remains reliable to 0.24 m. Gating improves velocity tracking in obstacle-free segments.Source: S5. Teacher–Student Transfer and Gating · p. 11, Figure S5 · p. 11

S6. Optimization and Training Efficiency

The supplement compares SAPO and PPO under three convergence criteria and reports parameter-sensitivity sweeps and simulation settings. Under the selected avoidance-success criterion, SAPO uses fewer environment steps but takes longer in wall-clock time.

S6. Optimization and Training Efficiency

  • 16M environment steps for SAPO versus 67M for PPO reach the avoidance-success plateau, while training takes 344 versus 166 minutes, respectively.The selected criterion directly measures the task objective; the sample-efficiency gain does not mean faster elapsed training.Source: S6. Optimization and Training Efficiency · p. 11, Table S1 · p. 11

S7. Parameter Sensitivity and Simulation Settings

  • Increasing the penetration penalty weight does not uniformly improve obstacle limits in the one-parameter-at-a-time sweeps.Limits use a 90% success threshold, with smaller overhang clearance and larger ground-obstacle width indicating better avoidance capability.Source: S7. Parameter Sensitivity and… · p. 11, Table S2 · p. 12
  • The supplement lists contact and friction settings shared across PPO, SHAC, and SAPO, and reports that entropy regularization is tuned automatically against a target.The velocity-shaping parameter is σ = 0.50; the task-specific base penetration weights are 40 for overhanging and 10 for ground obstacles.Source: S7. Parameter Sensitivity and… · pp. 11–12, Table S3 · p. 12, Table S4 · p. 12

Related papers

Loading 2610.19627v1…