PaperlayerView exact revision on arXiv

Source-linked AI summary

DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstacles

Sanghyuk Park, Kwanwoo Lee, Taekyung Kim, Seohyeon Lim, Yisoo Lee

arXiv:2609.38873v2cs.ROcs.SYeess.SY

TL;DR

Safety-filtered training can limit policy-generated experience, while dynamic-obstacle navigation requires accounting for relative robot-obstacle motion. DODGER trains with direct policy rollouts and DPCBF-based safety guidance; results include closer recovery of the HJ reference region and humanoid navigation demonstrations in simulation and the real world.

  • Problem

    Executing safety-filtered actions during training can restrict exposure to states induced by the policy itself, while dynamic-scene safety depends on relative robot-obstacle position and velocity.

    Source: B. CBF-RL · p. 2

  • Method

    DODGER directly rolls out policy-generated actions and uses DPCBF-filtered references and constraint violations to guide learning.

    Source: §I · p. 2

  • Results

    DODGER more closely recovers the HJ reference region and converges faster than safety-filtered CBF-RL baselines, with successful multi-obstacle humanoid navigation shown in simulation and real-world experiments.

    Source: §I · p. 2, C. Discussion · p. 7, §V · p. 8

  • Takeaways & Limitations

    The results support learning collision-avoidance behavior through direct policy interaction with DPCBF guidance, without requiring a runtime safety filter.

    Source: §V · p. 8

    Richer three-dimensional obstacle representations and full-order, perception-aware safety modeling remain future work.

    Source: §V · p. 8

Abstract

from arXiv · show

Robots operating in human-centered environments must safely navigate among multiple dynamic obstacles to avoid collisions with people and surrounding infrastructure. Control barrier functions (CBFs) provide an effective mechanism for safety filtering, and recent CBF-based reinforcement learning (RL) methods embed such safety information into learned policies. However, executing only safety-filtered actions during training can restrict policy exploration, a limitation that becomes particularly consequential in dynamic scenes where safety depends on relative robot-obstacle motion. We propose DODGER, a safety-guided RL framework that directly executes policy-generated actions to drive training rollouts while using CBF-filtered references and constraint violations to shape the policy toward collision-avoidance behavior. We evaluate DODGER through a Dubins-car safety analysis and demonstrate goal-directed navigation among multiple dynamic obstacles in full-order humanoid simulation and real-world humanoid experiments using LiDAR-based perception, without a runtime safety filter.

I. INTRODUCTION

Humanoid navigation must combine goal-directed commands with awareness of moving obstacles, while existing safety-filtered training can limit experience with the policy’s own actions. DODGER uses dynamic CBF guidance during learning while allowing policy actions to drive rollouts, and is evaluated in simulation and real-world humanoid navigation.

I. INTRODUCTION

  • Humanoid navigation must select goal-directed commands while accounting for the relative motion of nearby obstacles and people.A separate whole-body controller realizes commands from the low-dimensional navigation interface.Source: §I · p. 1

III. METHODOLOGY

  • DODGER uses DPCBF guidance and graph-based scene encoding while policy-generated actions drive training rollouts and filtered references and constraint evaluations guide learning.This formulation is intended to let the policy learn from the consequences of its own actions without safety-filtered rollouts.Source: §I · p. 2, §III · p. 2

I. INTRODUCTION

  • DODGER demonstrates goal-directed navigation among multiple dynamic obstacles in full-order humanoid simulation and real-world deployment on a Unitree G1.Its evaluation also includes a Dubins-car safety analysis.Source: §I · p. 2

A. Control Barrier Functions

  • A control barrier function can define a safe set whose forward invariance is ensured when the controller satisfies its barrier constraint.The paper describes enforcing this constraint with a CBF-based quadratic program that minimally modifies a nominal input.Source: A. Control Barrier Functions · p. 2

B. CBF-RL

  • CBF-RL executes filtered actions during training, so its experience follows the filter-induced distribution; dynamic scenes also require accounting for relative obstacle motion.The cited CBF-RL experiments focus on static environments.Source: B. CBF-RL · p. 2

A. Dynamic Parabolic CBF

DODGER formulates dynamic-obstacle safety using a reduced-order navigation model and a velocity-dependent parabolic barrier. Its high-level policy combines local goal information with graph-encoded obstacles, while barrier references and violations guide learning without replacing policy actions in rollouts.

A. Dynamic Parabolic CBF

  • The DPCBF forms a state-dependent parabolic safety boundary from relative robot-obstacle position and velocity, adapting its constraint to dynamic interactions.Its velocity-dependent formulation provides a first-order safety constraint for the reduced-order dynamics.Source: A. Dynamic Parabolic CBF · p. 2

1) Reduced-Order Dynamics:

  • The reduced-order model captures planar forward and lateral translation and heading, providing a low-dimensional interface for CBF constraints independent of full-order dynamics.Its state includes floating-base position and heading plus body-frame forward and lateral velocities.Source: 1) Reduced-Order Dynamics · p. 2

2) DPCBF Formulation:

  • The DPCBF evaluates relative position and velocity in a line-of-sight frame and models each obstacle with constant planar velocity during barrier evaluation.Its parameters adapt the parabolic boundary to the relative state; it becomes less restrictive for distant obstacles or smaller relative speeds and more conservative as obstacles approach or relative speed increases.Source: 2) DPCBF Formulation · p. 3

B. DODGER Overview

  • DODGER represents the robot, goal, and detected obstacles as a graph whose robot-centered attention encoder aggregates relative position, velocity, and boundary-distance features.The high-level policy combines the graph embedding with local observations; its action drives the rollout while a fixed pretrained low-level policy executes the navigation command.Source: B. DODGER Overview · p. 3, Figure 2 · p. 3

C. Safety-Guided Reinforcement Learning

DODGER executes policy-generated actions during training while using a CBF-filtered reference and constraint violations to guide learning. Its violation-dependent positive-return adjustment and two-head critic provide safety guidance while preserving negative task penalties.

C. Safety-Guided Reinforcement Learning

1) Safety Filter:

  • The training QP incorporates DPCBF constraints for detected obstacles and input bounds, and its unrelaxed form was feasible in 99.6% of solves.Slack relaxation handled the remaining 0.4%.Source: 1) Safety Filter · p. 4

2) Safety-Guided RL Training:

  • Normalized constraint violations increase δ_t, reducing positive-return and bootstrapping contributions while the rollout continues with actor actions.The constraint normalization scale is updated from the maximum violation observed during each rollout, with exponential smoothing.Source: 2) Safety-Guided RL Training · p. 4
  • A two-head critic applies the violation factor only to positive rewards, preserving the original magnitudes of task-specific negative penalties.This addresses the risk that scaling signed rewards would weaken negative penalties, including collision and workspace-violation penalties.Source: 2) Safety-Guided RL Training · pp. 4–5
  • DODGER combines a penalty for deviation from the QP reference with violation-dependent return adjustment; neither signal replaces the actor action during interaction.The action-correction reward contributes to the negative branch, while δ_t reduces the positive branch according to violation severity.Source: 2) Safety-Guided RL Training · pp. 4–5

A. Low-Dimensional Validation

On the Dubins-car benchmark, DODGER’s critic-induced risk region more closely matches the HJ unsafe set than CBF-RL’s. The comparison uses a tractable three-dimensional HJ reference and reports higher unsafe-set coverage with fewer false positives for DODGER.

A. Low-Dimensional Validation

  • The three-dimensional Dubins-car benchmark provides a tractable HJ reference because grid-based HJ reachability scales exponentially with state dimension.The paper states direct online application to its five-dimensional reduced-order navigation model is computationally impractical.Source: A. Low-Dimensional Validation · p. 5
  • The HJ unsafe set is defined by V_HJ(x) ≤ 0; the comparison uses a shared risk threshold ϵ = −0.5 and measures unsafe-set coverage and normalized false positives.Cunsafe is unsafe-set coverage, while CFP is the false-positive measure normalized by the measure of the HJ unsafe set.Source: A. Low-Dimensional Validation · p. 5
  • 94.7 ± 0.93% Cunsafe and 8.9±2.47% CFP for DODGER, versus 71.2±3.47% and 18.5±4.62% for CBF-RL.The reported metrics are means ± standard deviations over five random seeds; the figure compares the methods against the HJ safety boundary.Source: A. Low-Dimensional Validation · p. 5, Figure 3 · p. 5

B. Humanoid Dynamic-Obstacle Avoidance

DODGER was evaluated for full-order humanoid navigation among dynamic obstacles, with a curriculum reaching 20 obstacles at speeds up to 0.6 m/s. It converged faster than the CBF-RL baselines at speeds up to 0.6 m/s and was the only method to converge in the evaluation at speeds up to 0.8 m/s.

1) Experimental Setup:

  • The high-level navigation policy was trained in MuJoCo Playground [22] at 10 Hz, while a pretrained Unitree G1 velocity-tracking policy operated at 50 Hz during full-order humanoid navigation among multiple dynamic obstacles.LiDAR and the ROS 2 obstacle detector tracked obstacles within a 5 m radius; up to 10 were retained for GAT encoding and the multi-constraint QP.Source: 1) Experimental Setup · pp. 5–6
  • The four-stage curriculum increased obstacle density, velocity, state randomization, and observation noise, reaching 20 dynamic obstacles at speeds up to 0.6 m/s.Success required reaching the goal within 0.6 m before failure or timeout while meeting the stated heading, velocity, and angular-velocity thresholds.Source: 1) Experimental Setup · p. 5

2) RL Training Comparison:

  • CBF-RL and CBF-RL-Min both executed safety-filtered actions during training; CBF-RL-Min used only the most critical obstaclewise CBF penalty.CBF-RL summed CBF-condition penalties over detected obstacles.Source: 2) RL Training Comparison · p. 6
  • At speeds up to 0.6 m/s, DODGER converged after approximately 114 million transitions, versus 277 million for CBF-RL-Min; CBF-RL did not converge within 600 million transitions and finished at 72.3%.Fig. 4 plots evaluation success rates during the first 300 million transitions.Source: 2) RL Training Comparison · p. 6, Figure 4 · p. 6
  • At speeds up to 0.8 m/s, DODGER converged after approximately 536 million transitions; neither baseline converged, with final success rates of 88.5% for CBF-RL-Min and 0.4% for CBF-RL.Training used a 1200-million-transition budget.Source: 2) RL Training Comparison · p. 6

3) CBF Formulation Comparison:

DPCBF reaches curriculum stage 3 and converges in fewer environment transitions than the alternative formulations, while distance-based ECBF variants produce more aggressive yaw behavior. The section also reports real-robot navigation and evaluation beyond the training obstacle limit.

3) CBF Formulation Comparison:

  • DPCBF and both distance-based ECBF variants converge at curriculum stage 3, but DPCBF requires the fewest environment transitions; C3BF fails to converge and reaches only stage 1.The comparison uses the same DODGER training configuration across formulations.Source: 3) CBF Formulation Comparison · p. 6
  • Distance-based ECBF variants produce larger yaw-rate commands and more frequent near-saturation than DPCBF in both actor outputs and resulting G1 motion.Table III reports yaw-rate statistics for the different CBF formulations.Source: 3) CBF Formulation Comparison · p. 6, Table III · p. 7

C. Discussion

  • Removing the 10-obstacle evaluation cap yields 97.27% success with 12.54 observed obstacles on average and up to 20 simultaneously.68.65% of decisions involve more than 10 obstacles.Source: C. Discussion · p. 7
  • Across the HJ benchmark and full-order humanoid navigation, DODGER more closely recovers the HJ reference region and converges faster than safety-filtered CBF-RL baselines.Source: C. Discussion · p. 7
  • C3BF achieves 23% final success, with 76% of episodes timing out and 1% ending in collision, indicating overly conservative behavior in dense scenes.The distance-based ECBF converges but produces excessive rotational motion; yaw regularization reduces but does not eliminate the behavior, while DPCBF has lower yaw-rate RMS and near-saturation.Source: C. Discussion · p. 7

D. Real-Robot Experiments

  • The real-robot experiment uses LiDAR and IMU-based pose estimation and navigates among five moving pedestrians despite perception errors and imperfect locomotion tracking.A1 and A2 are initially merged into one obstacle; the estimate later splits them.Source: D. Real-Robot Experiments · pp. 7–8
  • During the real-robot run, the minimum DPCBF value becomes negative while the distance-dependent margin remains positive, and the robot avoids collision and reaches the goal.The reported negative values are associated with abrupt perceived-state changes, deviations from the constant-velocity obstacle model, and realized-motion deviations.Source: Figure 5 · p. 7, D. Real-Robot Experiments · p. 8

V. CONCLUSION

  • The conclusion identifies richer three-dimensional obstacle representations and full-order, perception-aware safety modeling as future work.Source: §V · p. 8

APPENDIX

The appendix introduces notation for the line-of-sight frame and presents the DPCBF condition for the reduced-order model.

1) Common Notation:

  • The appendix omits the obstacle index j for clarity and introduces the line-of-sight angle and relative velocity expressed in the LoS frame.Source: 1) Common Notation · p. 9

2) DPCBF for the Reduced-Order Model:

3) DPCBF for the Dubins-Car Model:

The appendix specializes the DPCBF to a Dubins-car model with a stationary obstacle and then presents the C3BF comparison formulation.

3) DPCBF for the Dubins-Car Model:

4) Collision Cone CBF:

5) Distance-Based Exponential CBF:

This appendix section presents the distance-based barrier, establishes its relative degree for the reduced-order dynamics, and specifies second-order ECBF enforcement.

5) Distance-Based Exponential CBF:

B. Graph Encoding and Network Architecture

The humanoid navigation actor encodes a robot-centered graph with attention, then combines its embedding with a local observation for policy output. The section also specifies the critic’s privileged input and keeps the pretrained locomotion policy fixed during high-level training.

B. Graph Encoding and Network Architecture

  • During training, the graph includes the robot, the goal, and at most 10 selected obstacles, represented with node type, body-frame position and velocity, radius, and a validity mask.Source: B. Graph Encoding and Network… · p. 11
  • Robot-centered attention computes only the robot-node embedding, using edges from the robot to each obstacle or goal encoded by an 11-dimensional feature.The edge feature includes node-type labels, relative position, boundary distance, and relative velocity; validity masks are used only to mask attention logits for padded nodes.Source: B. Graph Encoding and Network… · p. 11
  • The attention mechanism constructs an intermediate message for each robot-to-node edge and uses attention weights to form the resulting robot-node embedding.Source: B. Graph Encoding and Network… · p. 11
  • The actor combines a 16-dimensional graph embedding with a 13-dimensional local observation, while the critic uses noise-free privileged observations and separate positive- and negative-value heads.The pretrained low-level locomotion policy remains fixed during high-level policy training.Source: B. Graph Encoding and Network… · p. 11

Related papers

Loading 2609.38873v2…