PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped

Jagannath Prasad Sahoo, Saurabh Kumar, Surya Prakash S. K., Samiran Datta, Abhay Dwivedi, Amit Shukla

arXiv:2610.05855v1cs.ROcs.AIcs.HCcs.SYeess.SY

TL;DR

Underactuated bipedal obstacle avoidance must account for balance coupling and the need to anticipate moving obstacles. The paper jointly trains a high-level velocity-command policy and a velocity-conditioned low-level gait policy, and reports 98.0% static and 88.0% dynamic success with path lengths within 4% of the shortest-path reference.

  • Problem

    Underactuation couples obstacle-avoidance maneuvers to balance, gait limits available motion, and moving obstacles require anticipation because a biped may not have time to step aside once danger is detected.

    Source: §I · p. 1

  • Method

    A jointly trained high-level SAC policy maps pose, raycast, dynamic-obstacle, and local-goal observations to velocity commands executed by a velocity-conditioned low-level SAC gait policy.

    Source: §I · p. 2

  • Results

    The proposed method achieves 98.0% success in static and 88.0% in dynamic scenarios, with path length within 4% of the shortest-path reference.

    Source: §VII · p. 6

  • Takeaways & Limitations

    Ablations establish that the TTC term, proximity penalty, posture shaping, and reward clip each contribute materially to the reported performance.

    Source: §VII · p. 6

    The navigation advantage is discussed for a platform with a gait-limited, asymmetric command set, where some planner-demanded turns are infeasible.

    Source: §VI · p. 5

Abstract

from arXiv · show

A bipedal robot cannot deviate from its path to avoid an obstacle without disturbing its balance, and this coupling is most severe on underactuated platforms such as the biped considered here, which has four actuated joints per leg and no hip or ankle roll. This paper presents a Hierarchical Reinforcement Learning (HRL) framework in which a High-Level (HL) policy observes the robot pose, 36 raycast proximity measurements, moving-obstacle states, and a receding-horizon local goal, and outputs a body-velocity command $(v_x, v_y, \omega_{yaw})$ every ten control steps, while a velocity-conditioned Low-Level (LL) policy tracks each command through PD-controlled joint targets. Both policies are trained jointly with Soft Actor-Critic (SAC). Because the converged gait is task-agnostic, it is frozen and driven by classical planners over the same command interface, yielding three controlled baselines: SAC+A*, SAC+RRT*, and SAC+APF. Across 100 evaluation trials per method in randomized PyBullet environments, the proposed method reaches the goal in 98.0% of static and 88.0% of dynamic trials, against at most 78.0% and 68.0% for the planner hybrids, with path lengths within 4% of the A* reference, and ablations confirm that each observation channel and reward term contributes materially to this performance.

I. INTRODUCTION

Obstacle avoidance is difficult for underactuated bipeds because steering perturbs balance, their gait limits available motion, and moving obstacles require anticipation. The paper addresses this with jointly trained navigation and gait policies, then evaluates the learned navigation layer against controlled planner hybrids.

I. INTRODUCTION

  • Underactuation couples avoidance to balance, gait limits make the velocity set asymmetric, and moving obstacles require anticipation before a biped can step aside.The platform has no roll joints, further restricting lateral motion.Source: §I · p. 1
  • Flat RL entangles navigation with gait, while decoupled planning ignores balance coupling and, for global planners, moving obstacles [1]–.The paper motivates separating navigation from gait without severing their feedback.Source: §I · p. 1, Figure 1 · p. 1
  • The task is to reach a goal at least 6.0 m away without collision or fall, with a known static map and moving-obstacle trajectories unknown in advance.The robot senses static geometry through proximity and moving obstacles through relative position and velocity, while generating joint actions at 50 Hz.Source: §I · p. 1
  • The jointly trained hierarchy maps high-level observations to body-velocity commands and uses a velocity-conditioned low-level policy to execute each command for ten control steps.The high-level policy uses pose, raycasts, dynamic-obstacle states, and a local goal; the low-level policy produces joint targets.Source: §I · pp. 1–2
  • The high-level observation and reward design covers static layouts and moving obstacles, with a time-to-collision term for anticipatory avoidance.The contributions also include an ablation of observation channels and reward components against success, collision, and failure rates.Source: §I · p. 2
  • The frozen learned gait enables controlled comparisons with SAC+A⋆, SAC+RRT⋆, and SAC+APF hybrids that differ only in their command generator.This holds the low-level gait fixed to isolate the navigation layer’s contribution.Source: §I · p. 2

II. RELATED WORK

Prior work separates classical obstacle planning from balance-aware locomotion, while RL and hierarchical methods have addressed parts of the combined problem. This work targets an underactuated biped and compares learned navigation with classical planners using the same learned gait.

II. RELATED WORK

  • Classical global planners compute paths on known maps and reactive planners respond online, but their assumed faithful velocity tracking decouples bipedal avoidance from balance –.Reactive methods can also suffer local minima and oscillation; CLF-CBF controllers restore real-time safety guarantees but retain a separate gait-tracking problem.Source: §II · p. 2
  • RL advanced navigation and legged locomotion largely in separate lines, while the cited flat-RL bipedal avoidance work uses fixed, known obstacle positions [1].Navigation policies typically assume a stable base, whereas locomotion policies do not avoid obstacles.Source: §II · p. 2
  • Unlike prior hierarchical locomotion studies, this work combines dynamic-obstacle states and raycasts for an underactuated biped and benchmarks navigation against planners driving the identical learned gait.Earlier hierarchies include options, HIRO, DeepLoco, quadruped navigation, and bipedal social navigation [20].Source: §II · p. 2

III. PRELIMINARIES AND PROBLEM FORMULATION

The paper models low-level walking at a fast control timescale and high-level navigation as a semi-MDP over held velocity commands. Both levels use entropy-regularized SAC, with entropy supporting exploration in their different control problems.

A. Two-timescale decision process

  • The low-level control problem is a discrete-time MDP operating at 50 Hz with discount factor γℓ=0.99.These low-level parameters are listed in Table I.Source: A. Two-timescale decision process · p. 2
  • Because the high-level policy acts every H=10 low-level steps with its command held constant, its decision process is a semi-MDP.The high-level reward accumulates per-step task reward over the macro-step.Source: A. Two-timescale decision process · p. 2
  • The macro-step formulation reduces the effective navigation horizon by a factor of H, stabilizing credit assignment for long-range goal reaching.The high-level state is s_k, its held command is u_k, and low-level actions are sampled under that command.Source: A. Two-timescale decision process · p. 2

B. Soft Actor–Critic

  • Both policy levels are trained with SAC to maximize an entropy-regularized return.The objective includes expected reward and policy entropy.Source: B. Soft Actor–Critic · p. 2
  • In the SAC objective, ρπ is the induced state–action distribution, H is policy entropy, and α is the temperature auto-tuned at both levels from 1.0.These terms specify the distribution, exploration measure, and its weighting in the objective.Source: B. Soft Actor–Critic · p. 2
  • Entropy-regularized SAC aids exploration in both the contact-rich low-level control space and the high-level navigation problem.The passage also characterizes off-policy SAC with twin critics as more sample-efficient than on-policy alternatives such as PPO [22].Source: B. Soft Actor–Critic · p. 2

IV. METHOD

The method uses a static-map path to set a moving local goal, while a low-level policy turns velocity commands into joint targets for walking.

A. Planning layer and local-goal generation

  • A global path on the known static map supplies a receding-horizon local goal; dynamic obstacles are left to the high-level policy, and the planning layer issues no motion commands.The local goal is a lookahead waypoint that advances with the robot and becomes the final goal near the destination.Source: A. Planning layer and local-goal… · p. 3

B. Low-level velocity-conditioned gait policy

C. High-level navigation policy

The high-level policy uses obstacle and local-goal observations to issue held body-velocity commands, with rewards balancing progress, gait feasibility, and collision avoidance.

  • High-level inputs include the robot-frame local goal and positions and velocities of the five nearest dynamic obstacles, alongside raycasts treated as privileged stand-ins for sensing.Source: C. High-level navigation policy · p. 3
  • The high-level command is held for ten steps at 5 Hz, and its reward combines a clipped shaping sum with terminal terms added after clipping.Source: C. High-level navigation policy · p. 3
  • Reward shaping covers reference-path progress and gait feasibility through posture, height, lateral drift, torque, and swing-clearance terms, alongside obstacle-distance and time-to-collision safety terms.Source: C. High-level navigation policy · p. 3
  • The TTC term penalizes predicted collisions before proximity alone would trigger; regularization discourages standing still, while collision penalties allow recovery and falls or goal arrival end episodes.Avoiding an obstacle can incur a cross-track cost that is recouped through the safety term; reward weights were manually tuned.Source: C. High-level navigation policy · p. 3

D. Joint two-timescale training

The high- and low-level policies train jointly at different timescales, allowing the gait and navigation policy to adapt to one another.

  • The low-level policy updates at 50 Hz, while the high-level policy updates at 5 Hz from macro-step returns; both train simultaneously in the same episodes.Source: D. Joint two-timescale training · p. 3
  • Joint training lets the gait adapt to the navigation policy’s command distribution and lets navigation learn the gait’s evolving feasibility limits, with the high-level policy acting as an automatic curriculum.Source: D. Joint two-timescale training · p. 3
  • After convergence, the low-level policy is frozen and reused by all planner-based baselines.Source: D. Joint two-timescale training · p. 3

E. Planner-based baselines over the shared interface

Planner-based baselines reuse the co-trained gait policy and vary the command generator, enabling comparison of navigation approaches through a shared interface.

V. EXPERIMENTS

Experiments evaluate the controller in PyBullet across static and dynamic obstacle scenarios, alongside joint-training curves under increasing difficulty.

A. Setup

  • Experiments used PyBullet simulation with seed 42 and 100 trials per method per condition across 10 unseen layouts.The robot has eight actuated joints; metrics include success, collision, fall/timeout, path length, and time to goal.Source: A. Setup · p. 4, §V · p. 4

B. Static and dynamic scenarios

  • Static tests used unseen layouts with start and goal at least 6.0 m apart; dynamic tests included up to five movers traveling at 0.10–0.30 m/s.Dynamic encounters included crossing and head-on trajectories; the HL slowed and issued yaw-dominant commands to let obstacles pass before re-accelerating.Source: B. Static and dynamic scenarios · p. 4, Figure 3 · p. 4
  • Dynamic-obstacle training converged more slowly because the policy additionally had to learn velocity-aware avoidance through the TTC term.Fig. 5 plots mean evaluation return under three difficulty conditions, with per-evaluation variability from a single seed-42 run shown as shaded bands.Source: B. Static and dynamic scenarios · p. 4

C. Comparison with planner-based hybrids

The learned high-level policy outperformed planner hybrids in reported success rates, while path lengths remained close to the A* reference. Ablations show that observation and reward components affect safety and training stability.

C. Comparison with planner-based hybrids

D. Ablation study

  • Removing dynamic-obstacle velocity or position inputs degraded both regimes; removing TTC or proximity penalties had a comparable effect.The ablations indicate that velocity-aware anticipation and proximal safety signals both matter to performance.Source: D. Ablation study · p. 4
  • Removing posture terms shifted failures toward falls, while removing the shaping clip destabilized training with 90.0% fall/timeout in static trials.Across variants, fall or timeout remained the dominant failure mode rather than collision.Source: D. Ablation study · pp. 4–5

C. Comparison with planner-based hybrids

VI. DISCUSSION AND LIMITATIONS

The learned navigation layer couples obstacle avoidance to the biped’s gait limits and anticipates moving obstacles. It achieves high success in static and dynamic scenarios, while ablations show that multiple components contribute to performance.

VI. DISCUSSION AND LIMITATIONS

  • The navigation layer limits commands to what the gait can execute, while pure-pursuit trackers can demand infeasible turns that lead to falls and timeouts.Joint training and posture terms help internalize the platform’s gait-limited, asymmetric command set.Source: §VI · p. 5
  • TTC-based anticipation lets the robot yield before proximity alone would trigger, whereas global planners may encounter moving obstacles only at replanning instants.The biped needs several steps to change heading, limiting reactive avoidance time.Source: §VI · p. 5
  • Ablations shift failures toward falls or timeouts rather than collisions, indicating that the hierarchy’s coupled structure—not one term alone—supports its advantage.Fig. 6 covers observation-channel and reward-term ablations in static and dynamic environments.Source: §VI · p. 5, Figure 6 · p. 5
  • Evaluation is simulation-only, with privileged raycasts and obstacle states standing in for onboard sensing; seed variance and sim-to-real transfer remain unaddressed.The study also assumes a known static map and actuates only leg joints.Source: §VI · p. 5

VII. CONCLUSION

  • 98.0% static and 88.0% dynamic success were achieved, with path lengths within 4% of the shortest-path reference.These are the proposed method’s reported results.Source: §VII · p. 6
Loading 2610.05855v1…