PaperlayerView exact revision on arXiv

Source-linked AI summary

Energy-Efficient Gait Adaptation via Hierarchical Reinforcement Learning for Quadrupedal Locomotion Across Diverse Terrains

Ammar Issa, Anubhav Singh, Anton Tsaritsin, Sergey Kolyubin

arXiv:2610.10297v1cs.ROcs.LG

TL;DR

Quadruped locomotion must balance energy efficiency and robust performance across speeds and terrains, while end-to-end RL policies couple gait, motion, and energy objectives. This paper uses hierarchical RL to separate high-frequency tracking and stabilization from low-frequency gait adaptation; its reported evaluations include improved tracking after low-level finetuning and terrain-dependent gait adjustment.

  • Problem

    Quadruped robots must achieve low energy consumption while maintaining stable locomotion across speeds and terrains, and end-to-end RL policies couple gait, motion, and energy objectives in reward design.

    Source: §I · p. 1

  • Method

    A hierarchical RL framework separates high-frequency velocity tracking and stabilization from low-frequency, continuously modulated gait parameters with a CoT-related high-level objective.

    Source: §I · pp. 1–2

  • Results

    0.165 to 0.039 m/s on flat terrain and 0.182 to 0.047 m/s on rough terrain: overall velocity-tracking RMSE decreases after low-level finetuning.

    Source: C. Ablation Study · p. 7

  • Takeaways & Limitations

    During uphill traversal at 1.0 m/s, realized body height and stance length decrease while body pitch and stance width increase before settling into a different operating region.

    Source: 1) Gait Adaptation Behavior · p. 6

Abstract

from arXiv · show

While energy efficiency is a critical objective for legged-robot locomotion control, achieving low energy consumption while maintaining robust performance across different velocity ranges and terrain conditions remains a key challenge. This is particularly true for end-to-end RL policies, where gait generation, motion execution, and energy optimization are tightly coupled, leading to high sensitivity to reward design. In this work, we propose a hierarchical reinforcement learning (HRL) framework that separates a high-frequency policy for stable and robust joint-level motion execution from low-frequency gait adaptation that explicitly minimizes the cost of transport (CoT). The three-stage Isaac-based training procedure enables zero-shot sim-to-real transfer with improved tracking accuracy, robustness, and energy efficiency. The learned hierarchy exhibits automatic speed-dependent gait adaptation, transitioning from pacing at low speeds to trotting at higher speeds. We validate the proposed approach in simulation against representative single-policy and hierarchical locomotion baselines, demonstrating reduced CoT over a broad range of commanded velocities, while maintaining robust locomotion across flat, uneven rough, and inclined terrains. We further demonstrate its practical feasibility through zero-shot deployment on a physical Unitree AlienGo quadruped.

I. INTRODUCTION

Quadruped robots must balance stable locomotion and low energy use across speeds and terrains, while end-to-end RL policies couple gait generation, motion execution, and energy objectives. The proposed hierarchy separates high-frequency tracking and stabilization from low-frequency gait adaptation, with a high-level CoT objective.

I. INTRODUCTION

  • Different gaits become energetically favorable at different speeds, motivating adaptive gait selection for quadruped robots.Long-duration and energy-constrained tasks make energy efficiency important, while nonlinear dynamics, intermittent contacts, and speed-dependent gait energetics complicate the stability–energy trade-off.Source: §I · p. 1
  • End-to-end RL policies jointly optimize gait generation, stabilization, tracking, and energy efficiency, making them sensitive to reward weighting and less interpretable.Prior hierarchical approaches also use discrete switching, predefined gait libraries, or model-based low-level controllers.Source: §I · p. 1
  • The framework separates high-frequency velocity tracking and stabilization from low-frequency gait adaptation, with post-training finetuning.The high-level policy continuously modulates interpretable gait parameters, while the low-level policy executes motion.Source: §I · pp. 1–2
  • The high-level objective explicitly incorporates CoT-related energy optimization for continuous gait-parameter modulation.This enables gait adaptation without predefined gait libraries or manually designed transition rules.Source: §I · pp. 1–2
  • The evaluation covers tracking and CoT across speeds and terrains and includes zero-shot deployment on a physical Unitree AlienGo.Representative behaviors include flat-ground walking, slope traversal, and speed-dependent modulation from pacing at lower speeds to trotting at higher speeds.Source: Figure 1 · p. 1, §I · p. 2

II. HIERARCHICAL REINFORCEMENT LEARNING FRAMEWORK

The framework decomposes locomotion control across two temporal scales: a low-frequency policy selects gait parameters, and a high-frequency controller executes joint-level motion. The low-level policy is trained independently and kept frozen during high-level training, separating stabilization from long-horizon gait and energy decisions.

A. Problem Formulation

  • The policy is decomposed into a low-frequency high-level policy and a high-frequency low-level policy to account for gait adaptation and joint-level control at different temporal scales.The high-level command g_k is held for H consecutive low-level control steps; the low-level observation includes joint states, base orientation, and phase-based timing signals.Source: A. Problem Formulation · p. 2

B. Hierarchical Control Architecture

  • The high-level policy outputs a structured gait-parameter vector at a lower frequency, while the low-level controller computes joint-level commands at a higher frequency.Each gait command remains fixed for H low-level control steps, after which aggregated locomotion statistics are used to update the high-level policy.Source: B. Hierarchical Control Architecture · p. 2, Figure 2 · p. 2
  • The low-level policy is trained independently for robust velocity tracking and dynamic stabilization, then frozen during high-level training.This isolates contact-rich dynamics in the low-level controller and leaves long-horizon gait modulation and energy optimization to the high-level policy.Source: B. Hierarchical Control Architecture · p. 2

C. High-Level Gait Parameterization

The high-level policy modulates structured gait parameters at a slower timescale than joint-level control. Its reward combines energy efficiency with gait smoothness and low-level performance, separating long-horizon adaptation from stabilization.

C. High-Level Gait Parameterization

  • The high-level state summarizes realized velocity and body-frame gravity over each low-level execution window, while commanded velocities are not explicit state inputs.The asymmetric critic receives commanded gait parameters to improve value estimation of CoT-related signals.Source: C. High-Level Gait Parameterization · p. 2
  • Training proceeds through low-level pretraining, energy-aware high-level training with the low-level policy frozen, then low-level finetuning with the high-level policy frozen.The hierarchy reuses a pretrained low-level controller for joint-based execution.Source: Figure 3 · p. 3, C. High-Level Gait Parameterization · p. 3
  • The gait command includes leg-timing offsets, stepping frequency, body pose, stance geometry, foot-swing height, and stance duration.Timing offsets determine inter-leg phase relationships and can encode pronking, trotting, pacing, or bounding.Source: C. High-Level Gait Parameterization · p. 3
  • Structured gait parameters let the high-level policy modulate gait behavior while the low-level policy handles joint-level execution.Continuous timing offsets encode canonical gaits and allow interpolation between them.Source: C. High-Level Gait Parameterization · p. 3
  • The high-level reward combines energy efficiency, gait smoothness, and accumulated low-level performance.The low-level reward discourages unstable or poorly tracked energy minimization, while the smoothness term penalizes abrupt gait-command changes.Source: C. High-Level Gait Parameterization · p. 3

D. Low-Level Locomotion Policy

The low-level policy receives robot-state observations, gait commands, and phase clocks, while gait-derived contact targets shape training rewards rather than directly prescribing foot trajectories. Its reward combines velocity tracking, gait-specific auxiliary terms, and stable, smooth locomotion objectives.

D. Low-Level Locomotion Policy

  • The low-level observation combines commanded velocities and gait parameters with joint-position offsets, joint velocities, and four phase-clock signals.The policy uses a history of five observations.Source: D. Low-Level Locomotion Policy · p. 3
  • Timing parameters are converted into leg-wise phases and sinusoidal clock inputs, while soft stance/swing targets are used for reward shaping only.The contact targets are not policy inputs and do not directly prescribe Cartesian foot trajectories.Source: D. Low-Level Locomotion Policy · p. 3, Figure 4 · p. 4
  • The low-level reward groups velocity tracking, gait-enforcing auxiliary rewards, and fixed rewards for stable and smooth locomotion.Augmented terms track commanded orientation, foot-swing height, body height, per-foot contact schedule, and foothold placement.Source: D. Low-Level Locomotion Policy · pp. 3–4
  • Contact-force and foot-velocity magnitudes shape contact rewards, and a Raibert-style heuristic generates desired footholds from commanded gait parameters.The realized foot position is distinguished from the desired foothold; detailed reference construction follows.Source: D. Low-Level Locomotion Policy · p. 4

E. Training Pipeline

The three-stage training pipeline separates low-level locomotion learning from high-level gait adaptation, then fine-tunes the low-level policy on complex terrain. The learned high-level policy adapts gait with speed, from pacing at low speeds to trotting at higher speeds.

E. Training Pipeline

  • The pipeline trains the low-level policy, trains the high-level policy with the low-level policy frozen, then fine-tunes the low-level policy with the high-level policy frozen.The final stage targets tracking robustness on complex terrain without altering the high-level policy’s learned energy optimization behavior.Source: E. Training Pipeline · pp. 4–5
  • Stage 1 trains the low-level policy on flat terrain for stable locomotion and accurate tracking of velocity and gait commands.It uses the MoB conditional-policy framework with reinforcement learning and curriculum learning.Source: E. Training Pipeline · p. 4
  • The high-level policy exhibits pacing at low speeds, a four-beat walking gait at moderate velocities, and trotting at higher speeds.The three-stage framework is presented as supporting sample efficiency, stable simulation training, and zero-shot transfer to a physical quadruped.Source: E. Training Pipeline · p. 4, Figure 5 · p. 5
  • Stage 2 freezes the low-level policy and trains the high-level policy on rough and inclined terrain to modulate gait parameters with energy efficiency in view.The high-level policy updates less frequently than the low-level policy.Source: E. Training Pipeline · p. 5
  • Stage 3 freezes the high-level policy while fine-tuning the low-level policy on complex terrain to improve tracking robustness.This stage does not alter the energy optimization behavior learned by the high-level policy.Source: E. Training Pipeline · p. 5

A. Experimental Setup

Experiments primarily use the Unitree AlienGo for simulation training and real-world validation, with additional Unitree Go1 simulation experiments for direct comparison with prior benchmarks.

A. Experimental Setup

1) Simulation Environment and Training Details:

Policies are trained in Isaac Gym, with the high-level policy learning gait modulation over a frozen low-level controller. Evaluation spans commanded velocities and multiple terrain conditions, measuring energy efficiency, tracking, and robustness.

1) Simulation Environment and Training Details:

  • All policies use PPO in Isaac Gym, with 4096 parallel environments for data collection and policy optimization.Second- and third-stage terrains include flat ground, rough terrain with heights sampled from 0–10 cm, and slopes with randomly selected inclinations.Source: 1) Simulation Environment and Training… · p. 5

2) Evaluation Protocol:

  • Performance metrics are computed after an initial transient period to exclude startup effects.Source: 2) Evaluation Protocol · p. 5
  • Evaluation covers commanded forward velocities and flat, inclined, and uneven rough terrain, including uphill and downhill slopes.The robot was trained on inclinations up to 20°; all methods use identical terrain configurations and command distributions.Source: 2) Evaluation Protocol · p. 5
  • Energy efficiency is measured with cost of transport, while velocity tracking error assesses locomotion accuracy and robustness.The evaluation passage defines joint mechanical power as P_i = τ_i q̇_i.Source: 2) Evaluation Protocol · p. 5
  • Robustness testing aggregates 500 simulation runs for each terrain and velocity interval, with friction and restitution randomized uniformly from 0 to 1.The reported falling rates are organized across terrain types and commanded-velocity intervals; the text attributes higher low-speed slope fall rates likely to reduced momentum and longer support phases.Source: 2) Evaluation Protocol · p. 5, Table II · p. 5

B. Experimental Evaluation

The evaluation describes speed-dependent gait changes and terrain-responsive gait geometry. Its CoT figures cover terrain and energy-reward ablations as well as a comparison with locomotion baselines, though the supplied passages do not report those comparison outcomes.

B. Experimental Evaluation

  • The high-level policy shifts from pacing at low speeds toward trotting as velocity increases, without explicit gait-switching rules.The low-level policy preserves stable locomotion execution during this gradual contact-pattern transition.Source: 1) Gait Adaptation Behavior · p. 5
  • Fig. 6 evaluates CoT through AlienGo terrain and energy-reward ablations and a Go1 comparison with existing locomotion baselines.The supplied figure and table captions identify these comparisons but provide no outcome values.Source: Figure 6 · p. 6, Table III · p. 6
  • At 1.0 m/s, traversal from flat ground onto a 20° uphill slope decreases realized body height and stance length while increasing body pitch and stance width.The parameters settle around a different operating region after the terrain-transition phase.Source: 1) Gait Adaptation Behavior · p. 6, Figure 7 · p. 6
  • The hierarchy adapts timing-related gait parameters primarily with velocity and adjusts realized posture and stance geometry with terrain changes.The described timing parameters include stepping frequency, inter-leg timing offsets, and stance duration.Source: 1) Gait Adaptation Behavior · p. 6

2) Cost of Transport:

The proposed HRL framework achieves lower CoT than benchmark methods across much of the evaluated velocity range and maintains locomotion across terrain conditions. Zero-shot deployment and ablations report successful real-world trials and benefits from terrain exposure, the high-level energy objective, and low-level finetuning.

2) Cost of Transport:

  • On simulated AlienGo, CoT decreases from low to moderate velocities and remains stable at higher speeds, with limited variation under terrain disturbances.This result is reported for flat and rough terrain across the evaluated velocity range.Source: 2) Cost of Transport · p. 6
  • Across mid-to-high velocities, HRL achieves lower CoT than WTW, AER (α = 1), and F&E; at lower speeds it beats WTW and F&E, while AER has lower energy consumption.The interval-wise comparison is summarized in Table III.Source: 2) Cost of Transport · pp. 6–7

3) Real-World Deployment:

  • Zero-shot AlienGo deployment completed all 10 trials on flat, inclined, and uneven rough terrain without falls, including traversal of slopes up to 20°.The deployed gait transition–velocity relationship remained consistent with simulation, with smooth adaptation across commanded velocities.Source: 3) Real-World Deployment · p. 7
  • On AlienGo, an AER-style policy with α = 0.5 crouched because reduced joint effort left torques insufficient to support the robot, illustrating reward-weight sensitivity.The authors contrast this with their formulation, which confines energy optimization to interpretable gait parameters while preserving low-level stabilization.Source: 3) Real-World Deployment · p. 7

C. Ablation Study

  • With high-level training only on flat terrain, rough-terrain CoT in (0, 0.5] m/s rises from 1.846 to 5.029; complex-terrain training maintains comparable CoT on flat and rough surfaces.The reported ablation attributes improved generalization of high-level gait adaptation to exposure to terrain variations during training.Source: C. Ablation Study · p. 7
  • Removing the high-level energy term substantially increases CoT across velocity intervals and both terrains; accumulated low-level reward preserves locomotion but does not encourage energy-efficient gait selection.The ablation supports the explicit high-level energy objective as essential for reducing CoT.Source: C. Ablation Study · p. 7
  • Stage 3 finetuning reduces velocity-tracking RMSE from 0.165 to 0.039 m/s on flat terrain and from 0.182 to 0.047 m/s on rough terrain.The gain is most pronounced above 1.0 m/s; finetuning optimizes the low-level policy under gait commands generated by the frozen high-level policy.Source: C. Ablation Study · p. 7, Figure 9 · p. 7

IV. DISCUSSION AND CONCLUSION

The conclusion reports lower average CoT than three benchmark methods and speed-dependent gait adaptation without predefined schedules or discrete switching. It also identifies slower high-level updates, training cost, decimation sensitivity, and low-level controller robustness as limitations.

IV. DISCUSSION AND CONCLUSION

  • Average CoT is reduced by 27.7%, 33.5%, and 70.5% compared with AER, WTW, and F&E, respectively.The hierarchy transitions from pacing at low speeds to trotting at higher speeds without predefined gait schedules or discrete switching.Source: §IV · p. 7
  • The reported scope includes robust, energy-efficient locomotion across flat, rough, and inclined terrains, with terrain-dependent adjustment of realized body posture and stance geometry.Representative real-world trials span flat terrain, an inclined slope, and uneven rough terrain.Source: §IV · p. 7, Figure 10 · p. 8
  • Low high-level update frequency may reduce responsiveness to rapid terrain changes; training cost, decimation sensitivity, and low-level execution or estimation errors also constrain the framework.Errors in the low-level controller may propagate through the hierarchy.Source: §IV · p. 8
  • Future work proposes vision-based terrain perception for more challenging environments, while noting that expanding the high-level policy's scope may increase energy consumption.The stated objective is to extend traversability while preserving the low CoT of the current proprioceptive formulation.Source: §IV · p. 8
Loading 2610.10297v1…