PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion

Deepali Jain, Atil Iscen, Ken Caluwaerts

arXiv:2011.11722v1cs.ROcs.CVcs.LG

TL;DR

Legged robots must combine visual perception with fast, contact-rich locomotion to navigate complex terrain. This paper trains a hierarchical policy that links camera input to low-level leg control through a latent command, outperforming non-hierarchical policies across three navigation tasks while allowing skill transfer and lower-frequency visual processing.

  • Problem

    Visual control makes legged locomotion harder because robots must perceive the environment while handling fast, contact-rich leg dynamics.

    Source: §1 · p. 1

  • Method

    The method concurrently trains a vision-processing high-level policy and a sensor-driven low-level controller connected by a latent command.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The hierarchical method outperformed non-hierarchical baselines on three navigation tasks and achieved higher data efficiency and lower wall-clock training time.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Decoupling the levels allows variable-frequency visual processing and transfer of low-level skills between tasks and environments.

    Source: §5 · p. 7, §7 · p. 8

    Hardware results were not included because COVID-19 restrictions prevented access to the lab.

    Source: §7 · p. 8

Abstract

from arXiv · show

Legged robots navigating crowded scenes and complex terrains in the real world are required to execute dynamic leg movements while processing visual input for obstacle avoidance and path planning. We show that a quadruped robot can acquire both of these skills by means of hierarchical reinforcement learning (HRL). By virtue of their hierarchical structure, our policies learn to implicitly break down this joint problem by concurrently learning High Level (HL) and Low Level (LL) neural network policies. These two levels are connected by a low dimensional hidden layer, which we call latent command. HL receives a first-person camera view, whereas LL receives the latent command from HL and the robot's on-board sensors to control its actuators. We train policies to walk in two different environments: a curved cliff and a maze. We show that hierarchical policies can concurrently learn to locomote and navigate in these environments, and show they are more efficient than non-hierarchical neural network policies. This architecture also allows for knowledge reuse across tasks. LL networks trained on one task can be transferred to a new task in a new environment. Finally HL, which processes camera images, can be evaluated at much lower and varying frequencies compared to LL, thus reducing computation times and bandwidth requirements.

1 Introduction

The paper addresses the challenge of combining visual perception with fast, contact-rich leg dynamics by concurrently learning hierarchical vision and control policies. It reports improved navigation-task performance and training efficiency, alongside transferable low-level skills and lower-frequency visual processing.

1 Introduction

  • The concurrently trained high and low levels learn a pixels-to-motor-commands pipeline without requiring hand-designed or learned low-level behaviors.The high level processes vision and issues latent commands; the low level controls the legs at a higher frequency.Source: §1 · pp. 1–2
  • Low-level behaviors are largely task agnostic and transferable, and transferring them from a solved task can improve data efficiency even across environments.The paper describes reusing low-level policies for tasks of a similar nature.Source: §1 · p. 2
  • The high level processes visual inputs at 1.5 Hz–10 Hz, compared with 500 Hz for the low level, reducing computation and bandwidth requirements.The high level's variable frequency is computed by its neural network.Source: §1 · p. 2
  • The latent command forms an information bottleneck through which the high level extracts useful visual information and the low level learns task-relevant locomotion primitives.The paper reports analyzing the learned behaviors and selecting locomotion primitives through a low-dimensional latent command.Source: §1 · p. 2
  • The hierarchical policies outperformed non-hierarchical baselines on three visual navigation tasks, with increased sample efficiency and lower wall-clock training time.The introduction also reports that running the high level at low frequency reduced inference time and computation cost with minimal effect on task performance.Source: §1 · p. 2

2 Background and Related Work

Prior work decomposes decision-making hierarchically, but training useful low-level behaviors can require difficult-to-obtain reference data or auxiliary objectives. This work instead learns both levels from the task reward, applying the approach to vision-based quadruped locomotion.

2 Background and Related Work

  • Options, MAXQ value decomposition [5], and Hierarchical Abstract Machines are established HRL frameworks, but designing or training low-level policies for complex robotics tasks is challenging.Some approaches imitate reference data [7] [8], which depends on high-quality datasets that are often difficult to obtain.Source: §2 · p. 2
  • Concurrent-learning approaches include sub-goal-conditioned low levels [10] [11] [12], auxiliary rewards for skill diversity [15] [16], intrinsic rewards [17], and finite option sets [17, 18, 19].The paper notes that sub-goal conditioning may not suit high-dimensional camera observations [13], while auxiliary rewards can encourage irrelevant skills.Source: §2 · p. 2
  • The method trains both levels with gradient-free policy search from the main task reward, avoiding external priors, and uses a vector-space interface for a continuum of skills [1].The authors state that a continuum of skills is required for agile locomotion tasks.Source: §2 · p. 2
  • Prior legged-locomotion work includes separately learned levels and selecting among pre-trained skills [10]; this work learns locomotion from vision without designing or pre-training the low level.Vision-based robotics has also used hand-designed pipelines, while this method learns navigation directives and dynamic legged skills from vision.Source: §2 · pp. 2–3
  • The authors use Augmented Random Search (ARS) to optimize neural-network policies, choosing evolutionary methods because derivative-free approaches suit their hierarchical architecture.They place ARS among evolutionary methods previously applied in robotics [2] [21] [34] [35] [36].Source: §2 · p. 3

3 Method

The method links a vision-processing high-level policy to a proprioception-based low-level controller through a latent command and trajectory generator. The two policy levels are trained jointly, and a trained low-level policy can be held fixed while a new high-level policy is learned.

3 Method

  • A CNN high level maps depth-camera observations to a latent command and execution duration; a linear low level combines that command with proprioception and trajectory-generator state to produce motor residuals and trajectory parameters.The residual motor commands are added to the trajectory generator's output before actuation.Source: Figure 1 · p. 3, §3 · p. 3
  • During execution, the high level receives exteroceptive observations and periodically issues commands, while the low level uses IMU values, motor angles, and trajectory-generator state to control the robot.The high level is invoked again after its selected duration; the low level's outputs are combined with trajectory-generator actions.Source: §3 · pp. 3–4
  • The method jointly optimizes both policy levels to maximize expected episode return, using Augmented Random Search to estimate return gradients and perform gradient ascent.The hierarchical policy's parameter vector comprises the high- and low-level weights.Source: §3 · p. 4
  • For transfer, the trained low-level weights are kept fixed and the high-level weights are reinitialized and updated on the new task, reusing low-level skills across tasks and environments.The paper reports this approach is faster than learning from scratch on each task.Source: §3 · p. 4

4 Experimental Setup

The experiments use a simulated Laikago quadruped in a curved-cliff environment and a pillar-filled maze, with hierarchical policies built around vision and proprioceptive control.

Experimental Setup

  • The robot is a 12-degree-of-freedom Laikago simulated in PyBullet, with each leg controlled by a PD position controller.The tasks are set in a curved cliff and a maze.Source: §4 · p. 4

Curved Cliff Environment

  • In the curved-cliff task, the robot uses a first-person depth camera to steer along a randomized curve while maximizing forward velocity and avoiding the edge.The reward is capped velocity along the x direction.Source: §4 · p. 5

Maze Environment

  • The maze experiments include traversal away from the origin and goal finding at a randomly selected corner, with goal reward blending maze traversal and goal-directed progress.The goal-finding reward emphasizes traversal near the origin and goal progress as the robot approaches its destination.Source: §4 · p. 5

Experimental Setup

  • Episodes end upon loss of balance, falls, collisions, or 6,000 low-level control steps; goal-finding episodes also end within 0.5 m of the goal.The velocity cap is set to 0.002 m per time step, corresponding to 1 m/s.Source: §4 · p. 5

Solution Implementation details

  • A CNN-based HL converts a 16 × 16 depth image into a latent command and duration, while an LL combines that command with robot sensors to generate motor commands.The HL duration spans 50–300 time steps; the HL and LL have about 3,000 and 300 parameters, respectively.Source: §4 · p. 5
  • ARS training evaluates policy perturbations on 64 parallel workers and averages returns across three episodes per perturbation.Hyperparameters are selected using Gaussian-process bandit tuning.Source: §4 · p. 6

5 Experimental Results

The hierarchical policies complete three visual navigation tasks, with the high level adapting its decision frequency to the route. The experiments also compare learning curves, train larger CNN policies, and test low-level skill transfer across tasks.

5 Experimental Results

  • The hierarchical policies complete all three visual navigation tasks from vision, varying HL decision frequency with route demands and turning in place to face the goal.HL decisions are more frequent at sharp cliff turns and sparser on straighter sections; goal-facing decisions are also sparser when the goal is close.Source: §5 · p. 6
  • The study compares HRL and non-hierarchical CNN learning curves across three tasks, plotting return against training episodes with shaded standard deviations.The plotted averages use approximately 450 environment episodes.Source: §5 · p. 6, Figure 4 · p. 6

Learning HRL policies

  • ARS can train larger CNN policies with over 10^5 weights for maze traversal and goal finding, using 32 × 32 images and two additional 32D fully connected layers.Most experiments use policies with approximately 10^3 weights.Source: §5 · p. 7, Figure 5 · p. 7

Transferring Low Level Policies between Tasks

  • Transferring a frozen LL trained for goal finding improves learning efficiency on maze traversal and curved-cliff tasks, while LLs trained on the simpler cliff task transfer poorly to maze tasks.The goal-finding LL uses proprioceptive sensors and learns generic steering and turning-in-place primitives.Source: §5 · p. 7

6 Analysis of Hierarchical Policies

The analysis examines the latent command space and the effect of high-level execution frequency. Temporally abstracted policies retain similar performance while running inference faster.

Analysing 2D Latent Command Space

  • The 2D latent command space smoothly transitions among steering behaviors with varying velocities.The analysis evaluates LL trajectories generated from a uniform grid of artificial latent commands.Source: §6 · p. 7, Figure 7 · p. 7

Influence of Latent Command Dimension

  • On the curved cliff task, 1D latent commands are too restrictive, while 2D, 4D, and 8D spaces perform similarly.The 2D space reaches optimal performance while remaining easier to visualize, control, and transfer.Source: Figure 8 · p. 7, §6 · p. 8

Impact of High Level Frequency on Training and Inference

  • ≈100 times faster inference is reported for temporally abstracted policies, while the variants learn with only minor performance differences.The comparison covers fixed HL intervals of 1, 50, 150, and 300 time-steps, plus a variable interval.Source: §6 · p. 8, Figure 9 · p. 8

7 Discussion and Conclusion

The paper reports that hierarchical policies outperform non-hierarchical baselines on three quadruped navigation tasks. It also highlights variable-frequency HL processing and LL policy transfer, while noting that hierarchical policies were not evaluated on hardware.

7 Discussion and Conclusion

  • Hierarchical policies outperform non-hierarchical baselines on three navigation tasks and achieve higher data efficiency and lower wall-clock training time.The HL and LL can run at different frequencies, and the HL learns when to process a new image.Source: §7 · p. 8
  • LL policies transfer between tasks, and the learned skills could serve as a continuous low-level action space for other learning algorithms.The paper identifies this use by other algorithms as future work.Source: §7 · p. 8
  • COVID-19 restrictions prevented hardware results for the hierarchical policies; implementation on a real Laikago was planned.The authors report prior validation of similar LL policies and tests of the vision stack with predefined skills.Source: §7 · p. 8

A Feasibility of Deploying Our Learned Policies on the Robot

The appendix describes prior real-robot deployment of a forward-walking policy and reports that hierarchical policies transferred between simulated Gibson spaces.

A Feasibility of Deploying Our Learned Policies on the Robot

1 Feasibility of Deploying Our Learned Policies on the Robot

The appendix reports earlier deployment of a simulation-trained forward-walking policy on the real Laikago and transfer of trained policies to a new simulated space.

1 Feasibility of Deploying Our Learned Policies on the Robot

  • The figures show real depth-camera images, forward walking on hardware, and navigation in Gibson environments.Source: Figure 1 · p. 13, Figure 2 · p. 13, Figure 3 · p. 13
  • A simulation-trained forward-walking policy tracking desired velocity was previously deployed on the real Laikago.The hierarchical policies used similar training infrastructure, so the authors expected transfer to hardware.Source: §11 · p. 13, Figure 2 · p. 13
  • Policies trained in simulated 3D spaces with Gibson visuals transferred to a new space.Source: §11 · p. 13, Figure 3 · p. 13
Loading 2011.11722v1…