Legged Locomotion in Challenging Terrains using Egocentric Vision

Ananye Agarwal; Ashish Kumar; Jitendra Malik; Deepak Pathak

Source-linked AI summary

Legged Locomotion in Challenging Terrains using Egocentric Vision

Ananye Agarwal, Ashish Kumar, Jitendra Malik, Deepak Pathak

arXiv:2211.07638v1Nov 14, 2022cs.ROcs.AIcs.CVcs.LGeess.SY

TL;DR

The paper addresses visually guided locomotion over challenging terrain without relying on elevation maps or foothold planning. It trains a recurrent depth-based controller through cheap scandot reinforcement learning followed by supervised distillation, and demonstrates real-world traversal across stairs, curbs, gaps, and stepping stones on a small quadruped.

Problem
The paper targets locomotion over challenging terrain, where conventional elevation mapping and foothold planning are limited by noise, specialized hardware, and biological implausibility.
Method
The system trains with reinforcement learning on cheap scandots, then distills the policy through supervised learning into a recurrent controller using onboard depth and proprioception.
Results
The deployed policy traverses stairs, curbs, gaps, stepping stones, and varied natural terrain using a single front-facing depth camera on a quadruped robot.
Takeaways & Limitations
End-to-end coupling of egocentric vision and motor control enables terrain traversal without an explicit elevation map or foothold optimization objective.
Takeaways & Limitations
Failures can occur when simulation and real-world vision or terrain differ, requiring the situation to be engineered into simulation and the system retrained.

Abstract

from arXiv · show

Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-facing depth camera. The small size of the robot necessitates discovering specialized gait patterns not seen elsewhere. The egocentric camera requires the policy to remember past information to estimate the terrain under its hind feet. We train our policy in simulation. Training has two phases - first, we train a policy using reinforcement learning with a cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning. The resulting policy transfers to the real world and is able to run in real-time on the limited compute of the robot. It can traverse a large variety of terrain while being robust to perturbations like pushes, slippery surfaces, and rocky terrain. Videos are at https://vision-locomotion.github.io

1 Introduction

The paper develops egocentric visual locomotion for challenging terrain, using recurrent memory and learned gait emergence to control a small quadruped without conventional elevation-map planning.

Motivation: Vision becomes necessary for locomotion over challenging terrain such as stairs, rugged rocks, and stepping stones, beyond local-direction blind walking.The paper frames visually guided foot placement as tightly coupled to motor control in humans and robots.
Design principles: A recurrent policy uses recent egocentric views, proprioception, and action history to estimate terrain that is no longer visible beneath the robot or hind feet.This short-term memory addresses the camera’s front-facing viewpoint and delayed foot placement.
Learned gait emergence: The small robot faces stair obstruction while ascending and toppling risk while descending, addressed by a large hip-abduction motion that emerges during training.Figure 2 illustrates the terrain-specific behavior developed by the controller.
Learned gait emergence: Without pre-programmed gait priors, emergent behaviors let the 28cm-hip-height robot traverse stairs up to 25cm high, or 89% of its height.The authors attribute this capability to learning gaits directly rather than relying on predefined reference motions.
System constraints: The system uses a single front-facing depth camera and computes actions in one feedforward pass without tracking, avoiding real-time MPC or localization optimization.These design choices target the robot’s limited sensing and onboard compute.
Contribution: The approach couples egocentric vision directly with motor control through end-to-end learning rather than constructing a metric elevation map.The paper identifies this coupling and learning “all the way” as distinguishing aspects of the approach.

2 Method: Legged Locomotion from Egocentric Vision

The method trains terrain-aware locomotion in two stages: reinforcement learning with cheap scandots, followed by supervised distillation into a depth-and-proprioception policy with recurrent memory.

Training pipeline: The two-phase pipeline uses scandots for reinforcement learning, then distills the resulting policy into a depth-based recurrent policy using supervised action imitation.Scandots provide queried terrain heights cheaply, avoiding the cost of rendering depth throughout reinforcement-learning training.
Phase 2: Supervised Learning: The phase-2 policy uses onboard depth and proprioception to infer terrain history and directly predict target joint angles at 50Hz.The recurrent model implicitly tracks terrain beneath the robot from the front-facing camera’s history.
Architectures: The monolithic architecture maps raw observations through a GRU to joint angles, whereas RMA estimates terrain and extrinsics latents for a feedforward base policy.RMA separates latent estimation from control, while the monolithic design implicitly stores these factors in its weights.
Phase 1: Reinforcement Learning from Scandots: Phase 1 uses PPO without gait priors, with energy and hardware-damage penalties allowing stable gaits to emerge across varied terrains.The training environments vary terrain type and difficulty and use randomized parameters and observation noise for robustness.
Phase 2: Supervised Learning: DAgger with truncated backpropagation through time trains the student from phase-1 actions, minimizing action error over simulator rollouts of N = 24 timesteps.The RMA variant instead trains estimators for terrain and extrinsics latents while reusing the phase-1 base policy.
Deployment: The deployed student runs on available onboard compute and handles camera failures, asynchronous depth, pushes, slippery surfaces, rocky terrain, stairs, curbs, gaps, and stepping stones.These capabilities follow from the deployment policy trained with randomized conditions and onboard sensing constraints.

3 Experimental Setup

The experiments evaluate simulated locomotion policies against blind and noisy elevation-map baselines, using forward displacement and time to fall across varied terrains. The setup also measures real-time deployment on the robot’s onboard hardware.

The hardware uses a 12-joint A1 with a front-facing depth camera, and the policy runs at 50Hz while commands are converted to torques at 400Hz.
The study compares the proposed methods with blind and noisy baselines using identical learning-sample budgets.
Table 1 evaluates average forward displacement and mean time to fall across terrains with one policy trained for all terrains per method.
60-90% higher total mean time to fall and average displacement is reported for the monolithic and RMA methods versus noisy and blind baselines.
The noisy baseline adds elevation-map noise after distilling a teacher trained with ground-truth maps, while simulating 40ms latency in both phases.

4 Results and Analysis

The proposed egocentric-vision policies outperform blind and noisy baselines across challenging simulated terrains and transfer to diverse real-world environments. They achieve near-perfect task success while handling difficult foothold conditions, slips, pushes, and recovery situations.

Simulation Results: In simulation, the proposed methods travel up to 27% farther on slopes, up to 90% farther on stairs and discrete obstacles, and around 20m on stepping stones.
Real World Comparisons: The real-world evaluation uses separate phase-2 policies and camera placements for stairs versus stepping stones and gaps.
Real World Comparisons: The blind baseline completes downstairs with 100% success by using a high-impact falling gait that dislocated the robot’s rear right hip.
Urban Environments: The robot climbs stairs up to 24cm and curbs up to 26cm, using emergent hip abduction to overcome its small body clearance.
Gaps and Stepping Stones: The method achieves 100% success on gaps up to 26cm and 94% on difficult stepping stones, while the blind baseline achieves zero success on both.
Natural Environments: Outdoor deployments traverse rugged, rocky, slippery terrain and tree roots, although the robot sometimes gets stuck in crevices before recovering.

5 Related Work

Prior work commonly separates terrain perception, foothold planning, and control through elevation maps, while newer approaches act directly from egocentric depth. This paper positions its method as direct joint-angle prediction from egocentric depth without metric elevation maps.

Legged locomotion: Earlier locomotion research includes model-based techniques, heuristic reactive controllers, and reinforcement learning policies trained in simulation.
Locomotion from Elevation Maps: Classical elevation-map methods decouple perception from planning and control, often adding foothold feasibility estimation or traversibility prediction.
Locomotion from Elevation Maps: Elevation maps can be noisy or incorrect, motivating uncertainty modeling and simulated perception errors during policy training.
Locomotion from Egocentric Depth: Egocentric-depth approaches avoid explicit elevation maps, but prior examples include obstacle avoidance, hierarchical traversal, lidar generalization, and high-level gap-crossing actions.
Locomotion from Egocentric Depth: This work directly predicts target joint angles from egocentric depth without constructing metric elevation maps.

6 Discussion and Limitations

The approach demonstrates end-to-end walking across varied terrain, but real-world performance remains limited by visual or terrain mismatch between simulation and deployment. Addressing such failures currently requires engineering the mismatch into simulation and retraining.

Visual or terrain mismatch between simulation and the real world can cause the robot to fail in deployment.
Under the current paradigm, addressing these failures requires engineering the situation into simulation and retraining the policy.
The paper identifies future use of real-world data to improve both visual and motor performance.

A Proof of Theorem 3.1

The proof bounds the student policy’s performance gap by relating phase 1 value approximation, greedy teacher actions, and phase 2 policy closeness under a state mapping. Lipschitz continuity and bounded value assumptions extend the guarantee across the state space.

Assumptions: The theorem assumes phase 1 approximates the optimal value function within ϵ and phase 2 remains within η of phase 1 after mapping states.The phase 2 state space is related to phase 1 by an invertible mapping f.
Conclusion: Lipschitz-continuous rewards and transitions allow the phase 2 policy’s return to remain close to optimal under the stated approximation assumptions.The theorem’s conclusion applies everywhere in the mapped state space.
Proof strategy: The teacher action is greedy with respect to its Q-function, so its Q-value is at least that of the optimal action at the selected state.The proof compares the optimal action a∗ with the teacher action aT at the state of maximal student error.
Conclusion: The argument generalizes approximate optimal-value-function bounds to a teacher-student setting.The proof uses the teacher’s greedy action and the student’s deviation from the teacher policy.
Proof strategy: The proof selects the state s0 where the difference between optimal and student values is maximal, then propagates the resulting inequality to all states.This converts a worst-case local comparison into a global performance bound.

B Rewards

The reward design combines command tracking with penalties for energetic, damaging, or unstable behavior, while deliberately leaving gait selection to reinforcement learning. Command sampling and termination rules approximate operational behavior and prioritize survival on difficult terrain.

Reward terms: The reward includes absolute work, command-tracking, foot-jerk, foot-drag, collision, and survival terms.The absolute work penalty discourages exploiting contact-simulation inaccuracies, while other penalties target backlash, dragging, and limb collisions.
Reward terms: The reward scales are −1e−4, 7, −1e−4, −1e−4, −1, and 1, and no term specifies a gait prior.The optimal gait is therefore allowed to emerge through reinforcement learning.
Command sampling: Heading commands are sampled to mimic the command distribution encountered during operation.On flat ground, curve following, in-place turning, and complete stop are sampled uniformly.
Termination: Episodes terminate when pitch exceeds 90° or the robot base or head collides with an object.This termination rule defines explicit safety boundaries during training.
Observation processing: Training observations are shifted, normalized, and noised before being passed to the policy.The transformation is o′t = a(ot − b) + ϵ with ϵ ∼ N(0, σ).

C.1 Pseudo-code

Phase 1 uses policy-gradient reinforcement learning, while the phase 2 training procedure is specified separately in Algorithm 1.

C.2 Pseudo-code: Phase 1 is trained with policy gradients, and Algorithm 1 provides the pseudo-code for phase 2 training.The two phases separate initial policy learning from the subsequent training procedure.

C.2 Hardware

The system is implemented on a Unitree A1 with a single front-facing depth camera and split compute, while simulation supplies randomized terrains and camera-aligned depth training.

Hardware: The Unitree A1 has 12 actuated joints and uses a front-facing Intel RealSense depth camera for exteroception.The robot’s joints are arranged as three actuators per leg at the hip, thigh, and calf.
Hardware: The policy runs at 50Hz, while a low-level PD controller converts joint-position commands into torques at 400Hz.The controller uses stiffness Kp = 40 and damping Kd = 0.5.
Compute: Depth processing runs on a Jetson NX GPU, while the base policy runs on an UPboard connected over UDP.The split reduces the burden of expensive depth processing on the robot’s limited compute.
Simulation: Phase 2 renders depth from simulated cameras calibrated to the real camera, while the policy architecture combines recurrent, feed-forward, and convolutional modules.The architecture includes GRUs, MLPs, and a convolutional depth backbone.
Simulation: Simulation contains 100 sub-terrains in a 20 × 10 grid, with rows representing terrain types and increasing difficulty within each row.Terrain patches are 8m by 8m and include randomized fractal variations.