PaperlayerView exact revision on arXiv

Source-linked AI summary

Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion

Tairan He, Chong Zhang, Wenli Xiao, Guanqi He, Changliu Liu, Guanya Shi

arXiv:2401.17583v3cs.ROcs.AIcs.CVcs.LGeess.SY

TL;DR

Cluttered-environment locomotion requires both agility and safety, whereas existing approaches often emphasize one at the expense of the other. ABS combines an agile policy, a recovery policy, and a policy-conditioned reach-avoid value network, with learned exteroception for deployment. The framework achieves high-speed collision-free quadrupedal locomotion, while its dynamic-obstacle generalization remains bounded by the recovery policy’s velocity limit.

  • Problem

    Existing locomotion methods either use conservative speeds below 1 m/s for safety or pursue agility without considering collision safety.

  • Method

    ABS combines an agile policy, a recovery policy, a policy-conditioned reach-avoid value network for switching and recovery guidance, and a ray-prediction network trained in simulation.

  • Results

    ABS achieves high-speed collision-free quadrupedal locomotion, with success rates of 9 or 10 out of 10 across two indoor and one outdoor testbed and minimal collisions.

  • Takeaways & Limitations

    External reach-avoid shielding and guided recovery help ABS improve safety with only a minor decrease in agility relative to the learned agile policy.

  • Takeaways & Limitations

    Reach-avoid values are learned with static obstacles and generalize only to quasi-static environments; faster-moving objects can still cause collisions.

Abstract

from arXiv · show

Legged robots navigating cluttered environments must be jointly agile for efficient task execution and safe to avoid collisions with obstacles or humans. Existing studies either develop conservative controllers (< 1.0 m/s) to ensure safety, or focus on agility without considering potentially fatal collisions. This paper introduces Agile But Safe (ABS), a learning-based control framework that enables agile and collision-free locomotion for quadrupedal robots. ABS involves an agile policy to execute agile motor skills amidst obstacles and a recovery policy to prevent failures, collaboratively achieving high-speed and collision-free navigation. The policy switch in ABS is governed by a learned control-theoretic reach-avoid value network, which also guides the recovery policy as an objective function, thereby safeguarding the robot in a closed loop. The training process involves the learning of the agile policy, the reach-avoid value network, the recovery policy, and an exteroception representation network, all in simulation. These trained modules can be directly deployed in the real world with onboard sensing and computation, leading to high-speed and collision-free navigation in confined indoor and outdoor spaces with both static and dynamic obstacles.

I. INTRODUCTION

Agile But Safe addresses the trade-off between fast locomotion and collision safety by combining agile end-to-end control with control-theoretic safeguarding. Its dual-policy architecture and learned reach-avoid values support high-speed collision-free quadrupedal navigation.

  • Motivation: Existing approaches either limit velocity below 1 m/s for safety or pursue agility without accounting for collision safety.The paper frames both agility and safety as necessary for applications such as search and rescue, disaster response, and police robotics.
  • Relation to Prior Work: The approach targets limitations of model-based and hierarchical methods, including model mismatch, online computation, conservative behavior, and decoupled tracking.Model-free reinforcement learning supplies agile motor skills, while control-theoretic tools provide additional safety guidance.
  • Framework: ABS combines a model-free perceptive agile policy with a recovery policy that takes control when the agile policy may fail.A policy-conditioned reach-avoid value network estimates the agile policy’s risk and governs switching between policies.
  • Perception: The framework uses low-dimensional ray distances and a ray-prediction network to convert depth images into exteroceptive features for policy and reach-avoid training.The representation is intended to support generalizable collision avoidance with onboard sensing and computation.
  • Reported Capability: ABS reports collision-free quadrupedal locomotion at maximum velocity up to 3.1 m/s, including dynamic adversarial obstacles with peak velocity 2.5 m/s and average speed 1.5 m/s.These results are presented as exceeding prior end-to-end collision-avoidance locomotion limited to approximately 0.4 m/s.
  • Safety Mechanism: ABS learns reach-avoid values conditioned on the agile policy and feeds their gradients to the recovery policy in a closed loop.This distinguishes the approach from policy-agnostic reach-avoid estimation and avoids identifying a global reach-avoid set.

III. OVERVIEW AND PRELIMINARIES

ABS models robot dynamics and observations, then formulates reach-avoid behavior around target and failure sets. Its policy-conditioned reach-avoid value supports safety shielding through discounted Bellman updates.

  • Dynamics: The system maps each state and control input to a next state, while observations are generated through a sensor mapping from states.The formulation permits deterministic dynamics without requiring an analytical form and supports proprioceptive or exteroceptive observations.
  • Reach-Avoid Formulation: Reach-avoid behavior seeks target states while avoiding unsafe failure states throughout the remaining time horizon.The failure set includes states such as collisions, while the target set represents desired goal states.
  • Reach-Avoid Value: The policy-conditioned reach-avoid value satisfies a fixed-point Bellman equation and is transformed into a time-discounted form for data-driven approximation.The discounted formulation provides contraction properties needed to learn the value function from data.
  • Safety Shielding: A nonpositive reach-avoid value identifies states inside the policy-conditioned reach-avoid set, enabling threshold-based shielding.This connects the learned value estimate to maintaining safety under the selected policy.

C. System Structure

ABS combines an agile policy with a recovery policy, using a reach-avoid value estimate to switch control according to safety. Four simulation-trained modules provide agile control, recovery, safety estimation, and exteroceptive inputs for direct real-world deployment.

  • Dual-Policy Structure: The agile policy performs goal-directed locomotion with basic collision avoidance, while the recovery policy rapidly tracks twist commands to avoid collisions.Both policies output joint targets that a PD controller tracks.
  • Policy Switching: During deployment, a neural reach-avoid value estimate conditioned on the agile policy governs the policy switch using threshold Vthreshold = −ϵ.The threshold uses a small positive ϵ to determine when the agile policy remains appropriate.
  • Recovery Mode: When ˆV ≥ Vthreshold, the recovery policy tracks a twist command selected to move toward the goal while maintaining safety.The command is searched using the reach-avoid value as the safety criterion.
  • Agile Mode: When ˆV < Vthreshold, the agile policy controls navigation, while recovery remains a safeguard until the estimated value returns below threshold.The design expects agile control to be active most of the time.
  • Exteroception: ABS uses 11 ray distances predicted from depth images as low-dimensional exteroceptive inputs for the agile policy and reach-avoid value network.The representation is intended to generalize collision avoidance across scenarios.
  • Training and Deployment: Four modules are trained in simulation and then directly deployed in the real world: agile policy, reach-avoid value network, recovery policy, and ray-prediction network.The agile policy uses goal-reaching rewards to encourage high agility without collisions.

A. Observation Space and Action Space

The agile policy combines proprioceptive, goal, timing, action-history, and exteroceptive observations with joint-target actions. Its reward design jointly encourages goal reaching, heading alignment, standing, agility, and avoidance of stalling.

  • Observation Space: The agile policy observes contacts, base motion, gravity, goal commands, remaining time, joint states, previous actions, and logarithmic ray distances.Only orientation and odometry-related observations require state estimation; the other inputs come from raw sensors.
  • Action Space: The action space consists of 12-dimensional joint targets tracked by a PD controller.A fully connected MLP maps the observation vector to these joint targets.
  • Task Rewards: The task reward combines soft and tight position tracking, heading tracking, standing, agility, and stall penalties.The task reward is rtask = 60 · rpossoft + 60 · rpostight + 30 · rheading, with additional terms for standing, speed, and waiting behavior.
  • Task Rewards: Tracking rewards encourage reaching the goal before the episode deadline without explicit target-velocity constraints.The soft and tight position thresholds are 2 m and 0.5 m, with time thresholds of 2 s and 1 s, respectively.
  • Task Rewards: The agile term rewards forward velocity only when the robot moves in the correct direction, with vmax = 4.5 m/s as its upper bound.The correct-direction condition requires the heading-to-goal angle to be smaller than 105°.
  • Task Rewards: The stall term penalizes remaining static when the goal is distant and the robot is not oriented correctly.This term is intended to penalize time waste during goal reaching.

3) Regularization Rewards:

Regularization rewards constrain physically undesirable behaviors during agile-policy training. The policy is trained in parallel across randomized terrains and obstacle layouts using PPO.

  • Regularization Rewards: Regularization penalizes torque, joint-velocity, joint-position, action-change, and flight-related violations.The flight penalty applies when the robot has no ground contact because the base then becomes uncontrollable.
  • Training: The agile policy is trained with PPO in 1280 Isaac Gym environments running in parallel.The simulator is GPU-based.
  • Training: Training terrains are curriculum-randomized among flat surfaces, rough terrain, and low stumbling blocks.Across difficulty levels, terrain height differences increase from 0 cm to 7 cm.
  • Training: Obstacle training uses 0–8 randomly distributed cylinders with 40 cm radius in an 11 m × 5 m region covering the origin and goal.Higher curriculum levels contain more obstacles.

3) Obstacles:

The framework trains robust reach-avoid estimation and recovery around randomized obstacles using domain-randomized sensing and reduced safety-relevant observations. The learned RA values vary with velocity and indicate agile-policy safety when the failure function is softened.

  • Domain Randomization: Domain randomization models unseen geometries, motor sim-to-real gaps, and encoder offsets during agile-policy training.The illusion randomizes sufficiently distant ray observations, while ERFI-50 adds torque perturbations and joint positions receive random biases.
  • RA Value Estimation: The RA value network uses base twists, goal position, and exteroception rather than high-dimensional joint-level observations.These reduced observations are selected because they influence safety and goal reaching while improving generalization.
  • RA Value Estimation: The policy-conditioned RA network is trained from agile-policy rollouts using a data-driven RA Bellman loss.The dataset contains 200k simulated agile-policy episodes, and γRA = 0.999999 approximates the undiscounted RA value.
  • RA Value Estimation: Unlike policy-agnostic global reach-avoid estimation, the method learns only the agile policy’s failures through two-stage offline training.The stages collect policy trajectories first and train the policy-conditioned value afterward, avoiding minimization over the full action space.
  • RA Value Estimation: Softening the collision-failure function makes it approach Lipschitz continuity and improves the RA value’s safety indication.Without softening, the value estimate misses side collisions and develops local minima in front of obstacles.

C. Using RA Values for Recovery

Recovery uses the RA value to detect unsafe agile-policy behavior and optimize a safe twist command. A learned recovery policy then tracks that command rapidly as a backup shield.

  • Policy Switching: The recovery policy is triggered when the estimated RA value satisfies ˆV(oRA) ≥ Vthreshold, with Vthreshold = −0.05.The threshold compensates for learning errors without causing over-conservative shielding.
  • Twist Optimization: When recovery is active, the system optimizes a twist command that approaches the goal while satisfying ˆV < Vthreshold.The goal-distance estimate uses a short δt = 0.05 s prediction based on linearized robot displacement.
  • Twist Optimization: Gradient descent with a Lagrangian multiplier solves the twist optimization within five steps from the current twist.This enables real-time deployment, and the searched twist consistently satisfies the safety constraint.
  • Recovery Policy: The recovery policy is trained to track twist commands rapidly so it can serve as a backup shielding policy.Unlike the agile policy, it does not require exteroception and observes only locomotion, command, joint, and previous-action variables.
  • Recovery Policy: The recovery policy uses the same 12-dimensional joint-target action space and an MLP policy network as the agile controller.Its inputs differ because it tracks twist commands rather than directly using exteroceptive observations.

B. Rewards

The recovery policy combines task, penalty, and regularization rewards, while the training setup randomizes triggering states and uses ray-based exteroception for policy learning.

  • Rewards: The recovery policy allows knee-ground contacts for maximum deceleration while retaining the agile policy’s penalty and regularization rewards.This modifies the contact behavior specifically for recovery.
  • Rewards: The recovery policy’s task rewards track commanded twist, keep the robot alive, and maintain posture for seamless switching back to the agile policy.Its task reward combines linear-velocity tracking, angular-velocity tracking, an alive term, and posture regularization.
  • Rewards: Recovery-policy training changes episode length and initial-state randomization to better represent states that trigger recovery during agile running.Episodes last 2 s, with randomized initial roll, pitch, linear velocity, and angular velocity ranges.
  • Exteroception: Ray-prediction training uses simulated data, with obstacle configurations and image augmentation supporting depth-based prediction.The section presents obstacle collections and four augmentation types for training the representation network.
  • Exteroception: The agile policy and reach-avoid value network use 11 ray distances, which are predicted from depth images for deployment.The ray representation keeps policy inputs low-dimensional and interpretable while avoiding direct dependence on unavailable ground-truth rays.

A. Data Collection

The evaluation compares ABS, its agile policy alone, and LAG across randomized obstacle settings, measuring safety and agility over large simulation batches. Results show that RA-value shielding and recovery substantially improve safety with only a minor agility cost.

  • Data Collection: The benchmark compares ABS, the agile policy alone, and LAG to examine the agility-safety trade-off and the effect of recovery shielding.ABS includes both agile and recovery policies, while LAG uses PPO-Lagrangian with the agile policy formulation.
  • Data Collection: Testing uses aggressive, nominal, and conservative reward variants with eight obstacles packed into a 5.5 m × 4 m rectangle.The test distribution remains within training conditions but is harder than most training cases.
  • Data Collection: Each episode is labeled success, collision, or timeout, and evaluation reports success, collision, timeout, peak velocity, and success-case speed.Means and standard deviations are computed across three seeds using 10k random episodes.
  • Results: ABS breaks the observed agility-safety boundary, substantially improving safety at the cost of only a minor decrease in agility.The comparison reports this pattern across reward tuning and safe-exploration variants.
  • Example Case: In a route through eight obstacles, ABS runs quickly in open spaces and slows in tight spaces, whereas the agile baseline collides and LAG is slower.The RA values and recovery policy shield ABS when the route becomes constrained.

C. Real-World Experiments

Real-world experiments evaluate ABS on indoor and outdoor testbeds with onboard sensing and computation, alongside robustness and sim-to-real tests. ABS achieves strong safety and speed across environments while tolerating payloads and perturbations, though perception remains a limitation in dim settings.

  • Real-World Setup: ABS uses a Unitree Go1 with onboard Jetson Orin NX computation, a ZED Mini Stereo Camera, odometry, and built-in PD control.The sensing and computation stack supports deployment in the reported real-world experiments.
  • Results: Across two indoor and one outdoor testbed, ABS achieves the highest success rates and lowest collision rates, scoring 9 or 10 out of 10 in success rates.The reported environments include a dim narrow corridor, a furnished hall, and an outdoor playground.
  • Results: ABS combines high speed with high safety, while the fastest-running alternative incurs more collisions and LAG trades speed for safety.The comparison covers the agile policy alone, LAG, and ABS.
  • Robustness: ABS operates on slippery snow, carries a 12-kg payload equal to its own weight, and withstands external perturbations.These tests are presented as robustness evaluations of the complete system.
  • Policy Design: The goal-reaching formulation is chosen for the agile policy because it avoids decoupling locomotion and navigation and supports high-speed gait learning.The paper contrasts this with velocity-tracking formulations and compares them in a dedicated experiment.
  • Sim-to-Real: Illusion and ERFI-50 randomization address sim-to-real failures that otherwise cause trembling near unseen walls or head impacts during running.The paper identifies these randomization components as important for deployment robustness.
  • Safety Shielding: The safety threshold is set to V_threshold = −0.05, while scanning thresholds from −0.001 to −0.1 produces no significant overall-performance change.The threshold represents conservativeness and approximation discrepancy in the switching strategy.

2) Soft Lipschitz continuity for the failure indicator:

Softening the collision indicator improves safety while slightly increasing conservativeness. The framework also documents deployment trade-offs, failure cases, and supported operating boundaries.

  • Softening the discrete collision indicator significantly enhances system safety while slightly increasing conservativeness.
  • RA shielding improves LAG safety while slightly reducing agility, with LAG+RA still exceeding 1 m/s average speed.
  • Larger perception networks improve prediction accuracy but increase inference time, motivating ResNet-18 for real-time high-speed locomotion.
  • Instant steering is enabled by overwriting goal commands at runtime, allowing direct human involvement during agile locomotion.
  • The policy can fail in dense obstacles forming local minima, while dynamic objects moving faster than recovery limits can still cause collisions.
  • The system is limited to 2D locomotion without a flying phase, and temporal system identification is difficult to incorporate into the RA module.

APPENDIX

The appendix reports training costs, perception-data choices, simulation-ray limitations, and implementation trade-offs relevant to deployment.

  • The agile policy converges in approximately 20 minutes, while the recovery policy becomes near-optimal within 10 minutes.
  • The RA value network is trained in parallel with agile-policy rollout, taking approximately 40 minutes.
  • Ray-prediction training uses 250k labelled images, and the network can reuse data collected from different policies.
  • Analytical ray tracing improves training efficiency but is limited to simple geometries, preventing diverse objects from being used during data collection.
  • The low-dimensional representation, domain randomization, and data augmentation together support generalization to varied real-world objects.
  • The implementation includes regularization rewards adapted from prior locomotion methods and tuning experience.
Loading 2401.17583v3…