PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Navigation Behaviors End-to-End with AutoRL

Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, Anthony Francis

arXiv:1809.10124v2cs.ROcs.AIcs.LG

TL;DR

The paper addresses robust point-to-point and path-following navigation with noisy sensors and moving obstacles. It uses AutoRL to optimize shaped rewards and network architectures for end-to-end policies trained in small static simulations, achieving higher success in novel environments and reliable physical-robot deployment. The policies generalize to moving obstacles and remain robust to noise, although training is computationally expensive.

  • Problem

    Robust navigation behaviors must transfer across environments and physical robots, avoid unexpected obstacles, and handle noisy sensing without extensive tuning.

  • Method

    AutoRL uses large-scale gradient-free hyperparameter optimization to select a shaped reward and then a neural-network architecture for end-to-end navigation policies.

  • Results

    23% higher path-following success and 26% higher point-to-point success were achieved on average over the baseline in novel environments, with robust noise handling and physical-robot deployment.

  • Takeaways & Limitations

    The resulting point-to-point and path-following policies generalize to new environments with moving obstacles and can be deployed on a physical robot without tuning.

  • Takeaways & Limitations

    AutoRL is not sample efficient: training 1000 agents took 12 days.

Abstract

from arXiv · show

We learn end-to-end point-to-point and path-following navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are trained in small, static environments with AutoRL, an evolutionary automation layer around Reinforcement Learning (RL) that searches for a deep RL reward and neural network architecture with large-scale hyper-parameter optimization. AutoRL first finds a reward that maximizes task completion, and then finds a neural network architecture that maximizes the cumulative of the found reward. Empirical evaluations, both in simulation and on-robot, show that AutoRL policies do not suffer from the catastrophic forgetfulness that plagues many other deep reinforcement learning algorithms, generalize to new environments and moving obstacles, are robust to sensor, actuator, and localization noise, and can serve as robust building blocks for larger navigation tasks. Our path-following and point-to-point policies are respectively 23% and 26% more successful than comparison methods across new environments. Video at: https://youtu.be/0UwkjpUEcbI

I. INTRODUCTION

The paper targets robust local navigation behaviors that transfer across robots and environments despite noisy sensing and moving obstacles. AutoRL addresses deep-RL training challenges by automating reward shaping and neural-network architecture search, producing end-to-end policies evaluated in simulation and on a physical robot.

  • Motivation: Robust navigation behaviors must account for robot sensors, actuators, dynamics, geometry, noise, and unexpected obstacles without foreknowledge of the environment.The paper focuses on point-to-point local planning and path following as core behaviors for dynamic environments.
  • Motivation: Sparse rewards and neural-network architecture choices make deep-RL training difficult, especially for navigation in large spaces.The paper identifies reward design and architecture selection as key sources of training failure or performance variation.
  • Approach: AutoRL searches first for a shaped reward and then for network layer sizes using large-scale gradient-free hyperparameter optimization.The reward is optimized against the true task objective before architecture optimization is performed with the reward fixed.
  • Approach: AutoRL learns point-to-point and path-following policies in small static simulated environments that cope with static and dynamic obstacles.The method is designed to reduce hand-engineering while retaining end-to-end sensor-to-control behavior.
  • Evaluation: 23% higher path-following success and 26% higher point-to-point success were achieved on average over the baseline in novel environments.The policies were evaluated across three large building environments under varying noise levels and moving-obstacle counts.
  • Evaluation: On a real robot, the policies showed reliable obstacle avoidance in dynamic environments and collision-free navigation along 80+ m paths.The work positions the learned behaviors as components that can transfer from simulation to physical navigation stacks.

III. METHODS

The methods formulate navigation as continuous reinforcement learning under partial observability, with task-specific objectives represented through rewards. AutoRL tunes parameterized reward components while the policies use lidar and goal observations to control linear and angular velocity.

  • POMDP Setup: Point-to-point and path-following behaviors are learned with continuous reinforcement learning modeled as a partially observable Markov decision process.The two tasks share observations, actions, and dynamics but differ in their objectives and rewards.
  • POMDP Setup: The agent observes recent 1-D lidar vectors and goal information, then outputs a two-dimensional continuous action encoding linear and angular velocity.The dynamics are represented by the simulator or arise implicitly in the real world.
  • Task Objectives: Point-to-point navigation seeks arrival at a goal location, whereas path following seeks to traverse the entire path by reaching all waypoints.These objectives define the true navigation goal that the learned policy should accomplish.
  • Reward Design: Parameterized atomic rewards provide timely feedback for sparse objectives while encoding dynamical feasibility, smooth trajectories, obstacle avoidance, and sensory or motor limitations.The true objective is sparse, so the requirements are represented through reward terms to aid learning.

B. Point-to-Point (P2P) Task

The P2P behavior navigates a localized robot to a nearby goal without collisions, while PF follows waypoint-based guidance paths. Both use reward components that encode progress and obstacle avoidance.

  • P2P Task: P2P navigates to a goal without collision and can serve as a local planner for sampling-based planners.It assumes reliable localization and targets goals beyond clear line of sight but not requiring maze-level environmental knowledge.
  • P2P Task: The P2P true objective is to maximize the probability of reaching the goal during an episode.
  • P2P Task: The P2P reward combines step, goal-distance, collision, turning, clearance, and goal terms.These terms penalize time, distance, collisions, and angular speed while incorporating obstacle clearance and goal achievement.
  • Path-Following Task: PF follows a guidance path represented by workspace waypoints, treating the behavior as trajectory tracking.Guidance paths may come from PRMs, A*, or manual specification and need not be dynamically feasible.
  • Path-Following Task: PF interpolates waypoints at a searched separation and marks each waypoint reached only after its predecessor is reached and the robot enters the waypoint radius.This addresses long paths with varied waypoint spacing.
  • Path-Following Task: The PF objective is to reach as many waypoints per episode as possible using distance, collision, clearance, and step-related reward terms.

D. Reinforcement Learning Parametrization

The RL parametrization exposes network layer sizes and reward weights to evolutionary hyperparameter search. AutoRL first selects a reward using the true task objective, then selects actor and critic architectures using cumulative reward.

  • Network Parametrization: Feed-forward fully connected networks use fixed depth while the size of each layer remains tunable.The approach is stated to be applicable to other network architectures.
  • Network Parametrization: The actor learns policy π(s|Wπ) and the critic learns Q(s,a|WQ), with both architectures and the reward parameterized for optimization.
  • AutoRL: AutoRL separates reward shaping from network shaping because jointly searching their parameters grows exponentially with the number of tuning parameters.
  • Reward Shaping: During reward shaping, trials vary reward weights, train agents asynchronously, and select the reward with the highest true task objective.The first trials select reward weights randomly, after which completed trials guide gradient-free selection.
  • Network Shaping: During network shaping, AutoRL fixes the selected reward and searches actor and critic architectures to maximize cumulative reward.
  • Search Procedure: The hyperparameter selector returns the policy and selected reward and network architecture parameters after evaluating each trial.It runs a specified number of generations with a bounded number of concurrent trials and scales linearly with the number of trials.

A. Setup

Experiments use a simulated building-derived environment and compare AutoRL policies with conventional, hand-tuned, and imitation-based navigation methods. Networks and tuning are selected for onboard control feasibility.

  • Environment: The training environment is a 23m by 18m real-building floor plan without moving obstacles.
  • Robot and Observations: The simulated point-mass differential-drive robot observes a noisy 64-beam lidar and outputs bounded linear and angular velocities.The lidar spans 220 degrees, with Gaussian noise, while velocity bounds are specified for both controls.
  • Learning Setup: Actor and critic networks are three layers deep, and wide shallow architectures are chosen for fast onboard high-frequency control.DDPG is used with Vizier and CMA-ES for hyperparameter tuning.
  • Baselines: The P2P agent is compared with hand-tuned DDPG, APF, DWA, and behavior cloning, while PF is compared with PRM-based and combined baseline methods.

B. Training

AutoRL searches reward functions and network architectures for navigation policies, improving objectives and producing smoother, more successful behavior than hand-tuned alternatives.

  • B. Training: AutoRL trains 1000 trials with 100 agents in parallel, each running 5 million training steps.Each agent takes about 12 hours, and a complete AutoRL run takes several days.
  • B. Training: The final AutoRL shaping phase avoids the catastrophic forgetting observed after reward shaping alone.The best agent after both reward and network shaping does not exhibit the forgetfulness seen in DDPG training.
  • B. Training: 0.90 training objective is achieved by the P2P AutoRL agent, compared with 0.54 for the hand-tuned agent.
  • B. Training: 98.7% average success is achieved by AutoRL across three environments, exceeding the hand-tuned policy and PRM-DWA baseline.AutoRL is reported as the only method transferring to unseen, large, real-building-sized environments.
  • B. Training: AutoRL produces smooth forward motion, whereas hand-tuned policies alternate forward and reverse motion, creating longer, twirling trajectories.The 220-degree lidar cannot detect rear obstacles, lowering the twirling policy’s success rate.
  • B. Training: 89% average P2P success is highest across environments, while hand-tuned policies achieve 86.5% but show twirling and subpar noisy-condition performance.

D. Moving Obstacle Avoidance

AutoRL policies maintain strong obstacle-avoidance performance across moving-obstacle counts, long paths, noise, and physical-robot trials, while P2P performance remains distance-sensitive.

  • D. Moving Obstacle Avoidance: AutoRL outperforms hand-tuned DDPG, PRM-APF, and PRM-DWA across all tested numbers of moving obstacles.PRM-APF fails when it executes paths without reacting to obstacles, whereas PRM-DWA remains steady as obstacle count increases.
  • D. Moving Obstacle Avoidance: AutoRL PF success remains uniform for guidance paths up to 80 meters and decreases only slightly with moving obstacles.PRM-APF degrades rapidly with path length, while PRM-DWA is uniform but lower than AutoRL.
  • D. Moving Obstacle Avoidance: P2P success decreases as goal distance increases but is not affected by the number of moving obstacles.Across methods, complex static obstacles are identified as the main P2P failure cause rather than moving-obstacle collisions.
  • D. Moving Obstacle Avoidance: PF and P2P remain resilient when lidar noise exceeds three times the robot radius, including σ_lidar = 1 m.Lidar noise heavily influences APF and DWA methods.
  • F. Physical Robot Experiments: The physical robot reaches its goal without collision in all three runs, navigating roughly 240 m without collision.The tested PF guidance path was 80.6 m, and the P2P start-to-goal distance was about 13.4 m.
  • F. Physical Robot Experiments: The robot reliably navigates a corridor four times despite moving obstacles and a corridor only 0.3 m wider than the robot.
  • F. Physical Robot Experiments: A physical-robot failure occurred when obstacles such as human feet were below the 1D lidar’s field of view.

V. DISCUSSION

AutoRL produces robust end-to-end navigation behaviors that generalize across environments and noise conditions, but requires substantial training time. Its policies integrate perception, planning, and control, while remaining limited on large-scale local minima.

  • Training cost and deployment: 12 days were required to train 1000 agents, making AutoRL computationally expensive despite producing transferable, deployable navigation behaviors.The authors judge the extra training cost justified for navigation agents because of the resulting policy quality.
  • Noise robustness: End-to-end learning tightly couples perception, planning, and control, enabling robustness to noise in the partially observable navigation setup.Traditional APF and DWA methods are described as brittle because they lack tight sensor-control integration.
  • Path following: Reward tuning can overcome guidance-path imperfections by finding waypoint spacing and waypoint-radius parameters suited to the robot’s noise and capabilities.The authors connect this tuning to assigning waypoint credit at feasible distances.
  • Point-to-point limitations: AutoRL’s P2P policy is more robust to local minima than APF and DWA, but cannot avoid large-scale local minima such as moving between rooms.The policy was not designed for that larger-scale navigation problem.
  • Training cost and deployment: AutoRL policies transfer to new environments with moving obstacles, tolerate noise, and were deployed on a physical robot without tuning.These properties are reported alongside higher-quality behavior than hand-tuned RL and non-learned baselines.

APPENDIX

The appendix describes AutoRL’s two-stage optimization and the evaluation setup for path-following and point-to-point policies. Experiments include simulated noise, unseen environments, and real-robot trajectories.

  • Evaluation figures: Success rate is evaluated against lidar, localization, and process noise with 20 moving obstacles for both path following and point-to-point tasks.Path following is compared with hand-tuned RL, PRM-APF, and PRM-DWA; P2P is compared with hand-tuned RL, APF, and DWA.
  • Evaluation figures: Real-robot evaluations overlay magenta robot trajectories, black guidance paths, and green or red simulated trajectories on a 2D lidar map.Green denotes path following and red denotes point-to-point behavior.
  • Implementation: The agent operates at 5 Hz, with 100-second training episodes for both point-to-point and path-following tasks.Evaluation episodes are extended for longer trajectories, and localization and orientation observations come from the ROS navigation stack.
  • Evaluation setup: Simulated evaluations use three previously unseen large environments and repeat each evaluation 100 times.Point-to-point starts and goals are 5–10 meters apart, while path-following pairs require at least 35 meters Euclidean separation.
  • AutoRL diagram: AutoRL runs parameterized vanilla DDPG agents in parallel and evaluates each agent against the true goal objective.Gradient-free optimization selects subsequent reward and network parameters from the stored evaluations.
  • AutoRL diagram: AutoRL first shapes the reward with a fixed network architecture, then shapes the network architecture with the reward fixed.The two stages are shown as separate optimization steps around the same true task objective.

C. Robot noise modelling

The appendix specifies the noise model, comparison baselines, and guidance-path methods used to evaluate path-following and point-to-point policies. It also documents behavior-cloning training data and moving-obstacle guidance.

  • Noise model: Robot process, localization, and lidar noise are modeled with Gaussian distributions, using default values σLidar = 0.3 m, σSpeed = 0.1 m/s, σTurning = 0.1 rad/s, and σLocalize = 0.1 m.Process noise is added to linear and angular velocities, while localization noise affects the assumed robot localization.
  • Baselines: APF and DWA are selected as baselines because they are fast, well compared, and use clearance information available from 1D lidar.Many dynamic-environment planners are excluded because they require obstacle velocity or dynamics.
  • Guidance paths: PRMs generate guidance paths from floor plans or lidar maps, producing sequences of x, y positions connecting starts to goals without collision.The guidance paths support evaluation of path-following behavior and mitigation of local minima in baseline methods.
  • Behavior cloning: The behavior-cloning baseline trains on 100 000 observation-action transitions sampled from randomly generated successful paths of lengths 5–10 m.Its test set contains 10 000 transitions, using the same network architecture and environment settings as the RL point-to-point task.
  • Moving obstacles: For moving obstacles, path guidance is created for each obstacle and used to compute the desired velocity component in the Social Force Model.This is intended to alleviate the limitation of moving obstacles becoming trapped in local minima.
Loading 1809.10124v2…