PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations

Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, Juan Nieto

arXiv:1805.07095v2cs.RO

TL;DR

Map-less target-driven navigation must operate from local information despite the sample complexity of reinforcement learning and distribution mismatch in imitation learning. The paper combines expert-demonstration pre-training with reinforcement learning in an end-to-end policy and evaluates the approach across simulation and real robotics. Reinforced imitation reduces training time by around 80% while maintaining similar success and collision-avoidance performance, and the policy navigates unseen environments in simulation and the real world.

  • Problem

    Map-less navigation requires robust policies from local perception when reliable environment maps are unavailable, while imitation learning and reinforcement learning face distribution mismatch and high sample complexity, respectively.

  • Method

    The approach pre-trains an end-to-end navigation policy with expert demonstrations and imitation learning, then applies reinforcement learning with safety constraints and varied reward structures.

  • Results

    Around 80% reduction in reinforcement-learning training time accompanies similar final success-rate and collision-avoidance performance, with reliable navigation in unseen simulated and real environments.

  • Takeaways & Limitations

    Pre-training improves exploration even with 10 demonstrations and supports low-information reward structures such as sparse target rewards.

Abstract

from arXiv · show

This work presents a case study of a learning-based approach for target driven map-less navigation. The underlying navigation model is an end-to-end neural network which is trained using a combination of expert demonstrations, imitation learning (IL) and reinforcement learning (RL). While RL and IL suffer from a large sample complexity and the distribution mismatch problem, respectively, we show that leveraging prior expert demonstrations for pre-training can reduce the training time to reach at least the same level of performance compared to plain RL by a factor of 5. We present a thorough evaluation of different combinations of expert demonstrations, different RL algorithms and reward functions, both in simulation and on a real robotic platform. Our results show that the final model outperforms both standalone approaches in the amount of successful navigation tasks. In addition, the RL reward function can be significantly simplified when using pre-training, e.g. by using a sparse reward only. The learned navigation policy is able to generalize to unseen and real-world environments.

I. INTRODUCTION

The paper targets map-less navigation from local perception and combines imitation learning with reinforcement learning to improve training efficiency, safety, and generalization. It introduces reinforced imitation learning with expert pre-training followed by constrained RL, alongside evaluations in simulation and on a robotic platform.

  • Map-less navigation is needed when reliable global maps are unavailable, using local perception and a relative target position instead.
  • Imitation learning is sample efficient but can suffer from distribution mismatch, whereas reinforcement learning learns through exploration and reward signals.
  • Reinforced imitation learning pre-trains an end-to-end navigation policy with expert demonstrations before subsequent reinforcement learning.
  • Constrained Policy Optimization incorporates safety constraints during reinforcement learning, avoiding reliance on a fixed collision penalty.
  • The proposed model is designed for map-less end-to-end motion planning and generalization to unseen environments.
  • The study evaluates training and generalization performance, including unseen environments, through simulation and experiments on a real robotic platform.

II. RELATED WORK

Prior work spans inverse reinforcement learning, imitation learning, classical-control hybrids, and reinforcement-learning methods for navigation and collision avoidance. The paper distinguishes its approach by combining demonstration pre-training with constrained reinforcement learning for map-less, target-driven navigation and real-world applicability.

  • Inverse reinforcement learning infers a reward from expert demonstrations, while imitation learning directly infers a policy from those demonstrations.
  • Prior imitation-learning studies address collision avoidance, perception, or limited navigation settings, but some do not analyze overall navigation performance or require target-driven navigation.
  • Earlier end-to-end map-less navigation used global-planner demonstrations but faced generalization limitations and uncovered-situation failures.
  • Reinforcement-learning studies improve sample efficiency or transfer across conditions, but often assume known maps, simple environments, or restricted navigation tasks.
  • The closest related method uses ADDPG for simulated and real-world map-less navigation but trains from scratch without collision constraints.
  • This work combines prior demonstrations with CPO to address training speed and safety for real-world map-less navigation.

III. APPROACH

The approach targets map-less navigation by mapping local sensor measurements and a relative goal to robot control commands. It sequentially combines imitation learning and reinforcement learning to improve sample efficiency and robustness.

  • Map-less navigation uses local sensor information because maintaining accurate maps is difficult in unknown or changing environments.
  • The policy πθ maps sensor measurements y and relative target position g to control commands u.
  • The control commands comprise translational and rotational velocity.
  • The method combines IL and RL sequentially, using the IL result to initialize reinforcement learning.
  • The policy network processes normalized inputs through three fully connected tanh layers, with dropout between the first two layers during IL training.

B. Neural network model

The navigation policy uses pooled laser measurements, a relative target position, and a simplified fully connected network. Expert demonstrations pre-train the policy before reinforcement learning.

  • The model replaces CNN-based feature extraction with three fully connected layers to reduce overfitting to obstacle shapes.
  • The pooled laser measurements and relative target position are normalized before entering the network, while outputs are de-normalized into translational and rotational velocities.
  • Supervised IL pre-trains the policy from expert demonstrations to improve the succeeding RL stage's performance and sample complexity.
  • Unlike plain IL, R-IL can improve beyond the performance of the demonstrations through subsequent self-improvement.

D. Reinforcement learning

The reinforcement-learning formulation optimizes discounted rewards for a policy mapping states to actions. Because policy-gradient updates can be unstable and unsafe, the approach uses constrained policy optimization.

  • RL seeks a policy πθ that maps states to actions while maximizing expected discounted rewards.
  • In navigation, states contain laser measurements and target information, while actions are control commands.
  • Policy-gradient methods optimize J(θ) but can produce high-variance gradients and undesirably large policy updates.
  • CPO treats safety as a constraint on expected discounted cost rather than relying only on a tuned unsafe-state penalty.

2) Training process:

Training initializes the policy randomly or from IL, collects on-policy episodes, and updates it using reward feedback. The study varies reward information and encodes collision avoidance through constrained costs.

  • Training process: Training compares randomly initialized pure RL with IL-initialized R-IL policies.
  • Training process: Episodes begin from randomly selected start and target positions and end after a fixed horizon or target arrival.
  • Training process: The objective is to reach the target quickly while avoiding collisions, with reward functions providing learning feedback.
  • Training process: Sparse rewards provide minimal task information but make learning difficult because episode actions share credit for the outcome.
  • Training process: Distance-based rewards provide continuous feedback by rewarding or penalizing progress toward the target.
  • Training process: Collision avoidance is represented through a constrained expected crash cost rather than a negative collision reward.
  • Training process: The experiments set the crash constraint to α = 0.4 after testing values from 0.0 to 0.6, while allowing multiple crashes per episode for more crash samples.

IV. EXPERIMENTS

The experiments evaluate pre-training, RL procedures, reward functions, and generalization in simulation and on a real robotic platform. Training is performed in simulation, with the real platform used for deployment evaluation.

  • The experiments investigate pre-training, constraint-based versus fixed-penalty methods, reward functions, prior approaches, unseen scenarios, and real-world deployment.
  • Training maps vary significantly in difficulty, motivating evaluation across different map conditions.
  • The study does not aim to outperform global graph-based planners in known environments, but examines motion planning using local information only.
  • Models are trained purely in simulation because it is safe, fast, efficient, and flexible for changing environment structures.
  • The real platform is a differential-drive Kobuki TurtleBot equipped with a front-facing 270° Hokuyo laser range finder and onboard Intel NUC computation.

B. Model training

The training study varies imitation-learning data, maps, reward signals, and RL procedures to assess how initialization and training design affect navigation learning. Pre-training is motivated by the substantial cost of long RL runs.

  • The study compares pure IL, pure RL, and R-IL across different map subsets and training environments.R-IL uses IL on simple and complex maps before RL on the three TM maps.
  • The experiments vary demonstrations from 10 to 1000, compare CPO with TRPO, and test sparse, Euclidean, and shortest-distance rewards.
  • One RL iteration uses 60 k time steps and takes around 180 s, so 1000 iterations require around 50 hours in simulation.
  • Pre-trained models start with a nonzero success rate, whereas pure RL models require substantial iterations to reach the target in most cases.
  • Figure 4 tracks rolling success and crash rates over 20 steps for pure RL, pure IL, and R-IL models, with IL training performance as a reference.
  • Constraint-based RL can make early exploration difficult, strengthening the motivation for pre-training before applying safety constraints.

2) Problem of fixed penalty methods:

The simulation results show that pre-training improves training efficiency and unseen-environment performance, while fixed collision penalties create a success–crash trade-off. The strongest reported unseen-map result is a 79% success rate for c1000+CPO.

  • 2) Problem of fixed penalty methods:: Models with high and low TRPO collision costs achieve different success and crash trade-offs, making fixed penalty tuning difficult.
  • 2) Problem of fixed penalty methods:: R-IL reaches CPO123’s final performance after less than one fifth of the iterations, at approximately 200 iterations.Pre-training provides a good initial policy and makes stochastic exploration more target-aimed.
  • C. Simulation results: The unseen-map evaluation uses two 10 m × 10 m maps, 100 random start-target pairs per map, and outcomes of success, timeout, or crash.A timeout occurs when the target is not reached within 5 min.
  • C. Simulation results: 79% success rate is achieved by c1000+CPO, which uses shortest-distance reward and complex pre-training.
  • C. Simulation results: R-IL200 achieves similar performance to pure RL from scratch with only 200 RL iterations, reducing RL training time by around 80%.
  • C. Simulation results: V2R has a similar success rate to CPO123 but an approximately 50% higher crash rate despite using a collision penalty of 1.0.

D. Real-world experiments

Real-world tests assess transfer from simulation using an unknown environment and examine crashes, manual intervention, and efficiency relative to a global planner. R-IL models generalize well, with c1000+CPO performing best and s10+CPOsparse performing surprisingly well.

  • The real-world environment is unknown to agents, while the models are trained purely in simulation.The tests probe robustness against sensor noise and actuation delays.
  • Table II reports crashes, joystick interference, and distance and time relative to move_base, including averages and maxima over five runs.
  • R-IL models generalize well to the unseen real-world environment and show similar performance, with c1000+CPO performing best.
  • Sparse-reward s10+CPOsparse performs surprisingly well despite using only sparse reward information.
  • The sparse-reward model is reported to require neither environment information nor reward shaping for strong generalization performance.

V. CONCLUSION

The study evaluates an end-to-end learning approach for map-less target-driven navigation that combines imitation learning with reinforcement learning. Simulation and real-world experiments show improved training efficiency, safer constraint-based learning, and reliable navigation in unseen environments, while training remains simulation-only.

  • The end-to-end navigation model maps raw sensor measurements and relative target locations to motion commands using combined IL and RL.
  • RL training time in R-IL can be reduced by around 80% while achieving similar final success rate and collision avoidance.Prior demonstrations improve exploration during RL, even with only 10 demonstrations, especially under sparse target rewards.
  • Constraint-based methods enforce collision constraints early, making exploration harder but enabling safer training and deployment.The authors recommend enforcing collision avoidance as a constraint rather than using a fixed reward penalty, especially with IL.
  • Trained navigation models reliably navigate unseen environments in both simulation and the real world.
  • Training was conducted purely in simulation; future work will investigate real-world human demonstrations and extension to dynamic environments.
Loading 1805.07095v2…