Source-linked AI summary
Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation
Lei Tai, Giuseppe Paolo, Ming Liu
TL;DR
Traditional mobile-robot navigation relies on obstacle maps and precise sensing, motivating a mapless alternative for sparse observations. The paper trains an asynchronous continuous-control deep-RL planner end-to-end in simulation and reports transfer to unseen virtual and real environments without fine-tuning, with collision-free navigation demonstrated.
Problem
Traditional planners depend on obstacle maps and precise laser sensing, while rapidly generating navigation behaviors from sparse range information without a map remains challenging.
Method
An asynchronous deep-RL planner is trained end-to-end from sparse 10-dimensional range findings and relative target information to output continuous robot velocities.
Results
The learned planner transfers directly to unseen virtual and real environments without fine-tuning and navigates the nonholonomic robot to targets without collisions.
Takeaways & Limitations
Mapless continuous-control planning can provide a low-cost navigation solution for indoor robots using sparse range sensing.
Takeaways & Limitations
The planner produces more tortuous paths than Move Base, possibly because it lacks memory of previous observations and long-term prediction ability.
Abstract
from arXiv · showhide
We present a learning-based mapless motion planner by taking the sparse 10-dimensional range findings and the target position with respect to the mobile robot coordinate frame as input and the continuous steering commands as output. Traditional motion planners for mobile ground robots with a laser range sensor mostly depend on the obstacle map of the navigation environment where both the highly precise laser sensor and the obstacle map building work of the environment are indispensable. We show that, through an asynchronous deep reinforcement learning method, a mapless motion planner can be trained end-to-end without any manually designed features and prior demonstrations. The trained planner can be directly applied in unseen virtual and real environments. The experiments show that the proposed mapless motion planner can navigate the nonholonomic mobile robot to the desired targets without colliding with any obstacles.
I. INTRODUCTION
The paper addresses mapless navigation for nonholonomic mobile robots using continuous deep reinforcement learning. It targets navigation from sparse range information without relying on a prior obstacle map.
- Mapless navigation: Traditional mobile-robot planners depend on prior obstacle maps built from dense, precise laser range findings.Map construction and updating are time-consuming, while local cost-map prediction also depends on dense sensing.
- Mapless navigation: Generating appropriate navigation behaviors from sparse range information without an obstacle map remains challenging.
- Deep reinforcement learning: The proposed planner learns obstacle avoidance for a nonholonomic differential-drive robot through asynchronous deep reinforcement learning in virtual environments.
- Mapless navigation: The planner uses local observations and the target position relative to the robot to generate navigation behavior directly.
3) From virtual to real world:
The paper reduces the virtual-to-real transfer problem by training with a highly abstracted 10-dimensional sparse range observation. It reports direct generalization to a real robot without fine-tuning.
- From virtual to real world: Training uses 10-dimensional sparse range findings sampled from selected angles of raw laser measurements.The abstraction is intended to reduce the gap between virtual and real environments and support low-cost range sensors.
- From virtual to real world: The planner is trained end-to-end from scratch with asynchronous deep reinforcement learning and outputs continuous linear and angular velocities.
- From virtual to real world: The learned planner generalizes to a real nonholonomic differential robot without fine-tuning on real-world samples.
- Related approaches: Earlier learning-based approaches used image inputs, semantic information, or demonstrations, while some produced only discrete actions.Demonstration-based methods are dependent on demonstration information and require time-consuming data collection.
B. Deep Reinforcement Learning
The paper selects and extends continuous-control deep reinforcement learning for mapless mobile-robot navigation. It emphasizes asynchronous sampling to address data-collection demands and limitations of discrete-action methods.
- Deep reinforcement learning: The method chooses DDPG for continuous control and extends it to an asynchronous version to improve sampling efficiency.DDPG is selected because A3C requires parallel simulation environments, while DDPG uses fewer training parameters than NAF.
- Deep reinforcement learning: The paper focuses on a mapless motion planner based on low-dimensional range findings for continuous control of differential-drive mobile robots.
A. Asynchronous Deep Reinforcement Learning
Asynchronous DDPG separates sample collection from training so these processes can run in parallel. The paper evaluates this design on Pendulum-v0 and defines the planner as a state-to-velocity translation function.
- A. Asynchronous Deep Reinforcement Learning: Asynchronous DDPG moves sample collection to a separate thread from the original DDPG training process.The design can also use multiple data-collection threads.
- A. Asynchronous Deep Reinforcement Learning: On Pendulum-v0, asynchronous DDPG increases Q-value faster than original DDPG, indicating more efficient policy learning.
- A. Asynchronous Deep Reinforcement Learning: Almost four times more samples are collected per step by parallel asynchronous DDPG than by original DDPG.
- B. Problem Definition: The motion planner maps the instant robot state to the next-time-step velocity while requiring sufficient control frequency for immediate reactions.The state combines raw sensor observation, relative target position, and the previous robot velocity.
C. Network Structure
The planner is trained with asynchronous DDPG using merged sensor, action, and target information, while actor and critic networks produce continuous velocity-control outputs and Q-value estimates.
- Asynchronous DDPG trains the mapless motion-planning model.The problem is formulated as reinforcement learning and trained with extended asynchronous DDPG.
- The network structure uses Dense layers for fully connected processing and a Merge layer to combine input blobs.
- The actor receives a 14-dimensional vector combining 10-dimensional laser findings, the previous action, and relative target position.Range findings are sampled between -90 and 90 degrees and normalized to (0,1).
- The actor outputs angular and linear velocity commands, constrained by tanh and sigmoid activations.Angular velocity is constrained to (−1,1), while linear velocity is constrained to (0,1).
- The critic processes the state through three fully connected layers, merges the action in the second layer, and predicts a Q-value.The final Q-value uses a linear activation.
D. Reward Function Definition
The reward function distinguishes successful arrival and collision from ordinary progress, rewarding target approach through changes in distance during otherwise continuing episodes.
- The three reward conditions are passed directly to the critic network without clipping or normalization.
- Arrival receives a positive reward and collision receives a negative reward, with both conditions terminating the training episode.Arrival is detected by a distance threshold, while collision is detected from minimum range findings.
- Otherwise, the reward equals the change in target distance from the previous timestep multiplied by c_r.The distance-difference reward motivates the robot to move closer to the target.
IV. EXPERIMENTS
Training uses two simulated indoor environments with different obstacle compactness, learning both models from scratch and showing distinct Q-value dynamics between environments.
- Training in simulation: Both environment-specific models are learned from scratch using a Turtlebot platform.
- Training in simulation: Training uses equal actor and critic learning rates of 0.0001 and runs for 0.8m steps on one GPU for almost 20 hours.The optimizer is Adam, and the authors report that performance effects were not dependent on hyperparameter tuning.
- Training in simulation: Two 10 × 10 m^2 indoor V-REP environments contain walls and differently shaped obstacles for virtual training.Env-2 places obstacles more compactly around the robot’s initial position than Env-1.
- Training in simulation: Env-2 receives more collision samples, producing a much smaller mean Q-value than Env-1, while its mean Q-value increases much faster.Figure 5 plots mean Q-values of training-batch samples, with different y-axes for the two environments.
B. Evaluation
Evaluation deploys the planner on a Kobuki-based Turtlebot using sparse laser observations and externally provided localization for target-position calculation, with comparison against Move Base baselines.
- The robot’s real-time position is provided by AMCL to calculate the target position in polar coordinates.The paper does not test planner effects with different localization methods.
- Real-time evaluation uses a Kobuki-based Turtlebot, a SICK TiM570 laser range finder, and an onboard Intel Core i7-4700 laptop.The figure specifies that only 10-dimensional sparse range findings are used in evaluation.
- The evaluation compares deep-RL planners with Move Base, whose standard version uses full laser information for local cost-map calculation.A 10-dimensional Move Base baseline is implemented using range findings from specific angles, expanded through an RBF-kernel Gaussian-process regression.
2) Virtual Environment Evaluation:
In virtual tests, deep-RL mapless planners completed the target-sequence navigation collision free, whereas 10-dimensional Move Base failed; the learned planners also queried substantially faster.
- Virtual trajectory tracking: Deep-RL mapless planners completed all virtual navigation tasks collision free in Env-1 and Env-2.The 10-dimensional Move Base failed because incorrect local cost-map prediction prevented it from finding a path without human intervention.
- Virtual trajectory tracking: 10-dimensional Move Base was not able to finish the navigation tasks.
- Quantitative evaluation: The evaluation compared max control frequency, traveling time, and traveling distance across the planners.Traveling time and distance covered all 10 target positions.
- Quantitative evaluation: Almost 1ms query time made trained mapless planners 7 times faster than the map-based motion planner.Env-2 required almost the same total navigation time as 10-dimensional Move Base, despite a non-shortest path; Env-1 trajectories were less smooth.
3) Real Environment Evaluation:
In real-world tests, the Env-2 planner completed all tasks without collisions, while 10-dimensional Move Base failed in a narrow area and required human intervention. The authors position the method as a low-cost indoor-robot solution rather than a replacement for map-based planning.
- Real-world trajectory tracking: 10-dimensional Move Base could not cross the narrow area or find an effective path, so human intervention completed the task.Intervention segments were marked in black in Fig. 9.
- Real-world trajectory tracking: The Env-2 model completed all real-world navigation tasks, and obstacle collision never happened.A rotating recovery behavior was developed for the mapless planner, although narrow routes were not always traversed smoothly.
- Transfer to unseen environments: Virtual and real-world experiments showed direct transfer of the deep-RL mapless planner to unseen environments.The trained planner’s trajectories were influenced by the training environment; Env-2 used closer obstacles and was more aggressive.
- Discussion and limitations: The mapless planner produced more tortuous paths than original and low-dimensional map-based planners.The authors attribute this possibly to lacking previous-observation memory and long-term prediction, identifying LSTM and RNN as future-work options.
- Discussion and limitations: The method targets low-cost indoor service robots and is not intended to replace map-based planning in large-scale complex environments.The authors state that environmental maps can provide reliable navigation paths in those settings.
VI. CONCLUSION
The paper concludes that its mapless planner uses sparse range and target information for continuous-control navigation and transfers from virtual training to unseen real environments without fine-tuning. Compared with low-dimensional map-based planning, it was more robust in extremely complicated environments.
- The planner was trained end-to-end from scratch with continuous-control deep reinforcement learning.
- Using 10-dimensional sparse range findings and relative target position, it outputs continuous linear and angular velocities.
- The planner transferred directly from virtual training to unseen real environments without fine-tuning.
- Compared with the low-dimensional map-based motion planner, the approach was more robust to extremely complicated environments.