PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning based Automatic Exploration for Navigation in Unknown Environment

Haoran Li, Qichao Zhang, Dongbin Zhao

arXiv:2007.11808v1cs.RO

TL;DR

The paper addresses automatic exploration in unknown environments, where rule-based methods are limited and end-to-end learning suffers from training and simulation-to-reality difficulties. It decomposes exploration into mapping, decision, and planning modules and uses DRL to choose goals from partial maps. Experiments report improved learning efficiency, generalization, avoidance of frontier-method failures, and tolerable transfer to a physical robot.

  • Problem

    Automatic exploration in unknown environments is limited by rule-heavy approaches and by end-to-end methods’ long training, uncertain real-world generalization, and sensitivity to physical-robot errors.

  • Method

    The paper constructs independent mapping, decision, and planning modules and trains a DRL decision algorithm to select goals from partial maps.

  • Results

    Experiments report faster learning than end-to-end methods, better generalization across maps, avoidance of a frontier-based failure case, and tolerable transfer from simulation to a physical robot.

  • Takeaways & Limitations

    Combining DRL decision-making with existing navigation algorithms supports efficient and adaptable exploration while retaining modularity and physical-robot transferability.

Abstract

from arXiv · show

This paper investigates the automatic exploration problem under the unknown environment, which is the key point of applying the robotic system to some social tasks. The solution to this problem via stacking decision rules is impossible to cover various environments and sensor properties. Learning based control methods are adaptive for these scenarios. However, these methods are damaged by low learning efficiency and awkward transferability from simulation to reality. In this paper, we construct a general exploration framework via decomposing the exploration process into the decision, planning, and mapping modules, which increases the modularity of the robotic system. Based on this framework, we propose a deep reinforcement learning based decision algorithm which uses a deep neural network to learning exploration strategy from the partial map. The results show that this proposed algorithm has better learning efficiency and adaptability for unknown environments. In addition, we conduct the experiments on the physical robot, and the results suggest that the learned policy can be well transfered from simulation to the real robot.

I. INTRODUCTION

Automatic exploration builds a complete map in unknown environments, but traditional rule-based and end-to-end learning approaches face coverage, training, and transfer challenges. The paper addresses these issues with a modular framework and a DRL-based decision method using partial maps.

  • Motivation: Automatic exploration requires robots to build an environmental map while moving without prior knowledge, reflecting robotic-system adaptability.Applications include rescue-robot search and robot-sweeper operation in unknown environments.
  • Traditional methods: Frontier-based methods select boundary points, while information-based methods use Shannon entropy but often lack a mathematical optimum.Hybrid methods evaluate frontier-selected candidates with entropy before choosing the next point.
  • Learning-based methods: Existing intelligent-method research has applied DRL to exploration and navigation, including POMDP optimization and virtual-environment agents with memory or intrinsic curiosity.Examples include A3C with LSTM and curiosity-based world-model prediction.
  • Learning-based methods: End-to-end learning maps raw sensor data directly to control policy, forcing agents to learn effective movement and exploration from raw inputs.This approach ignores information from mature robotic navigation methods and increases training difficulty.
  • Proposed approach: The proposed framework combines mapping, decision, and planning modules, using a DRL decision method that selects target locations from partial grid maps.The framework is designed to combine established robotic mapping and navigation methods.
  • Proposed approach: The contributions target reduced training difficulty and simulation-to-robot transfer, while improving exploration efficiency and adaptability through DRL combined with classical robotics.The framework is contrasted with raw-sensor end-to-end control and traditional exploration methods.

III. PROBLEM STATEMENT

The problem statement identifies long convergence, uncertain real-world generalization, and physical sensor or mechanism errors as barriers to end-to-end exploration. The paper separates exploration into independent decision, mapping, and planning modules, with DRL selecting goals from mapped surroundings.

  • III. PROBLEM STATEMENT: End-to-end exploration methods require massive trial-and-error training, creating long convergence times, uncertain real-world generalization, and possible incompatibility with physical robots.These issues arise because simulation environments differ from reality and physical robots introduce mechanism and sensor errors.
  • III. PROBLEM STATEMENT: The proposed architecture separates exploration into decision, mapping, and planning modules so robot-specific planning can share a decision module.Planning algorithms can ensure safe movement in known environments without learning, while decision-making focuses on exploration strategy.
  • Decision module: The decision module receives the surroundings map and selects the next goal point for the planning module.The framework passes mapping output to decision-making, then sends the selected goal to planning.
  • Decision module: The decision function uses robot-position history and the built surroundings map to choose the robot’s next goal point.The paper identifies these inputs as l0:t and mt and the output as gt.
  • Decision module: Unlike frontier-based rule stacking, the DRL decision method learns visual exploration features and feasible strategies through trial and error for more complex scenes.The stated aim is a concise and general decision function.

B. Planning module

The planning and mapping components convert selected goals and sensor data into feasible robot control while maintaining the surroundings map. The framework emphasizes modularity, allowing these components to use established or varied implementations.

  • B. Planning module: The planning module determines a feasible trajectory from the robot’s current position to the selected goal point.Its role is to produce a trajectory suitable for robot control.
  • B. Planning module: Global planning finds a shortest map path, while local planning converts that path into a trajectory and adjusts it using real-time sensor data.The two stages divide map-based route selection from real-time trajectory control.
  • B. Planning module: The implemented planner uses A∗ for global search and timed-elastic-band for local conversion from a discrete path to robot velocity.The local planner can avoid obstacles absent from the built map by using real-time LiDAR point clouds.
  • Mapping module: The mapping module processes sequential control and observation data to produce the robot pose and surroundings map at each timestamp through SLAM.The paper distinguishes filter-based and graph-based SLAM approaches.
  • Framework design: The framework keeps mapping, decision, and planning modules mutually independent, allowing each module to be implemented with various methods.This modularity is presented as more flexible and interpretable than end-to-end methods.

IV. DRL-BASED DECISION METHOD

The paper formulates exploration as sequential decision-making over occupancy-grid maps, using DRL to choose target points while classical planning handles navigation. The design incorporates safety and efficiency rewards and separates decision from planning for better real-robot compatibility.

  • Objective function: The exploration objective balances map similarity to reality against path length, but the real map is unavailable during exploration.The estimated map and exploration path length define the objective, while unknown environments prevent direct minimization.
  • Objective function: The occupancy grid represents each cell as unknown, free, or occupied, enabling map uncertainty to guide exploration.The map uses occupied probabilities and Shannon-entropy-inspired uncertainty evaluation.
  • DRL decision method: Q-learning provides the decision basis, with deep networks approximating Q-values from experience replay and iterative value updates.The paper contrasts high-dimensional raw sensor inputs with map-based decision inputs to address sharply enlarged state spaces.
  • Decision framework: The decision module receives the map and current and historical robot positions, then produces the next goal point.The planning module subsequently plans a path from the current position to that goal.
  • Reward design: Safety rewards penalize targets too close to obstacles, while a terminal action receives positive reward only when explored-region ratio ρ exceeds 0.85.The reward design combines collision avoidance with exploration completion and uses α for training stability.

C. Action Space and Network Architecture

The action space consists of regularly sampled points on an occupancy grid, and a fully convolutional Q-network estimates their values. Its convolutional design supports variable map sizes with fewer parameters than flattened fully connected DQNs.

  • Action space: Exploration actions are discrete sampling points regularly rasterized from the occupancy grid map.Rasterized sampling reduces the search space for selecting the next target.
  • Network architecture: The network estimates values for map actions using convolutional layers whose output dimension matches the action space.The architecture is shown as a fully convolutional Q-network.
  • Network architecture: FCQN combines a score map for pointwise advantages with pooled whole-map features for terminal-action advantage and state-value estimation.The Q-value of each point action is derived from these value and advantage components.
  • Network architecture: Unlike original DQNs, FCQN uses fewer convolutional parameters and accepts different input map sizes, reducing overfitting risk and improving adaptability.Original DQNs flatten feature maps before fully connected layers, requiring fixed input dimensions.

D. Auxiliary task

The auxiliary task addresses the training difficulty caused by high-dimensional map inputs by teaching FCQN to segment navigation-relevant map edges. The segmentation branch identifies obstacle contours and frontiers.

  • Motivation: Map-image inputs enlarge the state-action search space, requiring more analysis of possible combinations and making training harder.Prior feature-based approaches can reduce state dimension and exploration time.
  • Auxiliary task: The auxiliary task performs edge segmentation because obstacle contours support collision avoidance and free–unknown boundaries define frontiers for candidate generation.The paper treats both obstacle contours and frontier boundaries as map edges.
  • Auxiliary task: A decoder-based segmentation branch is appended to FCQN, producing an output with the same dimensions as the input map.The decoder consists of two deconvolution operations after lower feature maps.
  • Auxiliary task: The segmentation loss classifies each map pixel into obstacle contours, frontiers, or other classes.The loss uses binary class indicators and predicted class probabilities.
  • Auxiliary task: Lower convolutional layers are updated through two-part back-propagation combining the decision and segmentation objectives.The learning rate and segmentation-task balance parameter govern this update.

V. WHOLE SYSTEM

The whole system combines mapping, DRL-based goal selection, and classical path planning. Robot odometry and point-cloud data feed mapping, while A* receives the selected goal.

  • Mapping module: The robot sends odometry and point-cloud data from environmental scanning to the mapping module.The paper uses gmapping to build the map and obtain the robot pose.
  • Decision module: FCQN and AFCQN use the built map and localization as inputs and output the next goal point.AFCQN is used as the example decision algorithm in the system workflow.
  • Planning module: The global planner receives the selected goal to plan the robot’s route.The system therefore combines learned decision-making with conventional navigation components.

VI. EXPERIMENTS

The experiments compare the DRL-based decision method with different algorithms in simulated and physical worlds, using graph-based mapping and A* global planning within the exploration framework.

  • VI. EXPERIMENTS: Experiments evaluate different exploration algorithms in both simulated and physical environments.The framework uses a DRL-based decision module, graph-based mapping, scan matching, graph optimization, and A* global planning.

A. Evaluation Metrics

The evaluation measures map completeness, travel cost, and exploration efficiency, then analyzes training behavior across maps and trials. AFCQN learns to reduce redundant movements while emphasizing explored-region completeness.

  • Evaluation Metrics: Explored region rate measures the completeness of the map built during exploration.It is defined as the number of explored free cells divided by the number of free cells in the real map.
  • Evaluation Metrics: Exploration efficiency represents entropy reduced per unit path length and balances map coverage against travel cost.Larger explored regions generally require longer paths, making coverage and path length inconsistent objectives.
  • C. Training Analysis: AFCQN reduces redundant movements through Q-value iteration and achieves better return during training.Edge segmentation initially speeds exploration but can cause movements that do not add new free cells; later Q-value updates crop these movements.
  • C. Training Analysis: All algorithms completely explore the training map in 50 trials, while AFCQN has the highest explored region rate and FCQN the highest exploration efficiency.AFCQN prioritizes map completeness, whereas FCQN avoids redundant movement by selecting the terminal action in time.
  • C. Training Analysis: Within 25 meters, AFCQN has better comprehensive performance than the other evaluated algorithms.FCQN’s explored-region rate is distributed around 0.8, while DQN is around 0.9 except in failed cases.

D. Generalization Analysis

The generalization experiments use test maps with layouts and sizes differing from the training map. Fully convolutional AFCQN and FCQN adapt to these changes, with AFCQN favoring coverage and FCQN efficiency.

  • D. Generalization Analysis: Test maps vary from the training map in layout and, for two maps, size.The experiments assess generalization across both same-sized maps with different layouts and differently sized maps.
  • D. Generalization Analysis: All algorithms generalize well to test maps with layouts different from the training map.AFCQN achieves the highest explored region rate in test map 1, while AFCQN and FCQN tie in test map 2.
  • D. Generalization Analysis: Fully convolutional AFCQN and FCQN adapt their output dimensions to different map sizes, unlike fully connected DQN.The convolutional methods preserve a downsampling rate while generating the action space.
  • D. Generalization Analysis: AFCQN has higher explored region rate, whereas FCQN has higher exploration efficiency on new environments.Changes in layout and scale increase the standard deviation of explored region rate.
  • D. Generalization Analysis: Map layout influences algorithm performance: spacious layouts favor FCQN’s exploration efficiency, while AFCQN emphasizes coverage.The training map and test map 2 share many rooms and doors, whereas test maps 1 and 3 are relatively spacious.

E. Relationship to Frontier-based Methods

AFCQN uses learned whole-map information to choose actions beyond frontier centers and to stop exploration. Compared with frontier-based methods, it generally trades some coverage for efficiency and avoids a documented stopping failure case.

  • E. Relationship to Frontier-based Methods: AFCQN initially values positions near frontiers but later learns choices that maximize newly revealed free cells.Its decisions consider both completeness and total path length rather than selecting frontier centers alone.
  • E. Relationship to Frontier-based Methods: AFCQN’s decision process represents occupancy maps, frontiers, and Q-values across eight exploration steps before selecting a terminal action.Each step contains the built occupancy grid map, frontier representation, and Q-value map.
  • E. Relationship to Frontier-based Methods: AFCQN selects a nearer frontier before a more expansive one when this ordering better balances completeness and whole-path length.In the described case, it chooses the upleft point first and the upright point second.
  • E. Relationship to Frontier-based Methods: The frontier-based method achieves higher explored region rate but usually follows longer paths because it continues until no reachable frontier remains.AFCQN estimates the whole map and selects the terminal action in time, producing higher exploration efficiency in the first three test maps.
  • E. Relationship to Frontier-based Methods: Under distance constraints, the frontier-based method outperforms AFCQN but can stop prematurely when the robot overlaps a frontier center.AFCQN includes the robot’s current and last decision positions, preventing the documented overlap case.

F. Physical World Experiments

The physical-robot experiments compare AFCQN with simulation and other exploration methods under differing sensor errors and environments. AFCQN’s decision process transfers across domains, while its map-completeness monitoring improves exploration efficiency.

  • Experimental Platform: The robot platform uses a mecanum-wheel chassis for omnidirectional movement and an RPLidar selected to resemble the simulated LiDAR.The algorithm outputs decision points rather than direct robot control, facilitating physical-robot transfer.
  • Exploration Results: AFCQN achieves the best explored efficiency because it monitors built-map completeness and stops exploration in time.Frontier-based methods achieve the highest explored region rate in both physical and simulated experiments.
  • Robustness Across Domains: DQN is sensitive to input differences and fails to explore the physical world in several trials.Sensor errors and environmental differences produce different observations between simulation and reality.
  • Simulation-to-Reality Transfer: AFCQN’s real-world exploration trajectories mostly overlap with its simulation trajectories, implying similar decision processes across the two experiments.The comparison is shown in Figure 16.
  • Experimental Configuration: Different local-planner parameters are used in simulation and reality to accommodate their different kinematic models, without changing exploration path-length variance.The local planner determines path length.
Loading 2007.11808v1…