Source-linked AI summary
Goal-Driven Autonomous Exploration Through Deep Reinforcement Learning
Reinis Cimurs, Il Hong Suh, Jin Han Lee
TL;DR
The paper addresses goal-driven exploration in unknown environments, where reactive navigation can suffer from local optima and autonomous systems lack prior map information. It combines POI-based global waypoint selection with a TD3-trained local motion policy. Experiments report reliable navigation to global goals, obstacle avoidance without precomputed paths, and escape from local optima in static and dynamic settings.
Problem
Goal-driven autonomous exploration must navigate and map unknown environments without human control or prior information, while reactive DRL can encounter local optima.
Method
The system selects POI-derived waypoints for global guidance and integrates a TD3-based DRL motion policy as the local navigation layer.
Results
Experiments show reliable arrival at global goals, reactive obstacle avoidance without precomputed paths, and escape from local optima through combined global and local navigation.
Takeaways & Limitations
The combined system supports fully autonomous goal-directed exploration while recording the environment without direct human supervision.
Abstract
from arXiv · showhide
In this paper, we present an autonomous navigation system for goal-driven exploration of unknown environments through deep reinforcement learning (DRL). Points of interest (POI) for possible navigation directions are obtained from the environment and an optimal waypoint is selected, based on the available data. Following the waypoints, the robot is guided towards the global goal and the local optimum problem of reactive navigation is mitigated. Then, a motion policy for local navigation is learned through a DRL framework in a simulation. We develop a navigation system where this learned policy is integrated into a motion planning stack as the local navigation layer to move the robot between waypoints towards a global goal. The fully autonomous navigation is performed without any prior knowledge while a map is recorded as the robot moves through the environment. Experiments show that the proposed method has an advantage over similar exploration methods, without reliance on a map or prior information in complex static as well as dynamic environments.
I. INTRODUCTION
The paper targets fully autonomous, goal-directed exploration in unknown environments, where reactive navigation can become trapped in local optima. It combines waypoint-based global guidance with a DRL motion policy to navigate without human control or prior environmental information.
- SLAM commonly relies on human decisions about which unmapped areas to visit and in what order.
- Reactive DRL navigation can achieve high-precision control but may encounter local optima because it lacks global information.
- The proposed system extracts, evaluates, and selects points of interest as waypoints guiding a DRL motion policy toward a global goal.
- The work contributes a global waypoint-selection strategy, a TD3-based navigation network, and their integration to mitigate local optima.
II. RELATED WORKS
Prior work established autonomous mapping and neural-network navigation, but goal-driven exploration still requires combining learned local motion with broader global guidance. The paper proposes that combination for exploration toward a specified goal.
- Robotics research has used cameras, LiDARs, and sensor combinations for environment recording and autonomous localization.
- Neural navigation methods can produce deliberate motions or avoid obstacles, while some exploration systems still rely on planners.
- The proposed system combines a lightweight learned motion policy with a broader global navigation strategy for goal-driven exploration and mapping.
III. GOAL-DRIVEN AUTONOMOUS EXPLORATION
The exploration system uses global navigation with waypoint selection and mapping alongside DRL-based local navigation. POIs are converted into waypoint inputs for local action selection from sensor data.
- The navigation structure has two parts: global navigation with POI-based waypoint selection and mapping, and DRL-based local navigation.
- At each step, the selected waypoint is provided to the neural network in polar coordinates relative to the robot’s position and heading.
- The policy combines waypoint information with sensor data to calculate and execute an action.
A. Global Navigation
Global navigation generates candidate POIs from laser observations, filters unusable candidates, and selects a waypoint using the IDLE evaluation method. The score combines distance, goal proximity, and map information.
- Global Navigation: Intermediate waypoints are needed to guide the robot toward the goal while exploring alternative routes in an initially unknown environment.
- Global Navigation: The system extracts POIs from laser-reading gaps or sequential non-numerical readings that indicate free space.
- Global Navigation: Unusable or unreachable POIs are removed, including candidates near obstacles, in visited areas, or unreachable over multiple steps.
- Global Navigation: IDLE selects the candidate with the smallest score as the optimal waypoint for local navigation.
- Global Navigation: The candidate score combines robot-to-POI distance, candidate-to-goal distance, and map-information terms, with distance limits controlling discounting.
B. Local Navigation
The local navigation layer replaces a conventional local planner with a TD3-based motion policy trained in simulation. It combines laser observations and waypoint coordinates, using rewards for goal reaching, collision avoidance, and motion behavior.
- Policy architecture: The approach replaces the local-planner layer in a planning-based navigation stack with a neural-network motion policy trained separately in simulation.The learned policy performs local motion between waypoints.
- Policy architecture: TD3 uses continuous actions, with laser readings and waypoint polar coordinates forming the actor-network input state.The actor outputs linear and angular velocity parameters.
- Critic evaluation: The critic networks evaluate state-action pairs and select the lower Q value to limit overestimation.Their parameter updates are delayed, allowing divergence between the two critics.
- Reward design: The reward function assigns positive reward near the goal, negative reward after collision, and otherwise uses linear and angular velocity.A delayed attribution method propagates goal reward backward across previous steps.
- Policy outcome: The learned policy reaches local goals while avoiding obstacles directly from laser inputs.Goal reward is attributed decreasingly over the preceding n steps.
C. Exploration and Mapping
The robot follows selected waypoints toward the global goal while simultaneously exploring and mapping the environment. Mapping combines laser and odometry data into an occupancy grid.
- Navigation and mapping: Waypoints guide the robot toward the global goal, after which the robot navigates directly to the goal when nearby.The environment is explored and mapped throughout this process.
- Navigation and mapping: Mapping uses laser and robot odometry sensors to construct an occupancy grid map.
- Navigation and mapping: The fully autonomous exploration procedure is specified in an algorithm that integrates waypoint following, navigation, and mapping.
IV. EXPERIMENTS
The experiments validate the proposed goal-driven exploration system in real-life settings, using an algorithm that repeatedly updates sensing, mapping, POIs, waypoint selection, and TD3 actions. Training environments randomize obstacle locations across episodes.
- Experimental validation: Experiments were conducted in real-life settings with varying complexity to validate the goal-driven exploration system.
- Exploration procedure: The exploration algorithm reads sensor data, updates the map, obtains POIs, and selects a waypoint before obtaining and executing a TD3 action.When the robot is sufficiently close to the global goal, the global goal becomes the waypoint.
- Training environments: Training environments randomize data by changing the locations of four box obstacles on every episode.The figure depicts three example arrangements, with blue fields representing laser readings and range.
A. System Setup
The local policy was trained in Gazebo with ROS commands and then transferred to compact computing platforms for full-system experiments. The robot used calibrated laser sensing for safe navigation.
- Training setup: The TD3 network was trained in Gazebo for 800 episodes, requiring approximately 8 hours of training.Episodes ended at goal reaching, collision, or 500 steps.
- Deployment setup: The trained network was transferred to an Intel NUC mini-PC for quantitative experiments and to a laptop for qualitative experiments.The platforms used different processor, memory, and graphics configurations.
- Robot sensing: Two laser sensors mounted at different heights provided readings up to 10 meters across 180° in front of the robot.Readings were calibrated and grouped into 21 bags using each group’s minimum value.
B. Quantitative Experiments
The quantitative experiments compare GDAE with frontier-, local-planner-, reinforcement-learning-, and path-planning methods across indoor environments. GDAE reached goals quickly while addressing local optima and complex obstacles, although GD-RL results included human-assisted runs.
- Comparative setup: GDAE was compared with Nearest Frontier, Goal-Driven Reinforcement Learning, Local Planner Autonomous Exploration, and a known-map Path Planner across two indoor environments.Experiments were repeated five times per method, recording distance, time, mapped area, and goal success.
- First quantitative experiment: In the first environment, all methods reached the global goal, while GDAE used comparable distance and less time than similar methods.Planning-based methods recalculated paths after waypoint selection, whereas GDAE began moving toward new waypoints immediately.
- First quantitative experiment: GD-RL became trapped in a local optimum in the first environment and escaped by following a wall, increasing travel distance and time.The environment contained smooth walls and multiple local optima.
- Second quantitative experiment: In the second experiment, GDAE reached the global goal in the shortest time with comparable distance despite complex obstacles and a locally optimal start.The environment included furniture, chairs, tables, and glass walls.
- Second quantitative experiment: NF and LP-AE failed on obstacles misclassified as sensor noise, while GD-RL initially failed to escape and required human intervention on later successful runs.The reported results for GD-RL were calculated over successful runs.
C. Qualitative Experiments
Qualitative experiments tested GDAE in indoor environments containing local optima, clutter, long routes, backtracking, and incomplete maps. Across these scenarios, the robot explored its surroundings, escaped confined or obstructed areas, and reached the global goal.
- Local-optimum scenarios: In a hallway, GDAE backtracked from a room corner after detecting a local optimum and reached the global goal through hallways.The goal was at coordinate (22, 12).
- Local-optimum scenarios: When starting inside a local optimum without prior map or POI information, GDAE explored the local environment and navigated out of a confined hallway.This scenario used a goal at coordinate (23, 2).
- Local-optimum scenarios: In a cluttered environment, GDAE navigated through obstacles from a local optimum toward the global goal.The experiment used a goal at coordinate (10, 10).
- Larger-scale scenarios: In larger-scale hallway and parking-lot scenarios, the robot followed selected waypoints toward distant goals, including with backtracking and incomplete map information.The parking-lot case began with an incomplete map and an obstructed starting pose.
- Overall observation: Across experiments, GDAE explored unknown environments, reactively avoided obstacles without pre-calculated paths, and escaped local optima while reaching global goals.The qualitative findings include static and dynamic settings.
V. CONCLUSIONS
The system combines reactive local navigation with global waypoint guidance to reach designated goals while recording unknown environments without direct human supervision. Its current training is robot-specific, and future work targets broader robot generalization and obstacles beyond sensor range.
- The system reaches designated goals while recording environments without direct human supervision.
- Combining reactive local and global navigation strategies alleviates shortcomings of end-to-end motion and mitigates local-optimum behavior.
- The obtained results work reasonably close to the optimal solution from a path planner using an already known environment.
- Motion-policy training used the same robot model as the real-life experiments, limiting demonstrated generalization across robot types.The authors propose separate robot-dynamics inputs and corresponding training for broader applicability.
- A long short-term memory architecture is proposed as future work to alleviate local-optimum behavior and help avoid obstacles beyond the current sensor range.