Source-linked AI summary
Mobile Robot Path Planning in Dynamic Environments through Globally Guided Reinforcement Learning
Binyu Wang, Zhe Liu, Qingbiao Li, Amanda Prorok
TL;DR
Large dynamic environments make it difficult for mobile robots to reach goals efficiently while avoiding moving obstacles, and re-planning can introduce detours. The paper proposes G2RL, which combines global guidance, local reinforcement learning, and an environment-independent reward structure. Across varied environments and multi-robot settings, it generalizes well, outperforms distributed methods, and performs similarly to centralized benchmarks.
Problem
Dynamic-environment path planning must balance efficient goal reaching with conflict avoidance, while existing RL planners face sparse rewards, overfitting, and scalability challenges.
Method
G2RL combines a globally planned path with a local RL planner, using dense environment-independent rewards and fully distributed control for multi-robot planning.
Results
G2RL generalizes across map types, obstacle densities, and robot numbers, outperforming existing distributed methods while performing similarly to centralized approaches.
Takeaways & Limitations
A fixed-sized, globally guided local planner supports scalable end-to-end path planning in arbitrarily large environments without requiring global dynamic-obstacle trajectories.
Abstract
from arXiv · showhide
Path planning for mobile robots in large dynamic environments is a challenging problem, as the robots are required to efficiently reach their given goals while simultaneously avoiding potential conflicts with other robots or dynamic objects. In the presence of dynamic obstacles, traditional solutions usually employ re-planning strategies, which re-call a planning algorithm to search for an alternative path whenever the robot encounters a conflict. However, such re-planning strategies often cause unnecessary detours. To address this issue, we propose a learning-based technique that exploits environmental spatio-temporal information. Different from existing learning-based methods, we introduce a globally guided reinforcement learning approach (G2RL), which incorporates a novel reward structure that generalizes to arbitrary environments. We apply G2RL to solve the multi-robot path planning problem in a fully distributed reactive manner. We evaluate our method across different map types, obstacle densities, and the number of robots. Experimental results show that G2RL generalizes well, outperforming existing distributed methods, and performing very similarly to fully centralized state-of-the-art benchmarks.
I. INTRODUCTION
The paper addresses dynamic-environment path planning challenges by combining global guidance with local reinforcement learning. G2RL uses environment-independent dense rewards and distributed control, achieving strong performance across planning settings.
- Motivation: Existing planners struggle with dynamic obstacles, sparse rewards, overfitting, and exponential state-space growth in large multi-robot systems.Re-planning can cause unnecessary detours, while prediction-based methods may produce freezing-robot failures.
- Approach: G2RL combines a globally optimal path with a local RL planner that uses environmental information to avoid static and dynamic obstacles.The global guidance is calculated first, while local actions attempt to follow it and resolve conflicts during motion.
- Contributions: The hierarchical framework supports end-to-end learning with a fixed-sized model, allowing navigation in large-scale environments.Global guidance directs the robot toward its destination without expanding the learning model with environment size.
- Contributions: The reward structure provides dense, environment-independent rewards without requiring strict guidance following, encouraging exploration and generalization.This design is intended to support scalability across environments while avoiding unnecessary detours.
- Contributions: The multi-robot application uses fully distributed control that can scale to an arbitrary number of robots.The paper applies the approach to multi-robot path planning rather than restricting evaluation to a single robot.
- Results: G2RL outperforms re-planning and distributed baselines while performing comparably to centralized approaches that assume global dynamic knowledge.Single-robot performance remains consistent across map types and obstacle counts, and the multi-robot method compares similarly with centralized benchmarks.
II. BACKGROUND AND RELATED WORK
The paper frames path planning as a choice between global and local methods, then defines a dynamic-grid planning problem using global shortest-path guidance and local observations. The robot must select actions that reach its goal efficiently while avoiding conflicts.
- Traditional path planning: Global planners assume complete environmental information, whereas local planners operate when part or nearly all environmental information is unknown.Global planning includes graph- and sampling-based methods, while local navigation can be effective in dynamic environments.
- Learning-based approaches: Reinforcement learning uses trial-and-error, but growing environments require exploration of more states before rewards are received, making learning more difficult.Imitation learning can provide dense rewards but may inherit expert-data bias and restrict exploration of alternative solutions.
- Problem formulation: The environment is represented as a discrete grid with free cells, static obstacles, dynamic obstacle positions, and motion-conflict constraints.Traversable edges connect neighboring free cells, while dynamic obstacles must avoid occupying conflicting positions.
- Problem formulation: Global guidance is the shortest traversable path from the start cell to the goal and is selected before local action generation.The paper notes that multiple shortest paths may exist and randomly chooses one instance.
- Problem formulation: The robot assumes known static obstacles and one-time global guidance computation, but dynamic-obstacle trajectories are not assumed known.Local observations include nearby environmental information within the robot’s field of view.
- Problem formulation: At each time step, the planner uses local and historical information together with guidance to choose an action that minimizes steps while avoiding static and dynamic conflicts.The action set is Up, Down, Left, Right, or Idle, and Figure 1 depicts the transformed inputs and action-producing network.
IV. RL-ENHANCED HIERARCHICAL PATH PLANNING
This section introduces the system structure before detailing the approach. The local RL planner is organized into modules for composing inputs, processing spatial information, and processing temporal information.
- The section first describes the overall system structure and then presents details of the approach.
- The local RL planner is presented as a modular system rather than as an undifferentiated network.
- The section establishes the organization used for the subsequent description of the approach.
A. System Structure
The system combines global guidance with a local reinforcement-learning planner that uses dense, guidance-aware rewards to avoid obstacles while permitting detours. Guidance cells are removed when rewarded, preventing repeated collection of the same path reward.
- Global guidance supplies long-term information, enabling frequent feedback even in large or locally complex environments.
- The reward gives a small penalty off guidance, a larger collision penalty, and a positive reward proportional to eliminated guidance cells.The positive reward is r1 + Ne × r3, where Ne counts guidance cells removed before rejoining the path.
- The robot may detour around a dynamic obstacle and continue moving away from guidance until rejoining it at a later guidance cell.In the example, the robot avoids an obstacle at t = 0 and rejoins guidance at t = 7 after seven cells are removed.
- Removed guidance cells become normal free cells, so each guidance-cell reward can be collected only once.
- Unlike imitation-learning approaches, the reward does not require strict stepwise adherence to global guidance and supports training from scratch.
C. Local RL Planner
The local planner encodes local obstacle observations, guidance, and temporal history, then uses a DDQN architecture to estimate action values. Spatial features are extracted by 3D CNNs, temporal information by an LSTM, and actions by fully connected layers.
- The input combines a local observation image with a guidance channel and a sequence of historical inputs.The observation image marks static and dynamic obstacles, while the guidance channel contains the local global-guidance segment.
- The planner uses Double Deep Q-Learning, with target values based on immediate rewards at the goal and bootstrapped values otherwise.
- DDQN parameters are updated by sampling transitions from the replay buffer and minimizing a batch loss.
- Three-dimensional CNN layers extract spatial information from the five-dimensional input tensor before recurrent processing.
- CNN embeddings are reshaped into temporal vectors, processed by an LSTM, and mapped by two fully connected layers to action values.
D. Application to Reactive Multi-Robot Path Planning
The approach extends to multi-robot planning by letting each robot combine its own global guidance with local observations in a fully distributed reactive policy. The same single-agent-trained policy can be used by any number of robots without communication.
- Each robot generates actions from its own global guidance and local observations, treating other robots as independent dynamic obstacles.
- Because robots need no trajectory information, motion intentions, or communication, the policy scales to arbitrary numbers of robots.
- The multi-robot extension is fully distributed and can be trained with a single agent before direct use by other agents.
A. Model Parameters
Experiments use three 100 × 100 map types and specified neural-network and training configurations. The setup includes regular, random, and free environments with differing static and dynamic obstacle arrangements.
- A* generates global guidance, while the default local field of view is Hl = Wl = 15 and the input sequence length is Nt = 4.
- The network repeats its convolutional block Nc = 3 times, uses batch size Nb = 32, and applies ReLU activations.
- The evaluation uses regular, random, and free maps, each with a default size of 100 × 100.
- Regular maps imitate warehouses, random maps vary static and dynamic obstacle density, and free maps contain only dynamic obstacles.
- Training uses one NVIDIA GTX 1080ti GPU, RMSprop, and ε-greedy exploration decreasing from ε = 1 to 0.1 over 200,000 steps.
D. Performance Metrics
The paper evaluates path-planning performance using moving cost, detour percentage, and computing time. These metrics compare actual motion with ideal or shortest paths and measure per-step computational demand.
- Moving Cost measures the ratio of actual moving steps to the Manhattan distance between start and goal cells.Ns denotes the number of steps taken, and the L1 distance represents the ideal movement without obstacles.
- Detour Percentage measures the percentage of additional travel relative to the shortest static-obstacle path calculated by A*.LA*(cstart, cgoal) is the shortest path length considering only static obstacles.
- Computing Time is the average computing time required at each step during testing.
VI. RESULTS
The experiments compare G2RL with re-planning baselines across single-robot settings, ablations, unseen environments, and multi-robot evaluations. Results indicate lower motion cost, consistent performance, and strong generalization across tested conditions.
- Single-Robot Results: G2RL is compared with Global Re-planning and Local Re-planning across three maps and goal distances of 50, 100, and 150.Each group uses 100 randomly selected start-goal pairs, with results reported as means and standard deviations across instances.
- Single-Robot Results: G2RL uses the fewest moving steps in all cases and has the smallest standard deviations among the compared single-robot approaches.
- Reward Comparison: 100% success rates are reported for G2RL, while the naive-reward approach achieves 68%–89% and incurs more detours, waiting steps, and possible deadlocks.
- Ablations: A 15 × 15 field of view and Nt = 4 balance performance and computation cost in the reported ablations.Larger fields of view beyond 13 × 13 and sequence lengths beyond three provide limited additional improvement.
- Multi-Robot Evaluation: Figure 4 reports reached robots over time and flowtime histograms, with failure-case flowtime fixed at time step 100.
- Generalization: The model trained on small maps performs consistently well in an unseen 200 × 200 environment with static and dynamic obstacle densities of 0.15 and 0.05.Testing uses 100 start-goal pairs with Manhattan distance 200.
B. Comparison with Multi-Robot Path Planning Methods
The multi-robot evaluation compares G2RL with distributed, centralized, velocity-based, and reinforcement-learning planners across maps and robot populations. G2RL remains competitive without communication or global trajectory information.
- Baselines: Compared methods include Global Re-planning, HCA*, ECBS, Discrete-ORCA, and PRIMAL.ECBS is used as a suboptimal centralized alternative when optimal CBS becomes intractable for larger robot numbers.
- Experimental Setup: The evaluation uses 40 × 40 regular, random, and free maps with 32, 64, and 128 robots and 100 random configurations per map.Static-obstacle densities are 0.45, 0.15, and 0, respectively.
- Results: G2RL maintains consistent performance across environments, outperforming Global Re-planning and PRIMAL and Discrete-ORCA in most cases.Discrete-ORCA is effective only in the free map because it cannot handle crowded static obstacles.
- Results: G2RL has success rates similar to ECBS and HCA* and higher than Global Re-planning, Discrete-ORCA, and PRIMAL.
- System Properties: Unlike centralized approaches, G2RL is fully distributed and non-communicative, requiring no trajectory information from other robots.
VII. CONCLUSION
The paper concludes that G2RL enables scalable hierarchical path planning with a fixed-sized model and supports distributed multi-robot planning. Experiments validate robustness, scalability, and generalizability across the tested settings.
- G2RL enables end-to-end learning with a fixed-sized model in arbitrarily large environments.
- Its distributed uncoupled multi-robot application scales naturally to an arbitrary number of robots.
- The multi-robot application outperforms existing distributed methods and performs similarly to centralized approaches requiring global dynamic knowledge.
- Future work will extend the approach to cooperative multi-robot path planning.