PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review

Fadi AlMahamid, Katarina Grolinger

arXiv:2208.12328v1cs.ROcs.AI

TL;DR

RL-based UAV navigation must address varied environments and tasks, and the review identifies a need for navigation-focused benchmarking and comparisons. It synthesizes navigation tasks, frameworks, and RL algorithms, then highlights research challenges and opportunities for future evaluation and development.

  • Problem

    UAV navigation studies face limited benchmarking and comparison across algorithms because environments can be oversimplified and vary across simulators and conditions.

    Source: §8 · p. 17

  • Method

    The review synthesizes UAV navigation tasks and frameworks and classifies more than fifty RL algorithms by their contributions, relationships, and application environments.

    Source: §4 · p. 5, §5.1.2 · p. 9, §9 · p. 19

  • Results

    The review identifies needs for navigation-focused evaluation and benchmarking, more complex environments, knowledge-transfer research, work on UAV complexity, and evaluation of state-of-the-art RL algorithms on navigation tasks.

    Source: §9 · p. 19

  • Takeaways & Limitations

    The review provides a process intended to help select applicable RL algorithms for UAV navigation tasks.

    Source: §9 · p. 19

    The review may omit relevant work because its search was limited to two databases, English-language papers, and UAV-navigation studies from 2016–2021, while algorithm identification could miss methods.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

There is an increasing demand for using Unmanned Aerial Vehicle (UAV), known as drones, in different applications such as packages delivery, traffic monitoring, search and rescue operations, and military combat engagements. In all of these applications, the UAV is used to navigate the environment autonomously - without human interaction, perform specific tasks and avoid obstacles. Autonomous UAV navigation is commonly accomplished using Reinforcement Learning (RL), where agents act as experts in a domain to navigate the environment while avoiding obstacles. Understanding the navigation environment and algorithmic limitations plays an essential role in choosing the appropriate RL algorithm to solve the navigation problem effectively. Consequently, this study first identifies the main UAV navigation tasks and discusses navigation frameworks and simulation software. Next, RL algorithms are classified and discussed based on the environment, algorithm characteristics, abilities, and applications in different UAV navigation problems, which will help the practitioners and researchers select the appropriate RL algorithms for their UAV navigation use cases. Moreover, identified gaps and opportunities will drive UAV navigation research.

1. Introduction

UAVs serve diverse applications that require autonomous navigation, and reinforcement learning is used to train agents to navigate while avoiding obstacles. The paper reviews navigation tasks, frameworks, simulation tools, and RL algorithms to help readers select methods for particular applications and environments.

1. Introduction

  • UAVs are used for package delivery, traffic collisions, military tracking, search and rescue, and other applications.The paper notes increasing interest in these varied uses.Source: §1 · p. 2
  • RL agents process environment states, choose actions that control UAV movement or propeller thrust, and receive rewards.Cameras and sensors capture information used to represent the state.Source: §1 · p. 2
  • The review addresses the gap in surveys of RL applications to UAV navigation, despite existing reviews of visual navigation, flocking, and path planning.It aims to systematically review RL algorithms across autonomous UAV navigation problems.Source: §1 · p. 2
  • The review aims to help researchers select RL algorithms based on application area and environment type, while explaining algorithm characteristics and relationships.It also classifies algorithms according to environment type.Source: §1 · p. 2
  • The paper reviews navigation frameworks by problem domain and identifies techniques and simulation tools used for UAV navigation tasks.It also presents algorithm-selection guidance and research challenges and opportunities.Source: §1 · p. 2

2. Review Process

The review describes its paper-identification process and validity threats. It focuses its UAV-navigation literature search on English-language work from 2016–2021, while drawing on broader sources to identify RL algorithms.

2. Review Process

  • The review process covers inclusion criteria, paper identification, and threats to validity.These elements structure the review methodology.Source: §2 · p. 2

2.1. Inclusion Criteria and Identification of Papers

  • The UAV-navigation search covered English-language papers from 2016–2021 containing terms for reinforcement learning, navigation, and UAVs.The search used terminology variants for UAVs and RL.Source: §2.1 · p. 2
  • The search yielded 159 papers after duplicate and unrelated papers were removed and additional RL algorithm papers were identified through author knowledge and references.The initial query returned 104 papers; 15 duplicates and 5 unrelated papers were removed, and 75 additional algorithm papers were identified.Source: §2.1 · p. 3

2.2. Threats to Validity

  • The authors identify database, language, time-window, and algorithm-reporting biases as threats to validity.The search used IEEE Xplore and Scopus, included only English papers, and limited UAV-navigation studies to 2016–2021; algorithm identification may miss methods.Source: §2.2 · p. 3

3. Reinforcement Learning

The paper introduces RL through state transitions, rewards, policies, and value estimates, then describes exploration, experience replay, and policy types. It explains how deep neural networks extend RL to complex navigation problems.

3. Reinforcement Learning

  • An RL agent interacts with an environment by taking actions that change its state and produce rewards, with learning aimed at maximizing discounted rewards over an episode or an infinite horizon.The discount factor reduces the weight of future rewards and helps the expected sum converge for an infinite horizon.Source: §3 · p. 3

3.1. Policy and Value Function

  • A policy specifies action probabilities by state, while value functions estimate state or action value; the optimal Q-value follows the Bellman optimality relation.An action with the highest Q-value may not lead to better rewards in the future.Source: §3.1 · p. 3

3.2. Exploration vs Exploitation

  • RL must balance exploration with exploitation because repeatedly choosing the highest-reward action can miss better rewards available through future alternatives.The review names ε-greedy, Upper Confidence Bound, and Gradient Bandits as strategies for balancing the two.Source: §3.2 · p. 4

3.3. Experience Replay

  • Experience Replay stores interaction tuples in a buffer and trains the agent on uniformly sampled batches, while Prioritized Experience Replay samples by TD error to improve convergence.Prioritized replay repeatedly trains on experiences with lower TD error.Source: §3.3 · p. 4

3.4. On-Policy vs Off-Policy

  • On-policy algorithms use the same policy to collect samples and determine expected returns, whereas off-policy algorithms collect with a behavior policy and evaluate a target policy.Off-policy methods can reuse past experiences from different episodes and do not require a full episode to generate training samples.Source: §3.4 · p. 4

3.5. Deep Reinforcement Learning

  • Deep RL combines neural networks with RL, choosing network types according to the problem and inputs, such as CNNs for images or RNNs for inputs across time steps.Deep agents take state inputs and output actions; applying deep networks has enabled work on complex tasks such as image-based drone obstacle avoidance.Source: §3.5 · p. 4
  • The neural-network loss, or Temporal Difference, is described as the difference between the network output and the Bellman-derived optimal Q-value.The passage introduces Equation 6 as the generic computation of this difference.Source: §3.5 · p. 4

4. Autonomous UAV Navigation using DRL

The review organizes DRL navigation around UAV control, obstacle avoidance, path planning, and flocking, with task objectives and environment information shaping agent inputs, actions, and rewards. It describes how these tasks use different sensing, navigation representations, and training arrangements.

4. Autonomous UAV Navigation using DRL

  • Obstacle avoidance and path planning are the two navigation objectives on which most reviewed research focuses.The review’s task summary identifies both as prominent objectives among the applications of RL to UAV navigation.Source: §4 · p. 5, Table 1 · p. 6

4.1. UAV Control

  • Multirotor control changes thrust to produce pitch, roll, yaw, and throttle, while fixed-wing UAVs use pitch, roll, and yaw.RL actions may directly change these movement values or indirectly produce commands that the UAV can execute.Source: §4.1 · pp. 5–6

4.2. Obstacle Avoidance

  • Obstacle-avoidance agents use camera images or distance-sensor readings as state inputs, with sensing choices shaping the neural-network architecture.The review describes CNNs for RGB or depth-map images and SNNs for event-based images.Source: §4.2 · pp. 6–7

4.3. Path Planning

  • Path planning seeks a safe route to meet the flight objective, which need not be the shortest or a straight-line route.The review distinguishes global planning toward a destination from local waypoint planning around static and dynamic obstacles.Source: §4.3 · p. 7

4.3.1. Map-Based Navigation

  • Map-based navigation uses 2D or 3D environment representations, whereas mapless navigation learns from scene images and may require substantial computation.Map-based agents can use GPS location and target-relative distance and angle; mapless inputs may combine scene images, destination, and localization, though some visual reactive approaches operate without GPS.Source: §4.3.1 · p. 7, §4.3.2 · pp. 7–8

4.4. Flocking

  • Flocking combines formation maintenance with tasks such as control and obstacle avoidance, using flock-centering, leader-follower, or neighbor topologies.The review describes communication between UAVs as part of learning flocking policies.Source: §4.4 · p. 8

4.4.1. Centralized Training

  • Centralized training shares flocking experience or policies, but the review warns that it may generalize poorly to neighbor flocking when neighbors have different dynamics.The stated limitation concerns learned policies that differ between neighbors because their dynamics differ.Source: §4.4.1 · p. 8

4.4.2. Distributed Training

  • Distributed flocking assigns an RL agent to each UAV and requires communicated flight information in each agent’s state; missing information can isolate a UAV.The reward can be customized to maintain distance and direction relative to other UAVs.Source: §4.4.2 · p. 8

5. UAV Navigation Frameworks and Simulation Software

The review classifies UAV navigation frameworks by objective and describes common framework types alongside simulation software. It surveys approaches for energy-aware routing, path planning, flocking, vision, and transfer learning, and outlines the components used in 3D navigation simulation.

5.1. UAV Navigation Frameworks

  • The review classifies UAV navigation frameworks by objective and distinguishes them from RL itself, including only frameworks that add to sensor- or camera-based navigation.The framework discussion includes path-planning, flocking, and energy-aware categories.Source: §5.1 · p. 8, §5 · p. 8, Table 2 · p. 9

5.1.1. Energy-Aware UAV Navigation Frameworks

  • Energy-aware navigation frameworks combine obstacle avoidance with battery-aware route planning and recharge stopovers.Examples monitor battery thresholds or battery, rotor, and sensor conditions when planning route changes.Source: §5.1.1 · p. 9

5.1.2. Path Planning Frameworks

  • Path-planning frameworks determine routes suited to objectives such as shortest, fastest, or safest travel, using modular, layered, staged, or collaborative designs.Examples include motion primitives for switching between near-hover and swift maneuvers, and UGV-UAV cooperation where battery and altitude constraints complement one another.Source: §5.1.2 · p. 9

5.1.3. Flocking Frameworks

  • Flocking frameworks can extend beyond formation navigation to scheduling UAV coverage of events and predicting drone or obstacle movement for route generation.The review describes spatiotemporal scheduling and prediction of dynamic obstacles as examples.Source: §5.1.3 · p. 9

5.1.4. Vision-Based Frameworks

  • Vision-based frameworks augment camera-driven navigation through sensor fusion, recurrent networks, memory, or image preprocessing.The review lists CNN fusion, LSTM cells, RNNs, and preprocessing as possible additions.Source: §5.1.4 · p. 9

5.1.5. Transfer Learning Frameworks

  • Transfer-learning frameworks reuse neural-network weights from a previous environment and retrain only part of the agent when the environment changes.One example trains virtually, then updates the last few CNN layers after deployment to a real-world environment.Source: §5.1.5 · p. 9

5.2. Simulation Software

  • UAV navigation studies commonly use simulation rather than actual UAVs because hardware and crash-replacement costs are high.The review says simulation comparisons are not its purpose; it describes common tools and the three main components of 3D simulation: an RL agent, a UAV flight simulator, and a 3D graphics engine.Source: §5.2 · pp. 9–10

6. Reinforcement Learning Algorithms Classification

The section classifies RL algorithms by state-space size and action type, extends an existing taxonomy, and examines their UAV-navigation applications. It surveys discrete-action methods from tabular Q-learning and SARSA to deep, distributional, and distributed variants.

6. Reinforcement Learning Algorithms Classification

  • The review extends a three-category state/action taxonomy with subclasses, analyzes more than 50 RL algorithms, and examines their use in UAV navigation.Table 3 classifies algorithms found in UAV-navigation studies alongside other prominent RL algorithms.Source: §6 · p. 10

6.1. Limited States and Discrete Actions

  • For limited states and discrete actions, Q-learning and SARSA maintain a Q-table and iteratively update state-action values to find an optimal policy.SARSA updates using the next action taken, whereas Q-learning computes the next state-action value using the Bellman equation and selects a greedy action.Source: §6.1 · p. 10

6.2. Unlimited States and Discrete Actions

  • For unlimited states and discrete actions, deep agents process state inputs with a neural network and output Q-values for the available actions.DQN commonly uses convolutional, pooling, and fully connected layers, but can overestimate action values and select a non-optimal action.Source: §6.2 · p. 10, §6.2.1 · p. 10

6.2.1. Deep Q-Networks Variations

  • DQN variants address overestimation, improve value estimation, integrate information across time, or combine these techniques.Double DQN uses policy and target networks; Dueling DQN aggregates state-value and advantage functions; DD-DQN combines Double and Dueling DQN, while DRQN uses an LSTM layer to process groups of states.Source: §6.2.1 · pp. 10, 12

6.2.2. Distributional DQN

  • Distributional methods model reward distributions rather than only expected values, with variants differing in how they represent or learn those distributions.C51 uses 51 fixed locations and adjusts their probabilities; QR-DQN adjusts quantile locations, IQN learns a full quantile function, and FQF parameterizes both quantile fractions and values.Source: §6.2.2 · pp. 12–13

6.2.3. Distributed DQN

  • Distributed DQN architectures separate acting from learning, with actors sharing a network and replay buffer and a learner training on prioritized experiences.R2D2 adds an LSTM and trains on sampled replay sequences using burn-in; NGA adds intrinsic rewards to encourage exploration, and Agent57 adds an adaptive exploration–exploitation mechanism to NGA.Source: §6.2.3 · p. 13

6.2.4. Deep SARSA

  • Deep SARSA uses a neural network and DQN-like extensions, but computes the current state-action value using the next action rather than a greedy action.The next action is taken to determine the current state-action value.Source: §6.2.4 · p. 13

6.3. Unlimited States and Continuous Actions

Continuous actions represent movement quantities rather than choices from a preset action list, a distinction the review makes for realistic UAV control. The section describes parameterized policies optimized by gradient methods and introduces policy-gradient algorithms.

6.3. Unlimited States and Continuous Actions

  • Continuous actions specify movement quantities such as roll, pitch, yaw, and throttle, unlike preset directional actions used in simulated environments.The review says discrete actions do not enable realistic movements in real-world scenarios.Source: §6.3 · p. 13
  • For continuous actions, a parameterized policy is optimized to maximize expected discounted rewards because action values cannot feasibly be determined for every action and state.The policy objective is maximized using gradient descent; the gradient and learning rate are denoted ∇ and α.Source: §6.3 · pp. 13–14

6.3.1. Policy-Based Algorithms

  • The policy-gradient family includes REINFORCE, TRPO, PPO, PPG, and SVPG, which apply different methods to improve gradient-descent performance.The policy-gradient theorem is also described as a foundation for actor-critic and multi-agent or distributed actor-critic algorithms.Source: §6.3.1 · p. 14, §6.3 · p. 14

REINFORCE

This section surveys policy-gradient methods and actor-critic variants for continuous-action settings. It describes approaches that adjust policy updates, combine policy and value estimation, or use recurrent networks for partially observed domains.

REINFORCE

  • REINFORCE uses Monte Carlo samples from entire episodes to update policy parameters, and subtracting a baseline reduces variance while maintaining bias.The review notes that Monte Carlo's high variance can slow learning.Source: REINFORCE · p. 14

Trust Region Policy Optimization (TRPO)

  • TRPO constrains policy-gradient steps with a KL-divergence trust region, while PPO applies the constraint as a penalty and clips its objective.PPG extends PPO with a recurring auxiliary phase that distills value-function information into the policy network while maintaining decoupling.Source: Trust Region Policy Optimization (TRPO) · p. 14

6.3.2. Actor-Critic

  • Actor-critic algorithms pair an actor that finds a policy with a critic that estimates its value, including methods such as DPG, DDPG, and TD3.The critic estimates the action-value function using a parameterized vector and a policy-assessment technique such as temporal-difference learning.Source: §6.3.2 · pp. 14–15

Deterministic Policy Gradients (DPG)

  • DPG uses a deterministic policy and averages its target objective over the behavior policy's state distribution; DDPG and TD3 extend DPG for continuous actions.TD3 uses the minimum value from two critics to address value-function overestimation.Source: Deterministic Policy Gradients (DPG) · p. 15

Recurrent Deterministic Policy Gradients (RDPG)

  • RDPG applies DPG with an RNN to partially observed domains, using LSTM cells to preserve information from past observations across time steps.The approach extends recurrent policy-gradient work to partially observed domains.Source: Recurrent Deterministic Policy… · p. 15

Soft Actor-Critic (SAC)

The section describes SAC's reward-and-entropy objective and surveys multi-agent and distributed actor-critic methods. These include parallel or replay-based training, coordinated agents, and architectures that separate action execution from policy learning.

Soft Actor-Critic (SAC)

  • SAC maximizes anticipated reward and policy entropy, with a temperature parameter controlling entropy's importance relative to reward.It uses separate actor and critic networks to estimate a soft Q-function, a state-value function, and a parameterized policy.Source: Soft Actor-Critic (SAC) · p. 15

Advantage Actor-Critic

  • Multi-agent and distributed actor-critic methods deploy agents across nodes, and A3C parallelizes training with workers that asynchronously exchange parameters with a global network.A2C differs from A3C by using a coordinator to synchronize agents before updating policy and value parameters; ACKTR uses K-FAC to optimize actor and critic.Source: §6.3.3 · p. 15, Advantage Actor-Critic · p. 15

Actor-Critic with Experience Replay (ACER)

  • ACER is an off-policy actor-critic method that uses experience replay and a single deep neural network to estimate policy and value functions.It adds a stochastic dueling network, trust-region policy optimization, and truncated importance sampling with bias correction.Source: Actor-Critic with Experience Replay… · pp. 15–16

Multi-Agent Reinforcement Learning (MARL)

  • D4PG adds n-step returns and prioritized replay to distributed DDPG, while MADDPG and MATD3 extend deterministic methods to multiple agents.MATD3 uses centralized training and decentralized execution, with two centralized critics per agent to control overestimation bias.Source: Multi-Agent Reinforcement Learning… · p. 16
  • IMPALA separates action execution from policy learning and corrects for actor–learner policy lag with truncated importance sampling; SEED RL combines IMPALA with R2D2.SEED RL moves inference to the learner while environments run remotely, introducing latency mitigated with a fast gRPC communication layer.Source: Multi-Agent Reinforcement Learning… · p. 16

7. Problem Formulation and Algorithm Selection

The section presents a six-step process for narrowing the RL algorithms suitable for a UAV navigation task. Selection emphasizes environment and objective formulation, and yields a set of candidates because performance can vary with implementation choices.

7. Problem Formulation and Algorithm Selection

  • The six-step process narrows the candidate RL algorithms by asking questions that guide an informed selection.At each step, the selected option limits the choices available at the next step.Source: §7 · p. 16
  • State representation, action type, and policy type constrain algorithm choices according to the navigation environment and RL objective.States may be limited or unlimited; actions may be discrete or continuous, while policies may be on-policy or off-policy.Source: §7 · p. 16
  • On-policy methods are described as faster-converging and stable but sub-optimal, while off-policy methods offer optimal policies and require a good exploration strategy.Prioritized experience replay and importance sampling can improve off-policy convergence for tasks seeking an optimal path.Source: §7 · p. 16
  • Processing needs and agent count further narrow choices; multi-agent learning can accelerate learning but requires more computational resources.The final stage returns a group of applicable algorithms because performance may vary with hyperparameters, reward engineering, and neural-network architecture.Source: §7 · p. 17
  • Section 6’s algorithm descriptions and references support comparing distinctions among the candidates after the selection process narrows the use case.The selection process is presented alongside the broader algorithm overview to help users make an informed choice.Source: §7 · p. 17

8. Challenges and Opportunities

The review identifies gaps in evaluation, environment complexity, knowledge transfer, real-drone training, and algorithm coverage. It calls for navigation-specific benchmarking and research that addresses varied conditions and efficient, generalizable methods.

8. Challenges and Opportunities

  • RL benchmarks based on fully observed Atari games may not reflect UAV navigation, where simulations provide partial observations and 3D image depth matters; navigation-focused evaluation is needed.The review notes that algorithms’ benchmark performance may differ in 3D simulation or the real world.Source: §8 · p. 17
  • Oversimplified environments and a lack of standardized benchmarks impede comparison across algorithms, tools, and settings.Relevant conditions include indoor versus outdoor environments, wind, lighting, and moving objects; complex simulators still need to be combined with new benchmarks.Source: §8 · p. 17
  • Training in one environment does not guarantee similar performance elsewhere, so the review calls for generalized methods beyond transfer learning’s focus on reducing retraining time.Differences can include obstacle types, background texture, lighting, and noise.Source: §8 · p. 17
  • Real-drone training and flight constraints motivate energy-aware navigation research using low-complexity, efficient RL algorithms.The review reports that only 10% of investigated research used real drones for navigation training, and typical flight time is 10 to 30 minutes.Source: §8 · p. 17
  • Recent RL algorithms not yet applied to UAV navigation may offer potential for better generalization, faster training, and complex tasks such as flocking.The review identifies evaluating state-of-the-art algorithms on navigation tasks as an opportunity.Source: §8 · pp. 17, 19

9. Conclusion

The conclusion surveys RL applications in autonomous UAV navigation, including obstacle avoidance, path planning, and flocking, alongside navigation frameworks and simulation software. It reviews more than fifty algorithms and identifies persistent research challenges, including benchmarking, complex environments, transfer, UAV complexity, and algorithm coverage.

9. Conclusion

  • RL algorithms address UAV control with obstacle avoidance, path planning, and flocking, while cameras and sensors such as LiDAR provide environmental state information.The review describes sensor data as supporting object-distance estimates and collision avoidance.Source: §9 · p. 19
  • The review identifies navigation frameworks and simulation software used in the experiments it surveys.It also identifies their use within the reviewed papers.Source: §9 · p. 19
  • The study discusses over fifty RL algorithms, their relationships and contributions, and classifies them by environment and UAV-navigation use.It also covers traits such as multi-threading, distributed processing, and multi-agent learning, and presents a process for finding applicable algorithms.Source: §9 · p. 19
  • Researchers tend to test Q-learning, DQN, Double DQN, DDPG, and PPO, although the review notes promising results from newer algorithms such as Agent57.The authors describe their work as the first systematic review, to their knowledge, focused on many RL algorithms applied to autonomous UAV navigation.Source: §9 · p. 19
  • The review identifies needs for navigation-focused benchmarking, more complex environments, knowledge-transfer research, energy-aware UAV training, and evaluation of state-of-the-art algorithms.These challenges and opportunities were identified through analysis of RL algorithms and their use in UAV navigation.Source: §9 · p. 19

A. Acronyms

The acronym section expands abbreviations for RL algorithms, concepts, and UAV-related terms.

A. Acronyms

  • The list expands actor-critic variants including A2C, A3C, ACER, and ACKTR, and defines AS as Autonomous Systems.It also includes names such as Ape-X DPG and DDPG.Source: A. Acronyms • · p. 19
  • It defines DQN-family names including DQN, Double DQN, and Rainbow DQN, and expands MARL as Multi-Agent Reinforcement Learning.The list includes many other RL methods and terms, including PPO and POMDP.Source: A. Acronyms • · p. 19
  • Further entries include TD3, TRPO, and UAV as Unmanned Aerial Vehicle, alongside sensing, memory, and robotics abbreviations.The list also expands UGV as Unmanned Ground Vehicle.Source: A. Acronyms • · p. 20

Related papers

Loading 2208.12328v1…