PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on DRL based UAV Communications and Networking: DRL Fundamentals, Applications and Implementations

Wei Zhao, Shaoxin Cui, Wen Qiu, Zhiqiang He, Zhi Liu, Xiao Zheng, Bomin Mao, Nei Kato

arXiv:2502.12875v1cs.NI

TL;DR

UAV network optimization is difficult under dynamic and uncertain conditions, where conventional methods have limitations. This survey connects DRL fundamentals to UAV optimization models and implementations; reviewed cases show that objective alignment varies.

Abstract

from arXiv · show

Unmanned aerial vehicles (UAVs) are playing an increasingly pivotal role in modern communication networks,offering flexibility and enhanced coverage for a variety of applica-tions. However, UAV networks pose significant challenges due to their dynamic and distributed nature, particularly when dealing with tasks such as power allocation, channel assignment, caching,and task offloading. Traditional optimization techniques often struggle to handle the complexity and unpredictability of these environments, leading to suboptimal performance. This survey provides a comprehensive examination of how deep reinforcement learning (DRL) can be applied to solve these mathematical optimization problems in UAV communications and networking.Rather than simply introducing DRL methods, the focus is on demonstrating how these methods can be utilized to solve complex mathematical models of the underlying problems. We begin by reviewing the fundamental concepts of DRL, including value-based, policy-based, and actor-critic approaches. Then,we illustrate how DRL algorithms are applied to specific UAV network tasks by discussing from problem formulations to DRL implementation. By framing UAV communication challenges as optimization problems, this survey emphasizes the practical value of DRL in dynamic and uncertain environments. We also explore the strengths of DRL in handling large-scale network scenarios and the ability to continuously adapt to changes in the environment. In addition, future research directions are outlined, highlighting the potential for DRL to further enhance UAV communications and expand its applicability to more complex,multi-agent settings.

I. INTRODUCTION

UAVs can extend communication services to remote areas, but their dynamic, resource-constrained networks make optimization difficult. The survey connects DRL fundamentals with implementations for UAV optimization problems and outlines future directions.

A. Background and Motivation

  • UAVs can extend communication services to remote areas, but mobility, resource limits, and real-time demands complicate optimization, especially in dense and heterogeneous 5/6G networks.Source: A. Background and Motivation · p. 1

B. DRL Emergence in UAV Systems

  • Traditional optimization struggles with UAV mobility, growing network scale, real-time decisions, uncertainty, and decentralized cooperation, motivating DRL for sequential network control.Source: B. DRL Emergence in UAV Systems · p. 2

C. Contributions and Organization of the Survey

D. Organization of the Survey

1) Overview:

  • The overview frames sequential decisions through MDPs and RL concepts, while distinguishing policy-based optimization from actor-critic methods’ systematic optimization and parallel sampling.Source: 1) Overview · p. 3, Figure 2 · p. 4

2) Markov Decision Process:

The section frames reinforcement learning as maximizing expected discounted reward in an MDP, then explains value evaluation and greedy policy improvement. Policy iteration alternates evaluation and improvement until the optimal policy is reached.

2) Markov Decision Process:

  • An MDP is a tuple (S, A, P, R, γ), and the agent seeks a policy maximizing expected cumulative discounted reward over its interaction trajectory.The state and action spaces, transition function, reward function, and discount factor characterize the decision problem.Source: 2) Markov Decision Process · p. 4

3) Policy Evaluation and Bellman Equations:

  • For a fixed policy, Vπ measures expected discounted returns from a state, while Qπ measures returns after taking a specified action; Bellman equations link them recursively.For finite state and action spaces, repeated updates can estimate these values until convergence.Source: 3) Policy Evaluation and Bellman… · p. 5

4) Bellman Optimality Equations:

  • Optimal value functions give the maximum expected returns, from which a greedy policy can be derived, but this requires knowing the optimal action-value function.Tabular MDPs can use policy iteration, which alternates evaluation and improvement, or value iteration, which repeatedly applies Bellman optimality updates.Source: 4) Bellman Optimality Equations · p. 5

5) Policy Iteration:

  • Policy iteration alternates policy evaluation with greedy improvement, and the policy improves monotonically across states until the optimal policy is reached.Each new policy is formed by selecting actions according to the current action-value estimates.Source: 5) Policy Iteration · pp. 5–6

6) Value Iteration:

Value iteration interleaves policy evaluation and improvement, refining value estimates and selecting actions greedily. Value-based DRL uses parameterized function approximators to handle infinite state spaces, with training techniques intended to improve stability.

6) Value Iteration:

  • Value iteration combines policy evaluation and improvement instead of waiting for evaluation to converge before each improvement, enhancing computational efficiency and facilitating faster convergence.Value estimates are refined using temporal-difference errors from observed trajectories, while greedy action selection improves the policy.Source: 6) Value Iteration · p. 6
  • SARSA’s update uses sampled state-action transitions because uncertain transition probabilities make next-state value estimation challenging in real systems.The update is based on discrete samples of states, actions, rewards, and subsequent states and actions.Source: 6) Value Iteration · p. 6
  • SARSA, Q-learning, Monte Carlo, and n-step methods differ in how they form target Q-values from immediate and future rewards.SARSA uses the current policy’s next action, whereas Q-learning uses the maximum next-action value.Source: 6) Value Iteration · p. 6

B. Value-based DRL

  • Neural function approximators generalize state-action value estimates across infinite state spaces, where tabular storage is impractical and discretizing continuous actions can reduce precision.The parameters are trained to minimize mean squared error between approximate and true state-action values.Source: B. Value-based DRL · p. 6
  • Stable value-based DRL training uses experience replay, target networks, and learning-rate scheduling to address correlated samples and unstable updates.The target in the approximate update is used instead of the expected cumulative reward, which is complex to compute.Source: B. Value-based DRL · p. 6

1) Deep Q Learning:

The section reviews value-based and policy-based DRL methods, describing how their architectures and update strategies address action-space, estimation, and training challenges. It also surveys extensions and variance-reduction approaches.

1) Deep Q Learning:

  • DQN uses neural networks for value estimation, experience replay to stabilize training and improve sample efficiency, and a periodically updated target network to reduce update instability.Replay stores experience tuples, while the current-value network updates continuously and the target network updates periodically.Source: 1) Deep Q Learning · pp. 6–7

2) Double DQN:

  • DDQN decouples action selection from value evaluation to address DQN’s overestimation of action values.In DQN, using the same target-network parameters for both selection and evaluation can produce overly optimistic estimates; DDQN’s decoupling is reported to improve stability and performance.Source: 2) Double DQN · p. 7

3) Dueling DQN:

  • Dueling DQN separates state-value and action-advantage estimation, which can be useful when relative action importance matters more than estimating every action value separately.Its architecture uses separate streams for V(s) and A(s,a).Source: 3) Dueling DQN · p. 7

4) Others:

  • Other value-based extensions address partial observability, large action spaces, risk sensitivity, estimation variance, sample efficiency, and distributed training.Examples include LSTM-based state histories, Deep Monte-Carlo, prioritized replay, distributional methods, Noisy DQN, and Rainbow’s integration of several improvements.Source: 4) Others · p. 7

1) Overview:

  • Policy-based methods directly optimize policy parameters and can handle continuous action spaces that value-based methods such as DQN cannot directly manage.Their gradient is complicated by dependence on both sampled actions and the policy-induced stationary state distribution, which is difficult to estimate when dynamics are unknown.Source: 1) Overview · pp. 7–8

2) Vanilla Policy Gradient:

  • Vanilla policy gradients use value, return, baseline, or advantage terms to weight policy updates, while subtracting a baseline can reduce the high variance associated with empirical trajectory returns.The derivation expresses the objective through Qπ and applies the log-derivative trick; the vanilla estimator remains unbiased despite its high variance.Source: 2) Vanilla Policy Gradient · p. 8

3) Trust Region Policy Optimization:

TRPO constrains policy updates to a trust region so its surrogate objective remains an accurate approximation of the true cost. The method uses importance sampling and a bound on performance difference to support updates while limiting policy change.

3) Trust Region Policy Optimization:

  • TRPO uses an approximate surrogate objective because evaluating candidate policies under their own visitation distributions would require extensive sampling.The surrogate supports policy updates when the new and old discounted visitation frequencies remain sufficiently close.Source: 3) Trust Region Policy Optimization · pp. 8–9
  • The performance difference between the expected reward and TRPO’s surrogate objective is bounded by 4ξ²γε/(1−γ)².Here ξ bounds policy divergence and ε is the maximum absolute advantage.Source: 3) Trust Region Policy Optimization · p. 9
  • TRPO limits each policy update with a KL-divergence trust region, using importance sampling to account for differences between training-data and policy state distributions.The constraint controls the maximum policy change per step.Source: 3) Trust Region Policy Optimization · pp. 8–9

4) Proximal Policy Optimization:

This section surveys actor-critic methods and variants, from PPO and deterministic policy gradients to DDPG, emphasizing their update mechanisms, efficiency, and exploration trade-offs.

4) Proximal Policy Optimization:

  • PPO clips the current-to-behavior policy probability ratio to [1 − δ, 1 + δ] in its surrogate objective, enabling multiple mini-batch update epochs.The survey reports improved sample efficiency over traditional policy-gradient methods and better sample complexity while retaining simpler implementation than TRPO.Source: 4) Proximal Policy Optimization · p. 9

5) Others:

  • ACER uses truncated importance sampling, a stochastic dueling network, and a trust-region method, while ACKTR uses Kronecker-factored curvature and is reported as more sample efficient.ACER's enhancements are described as improving stability and sample efficiency.Source: 5) Others · p. 9

1) Overview:

  • Actor-critic methods pair an action-selecting actor with a value-estimating critic, balancing policy-gradient variance against value-approximation bias.They can learn from complete episode returns and TD errors and incorporate techniques such as trust regions, target networks, and GAE.Source: 1) Overview · pp. 9–10

2) Actor Critic:

  • The actor-critic algorithm uses a neural network to approximate state values, reducing policy-gradient variance to improve policy updates.The actor and critic parameters are updated separately using the action-state value and its gradient.Source: 2) Actor Critic · p. 10

3) Deterministic policy gradient:

  • Deterministic policy gradients map states directly to actions, avoiding integration over actions and improving sample efficiency for large or continuous action spaces.Deterministic policies can limit exploration, particularly in low-noise environments; action noise or a stochastic behavior policy can mitigate this.Source: 3) Deterministic policy gradient · p. 10

4) Deep Deterministic Policy Gradient:

  • DDPG extends deterministic policy gradients to continuous actions with deep actor and critic networks, experience replay, target networks, and added exploration noise.Soft target updates and replay are described as stabilizing training by making gradual parameter adjustments and breaking temporal correlations in samples.Source: 4) Deep Deterministic Policy Gradient · p. 10

5) Twin Delayed Deep Deterministic:

TD3 and SAC are off-policy actor-critic variants that address challenges in continuous-action learning through value-estimation and training-stability mechanisms, with SAC also balancing reward and exploration.

5) Twin Delayed Deep Deterministic:

  • TD3 reduces overestimation bias with twin critics and clipped double Q-learning, and stabilizes training by updating the policy less frequently than the Q-function.It also adds clipped action noise and averages over mini-batches to address overfitting in deterministic policies.Source: 5) Twin Delayed Deep Deterministic · pp. 10–11

6) Soft Actor-Critic :

  • SAC maximizes expected return and policy entropy, using an adaptive temperature to balance reward optimization with exploration.The survey notes that an improper temperature can impair performance and that the target entropy varies by task and learning stage.Source: 6) Soft Actor-Critic · pp. 11–12

7) Others:

The section surveys actor-critic architectures and hybrid methods aimed at scalability, data efficiency, and improved use of off-policy experience.

7) Others:

  • IMPALA scales learning across thousands of machines using V-trace off-policy correction, while GA3C and PAAC use parallel CPU/GPU approaches and UNREAL adds auxiliary tasks for data efficiency.GA3C uses hybrid CPU/GPU approaches; PAAC uses single-machine parallelization.Source: 7) Others · p. 12
  • PGQ combines policy gradients with Q-learning, while the reactor architecture uses multi-step returns and retrace(λ) combines importance sampling with tree-backup methods for low variance and efficiency.The reactor framework further uses a β-leave-one-out policy gradient for practical sample efficiency.Source: 7) Others · p. 12

E. Multi-agent DRL

The section introduces multi-agent reinforcement learning frameworks and approaches, covering coordination settings and the non-stationarity challenge faced by independent learners.

1) Overview:

  • Multi-agent systems are presented as useful for complex tasks where cooperation, flexibility, adaptability, and redundancy can support safety, and MARL methods address coordination challenges beyond single-agent RL.The survey names multi-actor critic, multi-critic multi-DQN, MADDPG, and MAPPO among the approaches.Source: 1) Overview · p. 12

2) Multi-Actor Critic:

  • Markov games model interacting agents in fully observable settings, while Dec-POMDPs use a joint reward and POMGs use individual rewards under partial observability.Centralized learning faces exponential growth in observation and action spaces; decentralized learning reduces action-space size but may be unstable, while CTDE bridges the approaches.Source: 2) Multi-Actor Critic · pp. 12–13

3) Multi-Critic Multi-DQN:

  • Independent Q-learning lets each agent learn from its own observations and actions, but evolving policies of other agents make its environment non-stationary and can render replay data outdated or misleading.Multi-DQN gives each agent its own DQN; proposed responses include conditioning Q on joint actions or strategies and using off-environment importance sampling to stabilize replay.Source: 3) Multi-Critic Multi-DQN · p. 13

4) Multi-Actor-Critic MADDPG:

MADDPG addresses multi-agent learning challenges with a centralized critic and decentralized actors, while estimating other agents’ policies and using policy ensembles.

4) Multi-Actor-Critic MADDPG:

  • MADDPG uses neural network approximation and a centralized critic for each agent to address Q-function dimensionality and learn continuous policies in cooperative or competitive settings.Its centralized critic and decentralized actors extend DDPG, and agents can learn from estimated policies of others.Source: 4) Multi-Actor-Critic MADDPG · p. 14
  • MADDPG’s centralized action-value function takes all agents’ actions and state as input, while each agent’s critic is learned separately and can support distinct reward structures.The replay buffer stores joint interaction tuples, including state, next state, all agents’ actions, and their rewards.Source: 4) Multi-Actor-Critic MADDPG · p. 14
  • Agents can estimate other agents’ policies by optimizing action log-likelihood with entropy regularization, then use estimated values in place of target values.The estimated policies are intended to avoid requiring prior knowledge of other agents’ policies.Source: 4) Multi-Actor-Critic MADDPG · p. 14
  • MADDPG trains K policies per agent, randomly selects one for rollouts, and uses all K in gradient updates to reduce variance.The method is presented as addressing high variance in competitive or cooperative environments.Source: 4) Multi-Actor-Critic MADDPG · p. 14

5) Multi-Actor-Critic MAPPO:

The section describes MAPPO’s actor-critic training and distinguishes one-hop from multi-hop power-allocation problems.

5) Multi-Actor-Critic MAPPO:

  • MAPPO uses parameter sharing and collective experience while preserving behavioral diversity through unique observations; the section contrasts it with IPPO, which can face scalability issues and underperform QMix.The discussion recommends larger gradient-estimation batches, often split into no more than two mini-batches.Source: 5) Multi-Actor-Critic MAPPO · p. 14
  • MAPPO’s actor maps agent observations to action distributions, and its objective uses a probability ratio, GAE-based advantages, and entropy regularization.The critic is optimized by minimizing a separate loss function.Source: 5) Multi-Actor-Critic MAPPO · p. 14
  • Clipping policy and value functions is described as reducing non-stationarity from policy updates, while value normalization stabilizes value learning.The critic’s discounted reward-to-go is denoted by ˆRi, and n is the number of agents.Source: 5) Multi-Actor-Critic MAPPO · p. 14

III. VALUE-BASED DRL FOR POWER ALLOCATION OPTIMIZATION

  • Power-allocation applications are grouped into one-hop and multi-hop settings, with multi-hop problems requiring coordination across UAVs or relay stations and accounting for intermediate-node energy and interference.The survey notes that its summary of power-allocation formulations and DRL implementations appears in Table II.Source: §III · pp. 14–15, Table II · p. 15

A. Value-based DRL in One-hop Scenarios

The survey reviews value-based DRL applications in one-hop and multi-hop UAV networks, assessing whether their reward objectives match the original formulations and how constraints are implemented. Several reviewed approaches align with their formulations, while others have objective mismatches or unspecified constraint handling.

A. Value-based DRL in One-hop Scenarios

  • [94] optimizes subchannels and power for coverage, but its cumulative-reward objective does not fully match the formulation’s spectral-efficiency and rate-variance objectives.The survey suggests adding reward penalties for reliability and average-delay constraints.Source: A. Value-based DRL in One-hop Scenarios · pp. 15–16
  • For one-hop constraints, the survey proposes reward penalties, bounded or discretized actions, and output-layer choices such as sigmoid, tanh, or softmax.These suggestions address constraint implementations not detailed in the reviewed studies.Source: A. Value-based DRL in One-hop Scenarios · pp. 15–16
  • [95],, and are reported to align their DQN or DDQN reward objectives with their formulations for data transmission, throughput, and energy efficiency.[95] leaves constraint implementation unspecified; uses a rate-violation penalty, while adds a rate penalty and bounds discretized power actions.Source: A. Value-based DRL in One-hop Scenarios · p. 16

B. Value-based DRL in Multi-hop Scenarios

  • [98] uses spectral efficiency as the DQN reward, and the resulting solution aligns with the formulation’s average-spectral-efficiency objective.The example uses UAV relays to restore links after infrastructure damage.Source: B. Value-based DRL in Multi-hop… · pp. 16–17
  • [99] jointly optimizes UAV trajectories, power, and RIS phase shifts, but its per-slot throughput formulation differs from the DRL algorithm’s cumulative-reward objective.The survey notes penalties for some constraints and an action-mask approach for collision and power constraints.Source: B. Value-based DRL in Multi-hop… · p. 17
  • [100] jointly selects video quality and power for secure, energy-efficient transmission, with a potential mismatch between per-slot formulation objectives and cumulative rewards.The survey proposes penalties for secrecy-timeout violations and bounded action coefficients for power constraints.Source: B. Value-based DRL in Multi-hop… · p. 17
  • For multi-hop constraints, the survey proposes reward penalties, softmax-based action selection, bounded action coefficients, and action masks to restrict choices to feasible actions.These recommendations address constraints that reviewed articles do not explicitly implement.Source: B. Value-based DRL in Multi-hop… · pp. 17–18
  • [101] applies DRL to cognitive-radio UAV traffic forwarding, using energy efficiency as the reward to align with its multi-slot energy-efficiency objective.The survey proposes softmax and one-hot encoding for service-area selection, plus penalties and bounded coefficients for other constraints.Source: B. Value-based DRL in Multi-hop… · pp. 17–18

IV. POLICY-BASED DRL FOR CHANNEL ASSIGNMENT OPTIMIZATION

The section surveys policy-based DRL for channel assignment across one-hop and multi-hop UAV networks, outlining applications, objectives, and constraint-handling choices. It then introduces caching research grouped by whether UAVs coordinate.

A. Policy-based DRL in One-hop Scenarios

  • The survey groups one-hop channel-assignment work around energy efficiency, fairness, and interference, and around forwarding and task offloading,,,,.It describes DRL as dynamically allocating channels, reducing power, balancing fairness, and mitigating interference,,.Source: A. Policy-based DRL in One-hop Scenarios · pp. 18–19
  • In one-hop networks, DDPG jointly optimizes UAV trajectory and channel assignment while balancing fairness and energy efficiency; penalty rewards encode operational constraints.The surveyed applications also include task offloading and trajectory design with channel availability,.Source: A. Policy-based DRL in One-hop Scenarios · pp. 19–20
  • For, DDPG uses a fairness indicator to optimize throughput and fair resource distribution, but user locations and channel parameters are not provided.The objective covers throughput and fairness across consecutive time slots.Source: A. Policy-based DRL in One-hop Scenarios · p. 19
  • For, DDPG jointly optimizes channel assignment, power control, and user association for cumulative throughput under service-continuity and QoS requirements.The reviewed work assumes clustered users and modeled cluster movement and UAV trajectories determined before channel assignment.Source: A. Policy-based DRL in One-hop Scenarios · p. 19
  • One-hop constraint implementations include penalties for power or SINR violations and action processing such as masking, clipping, normalization, or bounded activations.The survey also suggests action discretization for feasible task association, bandwidth projection, and incorporating energy efficiency in the reward [p16].Source: A. Policy-based DRL in One-hop Scenarios · p. 20

B. Policy-based DRL in Multi-hop Scenarios

  • In multi-hop networks, optimizes energy efficiency through UAV trajectories and channel allocation, using penalties and network adjustments for collision constraints.The survey notes that minimum IoT QoS is not fully implemented in the reviewed work and proposes a dual-component reward to encourage threshold compliance.Source: B. Policy-based DRL in Multi-hop… · p. 20
  • For RIS-mounted UAV anti-jamming, uses PPO to jointly optimize channel assignment for long-term energy efficiency, but omits UAV mobility and combines constraints in one penalty.The survey notes that constraint impacts on energy efficiency vary and that moving closer to receivers can help mitigate jamming.Source: B. Policy-based DRL in Multi-hop… · pp. 20–21

V. ACTOR-CRITIC BASED DRL FOR CACHING OPTIMIZATION

  • The section also describes multi-hop AoI minimization with PPO and classifies caching studies by whether UAVs optimize independently or cooperate,,.The caching overview distinguishes independent caching decisions from collaboration in cached-content sharing, trajectory adjustment, and resource management.Source: B. Policy-based DRL in Multi-hop… · p. 21, §V · p. 21

A. Caching without UAV Interaction

The caching section reviews independent and collaborative UAV caching approaches, then describes multi-agent DRL designs for UAV-assisted networks. Across the examples, objectives include reducing delay and improving content delivery, while the discussion flags assumptions and implementation gaps.

A. Caching without UAV Interaction

  • Without UAV interaction, surveyed caching studies use TD3, and PPO,, for caching and related resource decisions.Examples include maximizing cache hit ratio while minimizing task-processing delay, minimizing content retrieval delay, and joint caching, trajectory, and power decisions.Source: A. Caching without UAV Interaction · pp. 21–23
  • The multi-UAV architecture in uses task popularity and a greedy algorithm; UAV-cached data serves users while the BS handles uncached data, with per-slot system delay as the objective.The passage says the PPO solution differs from that objective in each time slot.Source: A. Caching without UAV Interaction · p. 22
  • In, dual-clip PPO models the NP-hard average-download-delay problem as an MDP and adapts caching, trajectory, and transmission power; combines LSTM with PPO for video-adaptive caching,.The survey describes as improving performance through faster content delivery and reduced video-streaming latency.Source: A. Caching without UAV Interaction · p. 23

B. Caching with UAV Interaction

  • With UAV interaction, uses PPO for collaborative caching and trajectory planning, while uses DDPG for caching and offloading in a clustered-NOMA edge network,.The reviewed architecture includes local-UAV, neighboring-UAV, and cloud retrieval paths; uses penalties for task-offloading, program-caching, and distance constraints.Source: B. Caching with UAV Interaction · p. 23
  • The framework assumes static UAV positions; instead considers real-time UAV repositioning with shared cached content,.The survey says aims to reduce transmission delay, but its per-slot delay objective may differ from the DRL solution that maximizes cumulative reward.Source: B. Caching with UAV Interaction · pp. 23–24
  • For, the survey notes unspecified constraint implementation and a possible mismatch between per-slot delay minimization and maximizing cumulative reward across time slots.Suggested constraint mechanisms include discrete actions, reward penalties, boundary filters, and distance-based collision penalties.Source: B. Caching with UAV Interaction · p. 24

VI. MULTI-AGENT DRL FOR UAV-ASSISTED NETWORKS

  • Multi-agent DRL approaches model UAVs as homogeneous agents, heterogeneous agents, or components of hybrid methods that combine classical optimization with MADRL [VI].Heterogeneous models assign distinct characteristics, objectives, and observations to entities such as UAVs and users; hybrid methods use classical optimization for specific constraints or static scenarios and MADRL for dynamic multi-agent interactions.Source: §VI · p. 24

A. MADRL-based Solutions

The survey reviews multi-agent DRL approaches for UAV networking, hybrid methods combining optimization and MADRL, and open challenges in scaling and adapting DRL. It concludes that modeling problems as mathematical optimizations and progressing to implementation offers a practical guide to applying DRL in UAV communications.

A. MADRL-based Solutions

  • MADRL studies – vary in agent design and action spaces, from CommNet positioning and resource allocation to continuous actions and heterogeneous UAV-user agents.[118] uses discrete movement actions and does not completely prevent collisions; uses attention, while applies b-MAPPO to mixed binary and continuous decisions.Source: A. MADRL-based Solutions · pp. 24–25

B. Hybrid MADRL Solutions: Combining Traditional Optimization

  • Hybrid approaches combine MADRL with other optimization methods: for caching, for offloading and trajectory control, and [124] for URLLC-aware joint optimization.[8] combines DM-ACO with MADRL; uses mixed-integer nonlinear programming for offloading decisions, while [124] decomposes the problem using Lyapunov optimization and pairs UA-MADDPG with greedy resource allocation. The reward functions in are inconsistent with its mathematical objective.Source: B. Hybrid MADRL Solutions: Combining… · pp. 25–26

A. High Dimensionality and Computational Complexity

  • Large, dynamic UAV networks create high-dimensional state and action spaces, making DRL training computationally demanding and real-time use difficult.Function approximation, transfer learning, and model compression are proposed to reduce overhead.Source: A. High Dimensionality and… · p. 26

B. Scalability and Multi-agent Coordination

  • As the number of agents grows, interactions and action spaces make effective policy learning more complex, motivating decentralized or hierarchical DRL for scalability.The coordination tasks listed include power and resource allocation, channel assignment, and task offloading.Source: B. Scalability and Multi-agent… · p. 26

C. Modeling Realistic Environments and Uncertainty

  • Mobility, interference, and weather make UAV networks non-stationary, while conventional DRL often assumes stationary environments; robust adaptation remains an open challenge.Meta-RL and robust DRL are identified as possible approaches requiring further research and validation.Source: C. Modeling Realistic Environments and… · p. 26

D. Reward Function Design and Convergence

  • Reward design must balance competing objectives such as latency, throughput, and energy efficiency; poor rewards can produce suboptimal policies or training divergence.The survey calls for better multi-objective optimization and reward-shaping methods.Source: D. Reward Function Design and… · p. 26

E. Data Efficiency and Public Datasets

  • DRL data needs, costly real-world collection, and imperfect simulations limit data efficiency; few public UAV-specific datasets also hinder benchmarking and comparison.The survey identifies model-based DRL, transfer learning, and shared datasets as possible responses.Source: E. Data Efficiency and Public Datasets · p. 27

VIII. CONCLUSION

  • The survey frames UAV communication tasks as mathematical optimization models and outlines DRL applications from system modeling through implementation.It identifies scalability and real-time adaptability as future challenges and opportunities.Source: §VIII · p. 27

Related papers

Loading 2502.12875v1…