PaperlayerView exact revision on arXiv

Source-linked AI summary

Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV with Deep Reinforcement Learning

Yong Zeng, Xiaoli Xu, Shi Jin, Rui Zhang

arXiv:2003.07574v1eess.SPcs.ITcs.LG

TL;DR

UAV cellular connectivity lacks guaranteed ubiquitous sky coverage, creating a navigation challenge for missions that must balance completion time and communication outages. The paper uses DRL for coverage-aware trajectory design and introduces SNARM, which learns a radio map from UAV measurements to improve training efficiency. The reported results show effective avoidance of weak coverage and superior SNARM performance over direct RL-based navigation.

  • Problem

    Ubiquitous three-dimensional cellular coverage for UAVs in the sky is not guaranteed, motivating navigation that avoids coverage holes while completing missions.

  • Method

    The paper formulates a weighted mission-time and expected-outage optimization problem, transforms it into an MDP, and solves it with DRL and the SNARM radio-mapping framework.

  • Results

    SNARM effectively learns the radio map, while direct RL and SNARM navigate UAVs around weak coverage; SNARM achieves superior performance over direct RL-based navigation.

  • Takeaways & Limitations

    Coverage-aware DRL navigation can use UAV signal measurements without prior channel-model or propagation-environment knowledge, while SNARM reduces real UAV flights needed for data acquisition.

Abstract

from arXiv · show

Cellular-connected unmanned aerial vehicle (UAV) is a promising technology to unlock the full potential of UAVs in the future. However, how to achieve ubiquitous three-dimensional (3D) communication coverage for the UAVs in the sky is a new challenge. In this paper, we tackle this challenge by a new coverage-aware navigation approach, which exploits the UAV's controllable mobility to design its navigation/trajectory to avoid the cellular BSs' coverage holes while accomplishing their missions. We formulate an UAV trajectory optimization problem to minimize the weighted sum of its mission completion time and expected communication outage duration, and propose a new solution approach based on the technique of deep reinforcement learning (DRL). To further improve the performance, we propose a new framework called simultaneous navigation and radio mapping (SNARM), where the UAV's signal measurement is used not only for training the deep Q network (DQN) directly, but also to create a radio map that is able to predict the outage probabilities at all locations in the area of interest. This thus enables the generation of simulated UAV trajectories and predicting their expected returns, which are then used to further train the DQN via Dyna technique, thus greatly improving the learning efficiency.

I. INTRODUCTION

Cellular networks cannot guarantee ubiquitous sky coverage for UAVs, motivating coverage-aware navigation that uses controllable UAV mobility to avoid weak regions. The paper formulates this navigation problem and addresses it with DRL, including SNARM, which combines signal-based learning with radio mapping to improve learning efficiency.

  • Motivation: Cellular networks planned for ground coverage cannot generally guarantee ubiquitous coverage in the sky, with aerial-ground interference further worsening the issue.BS antennas are typically downtilted toward the ground, while strong line-of-sight channels with non-associated co-channel BSs can intensify interference.
  • Motivation: Because UAV mobility is predictable and controllable, coverage-aware trajectory design can circumvent sky-coverage issues without major cellular-network modifications.This approach exploits UAV mobility as a degree of freedom while missions proceed through cellular-supported airspace.
  • Problem formulation: The paper formulates trajectory optimization as minimizing a weighted sum of mission completion time and expected communication outage duration.The resulting problem is difficult for standard optimization because accurate tractable communication models are unavailable and the optimization is highly non-convex.
  • DRL navigation: The proposed DRL approach transforms the trajectory problem into an equivalent MDP and uses a DNN to approximate the action-value function for selecting UAV flying directions.Unlike table-based Q learning, the DNN handles the continuous state space and its effectively infinite number of state-action pairs.
  • SNARM: SNARM uses UAV signal measurements both to train the DQN and to construct a radio map for simulated trajectories and further Dyna-based training.The resulting framework reduces required UAV flights or learning episodes while achieving comparable performance to direct RL, and avoids requiring prior channel or propagation knowledge.

II. SYSTEM MODEL AND PROBLEM FORMULATION

The paper models cellular-connected UAVs navigating through a bounded 3D airspace toward a common destination while maintaining satisfactory cellular connectivity. It formulates navigation and cell association as minimizing a weighted tradeoff between mission completion time and expected communication outage duration, then motivates reinforcement-learning-based solutions for the resulting difficult optimization.

  • System model: UAVs fly from generally different initial locations to a common final location within a bounded cubic airspace.The model assumes collision and obstacle avoidance is guaranteed, for example through spatial or temporal separation and flight above building heights.
  • System model: The UAV trajectory is designed to reach the final location quickly while maintaining satisfactory communication connectivity with the cellular network.Mission completion time is denoted by T, and the trajectory is q(t) over t ∈ [0, T].
  • Communication model: Outage occurs when the UAV’s received SIR falls below a threshold, with outage probability determined by location and serving-cell association under small-scale fading.The channel includes antenna gain, path loss, shadowing, and small-scale fading, while noise is ignored because the setting is treated as interference-limited.
  • Problem formulation: The optimization minimizes T + µ ¯Tout, balancing mission completion time against expected outage duration through trajectory and cell-association design.Longer missions can provide flexibility to avoid weak coverage regions, creating a general tradeoff between the two objectives.
  • Problem formulation: Directly solving the formulation is difficult because expected outage lacks an easily derived closed form and the resulting problem is highly non-convex.The difficulty reflects sophisticated trajectory dependence in the SIR distribution and the need for accurate, tractable end-to-end channel and environmental models.
  • Solution approach: The proposed coverage-aware navigation uses UAV signal measurements with DRL, first applying a dueling DDQN with multi-step learning and then SNARM to improve learning efficiency and UAV performance.The approach does not require prior knowledge of the channel model; SNARM additionally supports radio mapping and simulated trajectory evaluation.

III. OVERVIEW OF DEEP REINFORCEMENT LEARNING

This section introduces the notation and basic concepts used for reinforcement learning and deep reinforcement learning. It points readers to a classic textbook for a more comprehensive treatment.

  • Overview: The section provides a brief overview of reinforcement learning and deep reinforcement learning, including key notation used later in the paper.It is intended as background rather than a comprehensive tutorial.

A. Basics of Reinforcement Learning

Reinforcement learning models sequential decision-making as an MDP and improves a policy through experience to maximize expected long-term return. Q-learning estimates action values from observed transitions and can converge under sufficient visitation and suitable learning rates.

  • RL agents follow a policy that maps states and actions to action probabilities and seek to maximize expected long-term return.
  • When the MDP is known, dynamic programming can recursively obtain optimal value functions; with incomplete knowledge, temporal-difference learning updates estimates from direct interaction.
  • An MDP is specified by state and action spaces, transition probabilities, and immediate rewards.
  • The action-value function Qπ(s, a) is the expected return from taking action a in state s and then following policy π.
  • Q-learning updates action values from state-action-reward-next-state samples using bootstrapping.
  • Q-learning can converge to the optimal action-value function when every state-action pair is visited sufficiently often and the learning rate is appropriate.

B. Deep Reinforcement Learning

Deep reinforcement learning combines RL with deep neural-network function approximation to handle large state-action spaces. DQN stabilizes training with target networks and experience replay, while related variants address overestimation and improve the network architecture.

  • Table-based RL becomes impractical when the number of discretized states and actions is very large.
  • Function approximation learns a parameterized action-value function, reducing storage demands and generalizing to unexperienced state-action pairs.
  • Artificial neural networks provide nonlinear function approximation, and combining deep ANNs with RL yields deep reinforcement learning.
  • DQN trains the action-value network with bootstrapped targets from state-action-reward-next-state transitions, but the target's dependence on the updated parameters can cause oscillations or divergence.
  • A target network is an infrequently updated copy of the Q network that keeps the training target relatively stable and improves convergence.
  • Experience replay randomly reuses stored transitions with mini-batch updates, reducing update variance by avoiding highly correlated data.
  • DDQN addresses Q-learning overestimation, while dueling DQN changes the network architecture and Rainbow combines major DQN improvements.

IV. PROPOSED ALGORITHMS

The paper presents DRL-based algorithms for solving the coverage-aware UAV navigation problem P1.

  • The proposed algorithms use deep reinforcement learning to solve the coverage-aware UAV navigation problem P1.

A. Coverage-Aware UAV Navigation as an MDP

Coverage-aware UAV navigation is reformulated as a discrete-time MDP whose state is location, action is flying direction, and reward balances travel time against outage probability. Empirical signal measurements estimate outage probabilities, while dueling DDQN and SNARM provide learning solutions.

  • A. Coverage-Aware UAV Navigation as an MDP: The continuous UAV trajectory is discretized into N time steps, with large-scale channel quantities treated as approximately constant within each step.
  • A. Coverage-Aware UAV Navigation as an MDP: Cell association is represented discretely as bn and remains unchanged within each time step under the large-scale channel-gain assumption.
  • A. Coverage-Aware UAV Navigation as an MDP: The empirical outage probability is computed from repeated SIR measurements for each location and cell association, converging to the true outage probability as J increases.
  • A. Coverage-Aware UAV Navigation as an MDP: For sufficiently frequent measurements, with J ≫1, the true outage probability can be evaluated using its empirical estimate.
  • A. Coverage-Aware UAV Navigation as an MDP: The MDP state is the UAV location, actions are unit flying directions, and transitions are deterministic under the UAV motion model.
  • A. Coverage-Aware UAV Navigation as an MDP: The reward R(q) = −1−µ P̂out(q) penalizes every time step and adds a weighted outage penalty, encouraging rapid arrival while avoiding weak coverage.
  • A. Coverage-Aware UAV Navigation as an MDP: The destination is the terminal state, and the navigation objective corresponds to undiscounted accumulated rewards over an episode.
  • A. Coverage-Aware UAV Navigation as an MDP: The solution first applies dueling DDQN with multi-step learning, then introduces SNARM to improve learning efficiency using real and simulated flight experience.

B. Dueling DDQN Multi-Step Learning for UAV Navigation

The paper applies dueling DDQN with multi-step learning to continuous-state, discretized-action UAV navigation, while retaining the continuous state space. Distance-based initialization encourages shortest-path flight before learned experience supports coverage avoidance.

  • Action-space design: Naively discretizing both state and action spaces can cause poor performance or an excessively large number of state-action pairs.The proposed design discretizes only the action space and uses an ANN to approximate the action-value function over continuous states.
  • Action-space design: The method keeps the continuous UAV state space but uniformly discretizes flying directions into K discrete actions for DQN-based control.The action set is represented as ˆA = {v⃗(1), ···, v⃗(K)}.
  • Dueling DQN: A dueling DQN separately estimates state value and action advantages, then aggregates them into action-value estimates.The network receives the UAV location and outputs one value for each discretized action.
  • Multi-step learning: Multi-step DDQN training combines truncated multi-step returns with separate networks for action selection and bootstrap evaluation.Experience replay, a target network, a sliding window, and double-Q updates are incorporated into the training procedure.
  • Initialization: Distance-based initialization makes the initial policy favor shortest-path actions while preserving continuous states and avoiding environment interaction during initialization.As experience accumulates, the policy balances flying distance against weak-coverage avoidance.
  • Learned navigation: After learning, the ϵ-greedy policy generates paths that balance minimizing flight distance and avoiding weak cellular-coverage regions.The first episode selects the shortest-path action with probability 1 − ϵ, apart from random exploration.

C. Simultaneous Navigation and Radio Mapping

SNARM jointly learns a radio map from UAV signal measurements and uses it to simulate trajectories for additional Dyna-based reinforcement-learning updates. This combines real interaction with model-generated experience while accounting for initially inaccurate maps.

  • Radio-map learning: SNARM treats radio-map learning as estimating outage probabilities throughout the airspace from incrementally collected UAV measurements.A feedforward ANN represents the map, and replay memory addresses incremental, temporally correlated measurements.
  • SNARM framework: Each measured outage probability supports both immediate navigation and prediction of outage probabilities at locations the UAV has not visited.This dual use of measurements defines simultaneous navigation and radio mapping.
  • Caveat: The radio map may be inaccurate initially, so relying too heavily on simulated experience can degrade performance through model error.The framework increases simulated updates as accumulated real experience makes the map more accurate.
  • Algorithm flow: SNARM uses real measurements to update the map and then predicts simulated next-state outage probabilities and rewards for further learning.The simulated reward is computed from the map’s predicted outage probability.
  • Simulated experience: The radio map provides estimated returns for candidate trajectories, enabling simulated experience without physically flying those trajectories.SNARM can generate as much simulated experience as desired and combine it with real experience.
  • Dyna integration: Dyna integrates real-experience updates with multiple simulated-experience updates based on the learned radio map.Algorithm 2 adds radio-map learning and double-DQN updates from simulated UAV experience to the direct-RL procedure.

V. NUMERICAL RESULTS

Numerical experiments evaluate coverage-aware navigation and radio-map learning in a modeled urban cellular environment. The learned map closely matches the true map, and SNARM improves returns and reduces required real flight episodes relative to direct RL.

  • Simulation setup: The simulated environment is a 2 km × 2 km urban area with 7 cellular BS sites and 21 sectors or cells.Each BS site contains 3 sectors, and the BS antenna height is 25 m.
  • Coverage environment: 3D antenna radiation and building blockage produce an irregular sky-coverage pattern with central regions below 30% coverage probability.These weak-coverage regions motivate detoured coverage-aware UAV paths.
  • Radio-map accuracy: The learned coverage map is almost identical to the true map, with only slight differences.The comparison validates SNARM’s radio-map estimation for coverage-aware path learning.
  • Radio-map accuracy: 87.1% of the MSE reduction achieved after 5000 episodes is reached after only 200 episodes.Map quality improves quickly as signal measurements accumulate.
  • Navigation performance: After 5000 episodes, average return reaches −1089 for SNARM and −1486 for direct RL.Both methods improve overall despite fluctuation, while SNARM ultimately achieves the higher return.
  • Learning efficiency: About 3200 actual UAV flight episodes are required for direct RL to reach −1500 return, versus about 1000 with SNARM.The reduction reflects SNARM’s use of learned-map simulated experience for path planning.
  • Coverage-aware paths: Both RL methods generally detour around weak coverage and produce sparser paths in areas with low coverage probability.SNARM discovers and follows a narrow radio bridge near x = 1000 m and y = 1000–1700 m.

VI. CONCLUSIONS

The paper addresses coverage-aware navigation for cellular-connected UAVs using DRL based only on UAV signal measurements. It introduces direct RL and SNARM, with numerical results showing effective navigation and superior SNARM performance over direct RL.

  • Problem: The paper studies coverage-aware navigation for cellular-connected UAVs to avoid cellular coverage holes during mission execution.The approach targets limitations of conventional optimization-based path design.
  • Approach: The proposed DRL algorithms use UAV signal measurements as input rather than requiring prior environmental knowledge.This is presented as a response to limitations of conventional optimization-based approaches.
  • Approach: A direct RL algorithm based on dueling DDQN with multi-step learning is followed by SNARM for radio mapping and reduced real UAV flights.SNARM uses model learning to support additional training experience.
  • Findings: Numerical results show effective coverage-aware navigation and superior SNARM performance compared with direct RL-based navigation.The conclusion reports this comparison at the paper level without specifying a single metric.
Loading 2003.07574v1…