PaperlayerView exact revision on arXiv

Source-linked AI summary

Path Planning for Cellular-Connected UAV: A DRL Solution with Quantum-Inspired Experience Replay

Yuanjian Li, A. Hamid Aghvami, Daoyi Dong

arXiv:2108.13184v2eess.SPcs.SYeess.SY

TL;DR

The paper formulates cellular-connected UAV path planning as minimizing weighted flight time and expected outage duration, a challenge when the wireless environment is unknown. It maps navigation to an MDP and proposes DRL with quantum-inspired experience replay; simulations report more efficient and steady learning performance.

  • Problem

    The paper considers how to plan cellular-connected UAV trajectories to minimize weighted flight time and expected outage duration without prior knowledge of the wireless environment.

    Source: §I · p. 4

  • Method

    The authors map navigation to an MDP and propose a DRL solution with quantum-inspired experience replay (QiER).

    Source: §I · p. 4

  • Results

    Simulations report that DRL-QiER achieves more efficient and steady learning performance.

    Source: §I · p. 4

  • Takeaways & Limitations

    DRL-QiER requires no extra neural networks like SNARM and much less hyper-parameter tuning than DCRL or DRL-PER.

    Source: §I · p. 4

Abstract

from arXiv · show

In cellular-connected unmanned aerial vehicle (UAV) network, a minimization problem on the weighted sum of time cost and expected outage duration is considered. Taking advantage of UAV's adjustable mobility, an intelligent UAV navigation approach is formulated to achieve the aforementioned optimization goal. Specifically, after mapping the navigation task into a Markov decision process (MDP), a deep reinforcement learning (DRL) solution with novel quantum-inspired experience replay (QiER) framework is proposed to help the UAV find the optimal flying direction within each time slot, and thus the designed trajectory towards the destination can be generated. Via relating experienced transition's importance to its associated quantum bit (qubit) and applying Grover iteration based amplitude amplification technique, the proposed DRL-QiER solution commits a better trade-off between sampling priority and diversity. Compared to several representative baselines, the effectiveness and supremacy of the proposed DRL-QiER solution are demonstrated and validated in numerical results.

I. INTRODUCTION

The section presents a DRL approach with quantum-inspired experience replay (QiER) for cellular-connected UAV trajectory planning that minimizes weighted flight-time cost and expected outage duration. It characterizes QiER as making the technique more flexible and reliable, and reports more efficient and steady learning than representative baselines.

  • The study models G2A pathloss from one local-building realization and fixed 3D directional-antenna radiation, then minimizes weighted flight-time cost and expected outage duration.Source: §I · pp. 3–4
  • The path-planning problem is mapped to an MDP and addressed with a DRL solution using novel QiER.Source: §I · p. 4
  • QiER initializes priorities for new experiences, updates sampled-transition priorities through Grover iteration, and measures sampling probabilities for mini-batch training.Source: §I · p. 4
  • Against DRL-ER, DRL-PER, DCRL, and SNARM, DRL-QiER is reported to achieve more efficient and steady learning.Source: §I · p. 4
  • DRL-QiER uses no extra neural networks like SNARM and requires less hyper-parameter tuning than DCRL or DRL-PER, supporting easier, more robust implementation.Source: §I · p. 4
  • Relative to the cellular-UAV setting in, the method expands flying directions from four to eight and extends QiER from 2D discrete to 3D continuous rotation.Source: §I · p. 4

I. INTRODUCTION

  • The introduction describes the QiER technique as more flexible and reliable.The passage links this characterization to additional hyper-parameters, but does not specify the relationship in the provided fragment.Source: §I · p. 4

II. SYSTEM MODEL

The model considers cellular-served UAVs flying from initial locations to a common destination while balancing travel time and wireless transmission quality. It models directional base-station antennas with three-dimensional radiation patterns.

II. SYSTEM MODEL

  • UAVs share cellular coverage and a common destination, with feasible trajectories intended to minimize travel time while maintaining satisfactory wireless quality.The navigation task focuses on an arbitrary UAV from the modeled set.Source: §II · p. 5

A. Antenna Model

  • Each base station covers three sectors using vertically placed M-element ULAs whose boresights point toward their sectors.The antenna model follows the 3GPP specification for cellular base-station antennas.Source: A. Antenna Model · p. 6

B. Pathloss Model

The pathloss model uses the local building distribution to determine whether UAV-to-sector links are line-of-sight or obstructed, then applies the 3GPP urban-macro model.

B. Pathloss Model

  • The adopted 3GPP urban-macro pathloss model gives LoS pathloss as a function of UAV-to-sector distance and carrier frequency.The distance is the Euclidean distance between the UAV and the sector’s ULA.Source: B. Pathloss Model · p. 7
  • Checking potential blockages between the UAV and sectors determines whether each G2A link is LoS or NLoS for the modeled building distribution.The distribution includes buildings’ horizontal ground locations and heights, as well as a corresponding 3D view.Source: B. Pathloss Model · p. 8

C. SINR at UAV

The section models received SINR and outage along the UAV’s trajectory, then formulates navigation as minimizing a weighted sum of flight time and expected outage duration. The formulation is discretized into time slots for DRL-based solution.

C. SINR at UAV

  • The received SINR is modeled for the UAV using sector transmissions, small-scale fading, and additive Gaussian noise, with association to a serving sector.The serving-sector SINR is random because of small-scale fading.Source: C. SINR at UAV · p. 8
  • The paper’s building-aware G2A pathloss method is described as more practical than probabilistic models that characterize only average pathloss.The method determines LoS or NLoS from potential blockages in a realization of the local building distribution.Source: C. SINR at UAV · p. 8

D. Problem Formulation

  • The optimization minimizes a weighted sum of flight time and ergodic outage duration by designing the trajectory and sector association.The two costs form a dilemma: shorter flight time may leave less opportunity to visit areas with lower transmission outage probability.Source: D. Problem Formulation · p. 9
  • Discretizing the flight into time slots makes navigation trackable, while slot duration trades approximation accuracy against computational complexity.Within a slot, distance, pathloss, and antenna gain are treated as approximately static; outage probability can be estimated from signal measurements.Source: D. Problem Formulation · p. 9
  • A DRL-aided solution with a quantum-inspired experience replay framework is proposed for the optimization problem.The formulation notes that antenna gain and LoS/NLoS condition depend on UAV location, complicating standard optimization.Source: D. Problem Formulation · p. 10

III. DEEP REINFORCEMENT LEARNING

The section introduces reinforcement learning as an MDP-based process of selecting actions, observing rewards, and learning state-action values. It then describes deep Q-function approximation and refinements used in deep RL.

III. DEEP REINFORCEMENT LEARNING

  • An MDP represents states, actions, transition probabilities, rewards, and discounting; an agent learns by acting and collecting state transitions and rewards.The state-action value function represents discounted long-term return, and the optimal policy selects actions to maximize it.Source: §III · p. 11
  • Q-learning iteratively estimates the optimal state-action values through temporal-difference updates from sampled experience.The optimal values require sufficiently experienced state-action pairs and a properly chosen learning rate.Source: §III · p. 12

A. Deep RL

  • Deep neural networks approximate the Q function when a tabular Q-table is impractical for high-dimensional state or action spaces.A target network is used to enhance learning stability.Source: A. Deep RL · p. 12
  • DDQN separates action selection from evaluation to relax maximization bias, while dueling DQN separates state value from action advantages.Dueling DQN is described as offering better policy evaluation quality, especially when many actions have similar values.Source: A. Deep RL · p. 12

B. Multi-Step Learning

Multi-step learning balances one-step updates against the computation burden of Monte Carlo methods by using a chosen number of future steps. Experience replay reuses stored transitions, while prioritized replay favors transitions considered more valuable; the section also introduces quantum-state concepts used later.

B. Multi-Step Learning

  • Multi-step learning uses N_ms-step-forward knowledge to balance one-step learning against Monte Carlo computation, with performance depending on a carefully chosen step length.The N_ms-step discounted accumulated reward is used to derive the corresponding loss function.Source: B. Multi-Step Learning · p. 13

C. Prioritized Experience Replay

  • Experience replay stores transitions in a finite-capacity buffer and samples mini-batches, mixing recent and earlier experiences to reduce temporal correlations.The stated trade-offs are added computation and memory, weighed against the resource cost of agent-environment interactions.Source: C. Prioritized Experience Replay · p. 13
  • Prioritized experience replay samples recorded transitions according to their value rather than their occurrence frequency, addressing inefficient sampling and loss of rare meaningful transitions.The background also identifies highly correlated transitions as a concern for the independent-and-identically-distributed transition assumption.Source: C. Prioritized Experience Replay · p. 13

IV. QUANTUM STATE AND QUANTUM AMPLITUDE AMPLIFICATION

  • Quantum-state and amplitude-amplification concepts are introduced as foundations for the DRL-QiER solution.A quantum state is described as a unit vector, with measurement collapsing it onto an eigenstate according to its probability amplitude.Source: §IV · p. 14, A. Quantum State · p. 14

A. Quantum State

  • The n-qubit state follows the superposition principle and can be regarded as a superposition of 2^n eigenstates.For a single qubit, the state is a superposition of eigenstates |0⟩ and |1⟩, whose measurement probabilities are determined by their amplitudes.Source: A. Quantum State · p. 14

B. Quantum Amplitude Amplification

Flexible Grover-iteration phase parameters can change a qubit’s collapse probabilities through amplitude amplification. The UAV navigation task is formulated as an MDP with discrete direction choices and rewards that trade movement cost against outage duration and coverage quality.

B. Quantum Amplitude Amplification

  • Flexible phase parameters in Grover iteration modify a qubit’s probability amplitudes and can produce arbitrary parametric rotations that change the probability of collapsing onto |0⟩.The paper chooses one iteration with dynamic phase parameters because fixed phases with a variable iteration count change amplitudes only discretely.Source: B. Quantum Amplitude Amplification · pp. 14–15

V. DRL-QIER ALGORITHM

  • The DRL-QiER algorithm is developed to solve the trajectory-planning optimization problem by mapping the navigation task into an MDP.The reward design encourages reaching the destination in fewer steps while avoiding boundaries and weak-coverage areas.Source: §V · p. 16, A. The MDP Formulation · pp. 16–17

A. The MDP Formulation

  • The MDP uses UAV location as a continuous state and discretizes feasible flying directions into eight options to address the continuous-action search space.Its reward penalizes each movement and expected outage duration, while destination arrival and boundary collision terminate an episode with positive and negative rewards, respectively.Source: A. The MDP Formulation · pp. 16–17

B. Quantum-Inspired Representation of Experience’s Priority

QiER represents each stored transition’s priority with an associated qubit, whose collapse probability determines whether the transition is accepted or denied. Its initialization and preparation phases set and update those probabilities, using Grover iteration to incorporate transition importance and replay history.

B. Quantum-Inspired Representation of Experience’s Priority

  • In the proposed DRL-QiER solution, each experienced transition’s priority is represented by the k-th qubit, with k indicating that transition’s location index in the QiER buffer; eigenstate |0⟩ means accepting.The amplitudes satisfy |αk|² + |βk|² = 1.Source: B. Quantum-Inspired Representation of… · p. 17

C. QiER Framework

1) Quantum Initialization Phase:

  • A newly recorded transition’s qubit is initialized to |0⟩, giving an unsampled transition the highest priority so the agent is more likely to learn from it.This choice reflects that the agent has not yet trained on that transition.Source: 1) Quantum Initialization Phase · p. 18

2) Quantum Preparation Phase:

  • After sampling, the associated qubit is reset to the uniform state and one flexible-parameter Grover iteration maps the updated priority into that qubit.The absolute TD error reflects transition priority; the update also incorporates replay count and training progress, including the episode ratio used to steer a phase parameter early in training.Source: 2) Quantum Preparation Phase · pp. 18–19
  • To address over-training from experiences sampled at undesirably high frequency and unfair, biased sampling associated with a finite QiER buffer, replay time of each stored transition is considered in quantum preparation to enrich sample diversity.Source: 2) Quantum Preparation Phase · p. 19

3) Quantum Measurement Phase:

The quantum measurement phase converts each stored qubit’s collapse probability onto |0⟩ into a normalized probability of selecting its transition. The sampled transitions remain in the buffer until discarded, and the quantum operations can be imitated on conventional computing devices.

3) Quantum Measurement Phase:

  • Measurement assigns each stored transition a picking probability by normalizing its qubit’s |0⟩ collapse probability by the sum across stored experiences.The buffer’s picking probabilities guide repeated selections until the mini-batch size is reached.Source: 3) Quantum Measurement Phase · p. 20
  • The initialization, preparation, and measurement processes can be imitated on conventional computing devices without real quantum computation.The total number of picks during sampling equals the mini-batch size.Source: 3) Quantum Measurement Phase · pp. 20–21
  • Sampled transitions remain in the QiER buffer until discarded, and their associated qubits are reset to the uniform state before another preparation phase.The mini-batch is sampled from the buffer for network training.Source: 3) Quantum Measurement Phase · pp. 20–21

D. The Proposed DRL-QiER Solution

The section presents a DDQN-based DRL-QiER navigation learner and outlines its training workflow, simulation setting, and network configuration.

D. The Proposed DRL-QiER Solution

  • D3QN approximates the Q function, with Nms-step learning and a target network used to stabilize learning.The online network is trained by minimizing a loss with stochastic gradient descent, while the selected action in the target uses the online D3QN.Source: D. The Proposed DRL-QiER Solution · pp. 21–22
  • The UAV selects actions with an ϵ-greedy policy, records transitions in a sliding buffer, and trains the online D3QN from mini-batches after the QiER buffer fills.Episodes end at the destination, a boundary, or the step threshold; the target network is periodically updated to the online network.Source: D. The Proposed DRL-QiER Solution · p. 21

VI. NUMERICAL RESULTS

  • The numerical-results section reports that it evaluates DRL-QiER and compares its performance against several baselines.The supplied passages state that comparisons are performed but provide no outcome or metric values.Source: §VI · p. 23

A. Simulation Environment Setups

  • The simulation setting specifies the UAV airspace, building distribution, G2A fading model, and a fixed common destination.Building heights follow a Rayleigh distribution, and the G2A small-scale fading component uses a block Nakagami-m channel model.Source: A. Simulation Environment Setups · p. 23

B. Structure of DNNs and Hyper-parameter Settings for Learning Process

  • The online D3QN uses fully connected layers, including a duelling layer that estimates state value and action advantages.The first three hidden layers contain 512, 256, and 128 neurons; the target D3QN shares the online network's structure.Source: B. Structure of DNNs and… · pp. 23–24

C. Training of the DRL-QiER algorithm

The section examines DRL-QiER’s training behavior and learned trajectories, reporting converging moving-average returns despite fluctuations and lower average EOD and time costs than the baselines, including the lowest average EOD overall and the cheapest time cost among the DRL-aided solutions.

C. Training of the DRL-QiER algorithm

  • DRL-QiER’s trajectories guide UAVs from varied initial locations to the destination while avoiding higher-TOP regions, including narrow near-zero-TOP slots.The paths are adapted to the TOP distribution while seeking fewer moving steps.Source: C. Training of the DRL-QiER algorithm · p. 25

D. Performance Comparison

  • Early in training, DRL-PER, DRL-QiER, and DCRL achieve higher moving-average returns than DRL-ER, while SNARM offers satisfactory learning performance.The comparison identifies episodes 500–750 as an example of the early-stage advantage over DRL-ER.Source: D. Performance Comparison · p. 27
  • DRL-QiER shows steadier learning with less fluctuation than DRL-PER and DCRL, as QiER balances sampling priority and diversity.DRL-PER’s fluctuations are attributed to ignoring replay frequency, which can repeatedly sample transitions with large TD errors.Source: D. Performance Comparison · p. 27
  • Compared with SNARM’s extra-DNN training burden and DCRL’s difficult function and hyperparameter choices, DRL-QiER needs less tuning and no extra DNN.The paper states these features make DRL-QiER easier and more robust to implement.Source: D. Performance Comparison · p. 27
  • The straight-line strategy has the cheapest average time cost but the highest average EOD.The paper characterizes that EOD outcome as undesirable.Source: D. Performance Comparison · pp. 27–28
  • In the late training state, specifically the 1800–2000 episode slot, the proposed DRL-QiER method outperforms other baselines on both average EOD and average time cost.Source: D. Performance Comparison · p. 27

VII. CONCLUTION

The paper formulates cellular-connected UAV navigation as minimizing weighted time cost and expected outage duration, and proposes a DRL-QiER solution. The authors report superiority over DRL-aided baselines and straight-line strategy, and suggest QiER may extend to other experience-replay-dependent DRL frameworks.

VII. CONCLUTION

  • The navigation objective is to minimize the weighted sum of time cost and expected outage duration for UAV trajectories toward a common destination.The initial UAV locations are randomly generated.Source: §VII · p. 28
  • DRL-QiER is proposed to navigate the UAV, with QiER intended to improve the DRL agent’s learning efficiency..Source: §VII · p. 28
  • The authors report that DRL-QiER is effective and superior to several DRL-aided baselines and the straight-line strategy..Source: §VII · pp. 28–29
  • The authors suggest QiER may extend to other existing DRL frameworks that depend on experience replay..Source: §VII · p. 29

Related papers

Loading 2108.13184v2…