PaperlayerView exact revision on arXiv

Source-linked AI summary

Cellular-Connected UAVs over 5G: Deep Reinforcement Learning for Interference Management

Ursula Challita, Walid Saad, Christian Bettstetter

arXiv:1801.05500v1cs.ITcs.AIcs.GT

TL;DR

The paper tackles interference management for cellular-connected UAVs while balancing energy efficiency and wireless latency. It formulates the problem as a dynamic game and applies deep reinforcement learning with ESN cells to jointly learn paths and resource decisions. The proposed approach reaches SPNE upon convergence and reports improved latency and ground-UE rate with altitude-dependent interference control.

  • Problem

    Cellular-connected UAV deployment creates interference-management challenges, while existing work provides limited performance analysis and no concrete joint optimization solution for aerial and ground UEs.

  • Method

    A deep reinforcement learning algorithm based on ESN cells solves a dynamic noncooperative game by learning each UAV’s path, transmit power level, and cell association vector.

  • Results

    The proposed approach achieves better wireless latency per UAV and rate per ground UE while requiring a number of steps comparable to the shortest path scheme.

  • Takeaways & Limitations

    UAV altitude is an important control factor because its suitable value varies with ground network density and the UAV’s objective.

Abstract

from arXiv · show

In this paper, an interference-aware path planning scheme for a network of cellular-connected unmanned aerial vehicles (UAVs) is proposed. In particular, each UAV aims at achieving a tradeoff between maximizing energy efficiency and minimizing both wireless latency and the interference level caused on the ground network along its path. The problem is cast as a dynamic game among UAVs. To solve this game, a deep reinforcement learning algorithm, based on echo state network (ESN) cells, is proposed. The introduced deep ESN architecture is trained to allow each UAV to map each observation of the network state to an action, with the goal of minimizing a sequence of time-dependent utility functions. Each UAV uses ESN to learn its optimal path, transmission power level, and cell association vector at different locations along its path. The proposed algorithm is shown to reach a subgame perfect Nash equilibrium (SPNE) upon convergence. Moreover, an upper and lower bound for the altitude of the UAVs is derived thus reducing the computational complexity of the proposed algorithm. Simulation results show that the proposed scheme achieves better wireless latency per UAV and rate per ground user (UE) while requiring a number of steps that is comparable to a heuristic baseline that considers moving via the shortest distance towards the corresponding destinations. The results also show that the optimal altitude of the UAVs varies based on the ground network density and the UE data rate requirements and plays a vital role in minimizing the interference level on the ground UEs as well as the wireless transmission delay of the UAV.

I. INTRODUCTION

The paper addresses interference-aware path planning for multiple cellular-connected UAVs, whose deployment affects both aerial and ground users. It proposes an online deep reinforcement learning framework that jointly learns UAV trajectories and resource decisions while balancing energy efficiency, latency, and interference.

  • Motivation: Cellular-connected UAVs can create substantial mutual interference and interference on ground users, making interference management necessary for wide-scale deployment.
  • Research gap: Existing studies largely analyze the impact of cellular-connected UAVs on ground networks without providing a concrete performance-optimization solution for jointly served aerial and ground UEs.
  • Research gap: Prior path-planning work is mainly designed for UAVs acting as base stations, does not account for UAV-UE interference, or considers only one cellular-connected UAV with offline optimization.
  • Approach: The proposed framework uses deep reinforcement learning with echo state network cells to optimize the online trajectories of multiple cellular-connected UAVs.
  • Approach: Each UAV jointly learns its path, transmit power level, and cell association vector through a dynamic noncooperative game and self-organizing resource allocation.
  • Results: The algorithm reaches a subgame perfect Nash equilibrium upon convergence, derives altitude bounds, and improves the tradeoff among energy efficiency, wireless latency, and ground-network interference.

II. SYSTEM MODEL

The system models uplink communication between cellular-connected UAVs, ground UEs, and base stations over shared resource blocks. UAV mobility, propagation, interference, achievable rate, and wireless latency are specified for this network.

  • Network architecture: UAVs and ground UEs communicate with base stations over an uplink cellular network divided into resource blocks.Each resource block is allocated to at most one UAV or ground UE at a base station.
  • UAV mobility: Each UAV moves from a predetermined initial location to a destination at fixed altitude and constant speed.Mobility is represented on a discretized grid of unit areas, producing finite candidate paths.
  • UAV mobility: A UAV path is an ordered sequence of unit areas beginning at its initial location and ending at its destination.The discretization is chosen so locations are approximately constant within each area during traversal.
  • Channel model: The UAV-BS channel uses a Rician fading model, while terrestrial UE-BS links use Rayleigh fading.The Rician model accounts for line-of-sight and multipath effects on UAV-BS links.
  • Performance model: Achievable rates and wireless latency depend on transmit power, resource-block allocation, channel gains, interference, and packet arrival rates.UAV latency is modeled using an M/D/1 queue, with service rate determined by the achievable UAV data rate.

B. Problem Formulation

The problem jointly optimizes each UAV’s path, cell association, and transmit power while balancing interference, transmission delay, and path length. Because the resulting mixed-integer nonlinear program is difficult for large networks, a distributed online formulation is motivated.

  • Objective: The objective is to find each UAV’s optimal path according to its mission objectives.The formulation jointly considers trajectory, communication, and network-impact decisions.
  • Decision variables: The decision variables jointly specify path formation, UAV-BS association, and transmit power at each visited location.Binary path and association variables determine movement and serving-cell selection, while power is bounded by each UAV’s maximum.
  • Constraints: Constraints enforce valid trajectories, one serving base station per visited location, feasible power allocation, and an SINR threshold.The SINR threshold is application- and mission-specific and supports successful packet decoding at the serving base station.
  • Objective: The optimization minimizes interference caused to ground UEs and neighboring UAVs, wireless transmission delay, and path length.Multi-objective weights control the tradeoff among these three metrics and can reflect mission requirements.
  • Solution setting: The formulation is a mixed-integer nonlinear program whose computational complexity becomes problematic for large networks.A distributed online approach is preferred because centralized control can incur latency and requires complete network knowledge.

A. Game-Theoretic Formulation

The paper formulates multi-UAV path planning, power control, and cell association as a finite dynamic noncooperative game. Altitude bounds reduce the action space while preserving SINR and ground-UE rate requirements.

  • Game formulation: Each UAV autonomously selects its next movement, transmit power level, and cell association as a joint action.Movement actions use fixed steps in four directions or no movement, while power and association are selected simultaneously.
  • Game formulation: The multi-agent path-planning problem is modeled as a finite dynamic noncooperative game with perfect information.The game includes UAVs, stages, action sets, observed states, behavioral strategies, and payoff functions.
  • Utility: Each UAV’s utility incorporates interference, delay, movement toward its destination, and penalties for violating the SINR constraint.The utility depends on the UAV’s state, its action, and the actions of the other UAVs.
  • Altitude bounds: Altitude bounds are derived for every network state and action, for all values of the multi-objective weights and penalty coefficient.The lower bound is tied to the minimum flight altitude, while the upper bound supports the required communication constraints.
  • Altitude bounds: The optimal altitude depends on the UAV objective, ground-base-station locations, network parameters, and interference from other users and UAVs.UAVs adjust altitude according to the derived bounds, reducing the game’s action space and simplifying equilibrium search.

B. Equilibrium Analysis

The equilibrium analysis defines each UAV’s objective as maximizing expected discounted utility over future states and actions. Because exact SPNE computation requires knowledge of future possibilities that grows difficult with more UAVs, deep ESN-based reinforcement learning is introduced.

  • Equilibrium concept: The game seeks a subgame perfect Nash equilibrium in which strategies form a Nash equilibrium in every subgame.Behavioral strategies assign probabilities to actions at each network state.
  • UAV objective: Each UAV maximizes the expected sum of discounted rewards over immediate and future utilities.The discount factor γ^l lies between 0 and 1, and expectations are taken over state-action trajectories.
  • Coupled decisions: Trajectory optimization, cell association, and transmit power are coupled, and their optimal values vary with UAV objectives.This coupling is represented within each UAV’s utility and strategy selection.
  • Computational challenge: Computing an SPNE becomes challenging as the number of UAVs increases because future actions and network states must be considered.The analysis notes that each UAV would need full knowledge of future reward functions and possible actions.
  • Learning solution: The proposed solution uses deep reinforcement learning based on echo state networks, a recurrent architecture for solving the game’s SPNE.Recurrent memory supports use of previous state information when predicting future actions.

IV. DEEP REINFORCEMENT LEARNING FOR ONLINE PATH PLANNING AND RESOURCE MANAGEMENT

The paper introduces a deep ESN-based reinforcement-learning framework that lets UAVs store previous states and learn future network states for SPNE-based path planning and resource management.

  • The proposed deep ESN architecture stores previous network states while learning future network states for UAV decision-making.
  • The reinforcement-learning algorithm uses the deep ESN architecture to learn a subgame perfect Nash equilibrium for each UAV.

A. Deep ESN Architecture

The deep ESN stacks nonlinear reservoir layers to represent temporal information hierarchically while preserving efficient reservoir-computing training. Its inputs encode network states, and its trained output approximates UAV rewards for SPNE learning.

  • ESNs use feedback-connected reservoir units, with input, recurrent, and output weight matrices forming the reservoir-computing architecture.
  • Only the output weights are trained, making ESN training typically quick and computationally efficient compared with other recurrent neural networks.
  • Deep ESNs stack multiple nonlinear reservoir layers to learn hierarchical temporal representations at different abstraction levels.
  • Each UAV’s external input v_j(t) represents the current network state, while deeper layers receive the previous layer’s state.
  • The deep ESN output is trained to approximate each UAV’s reward function from network state and action information for SPNE learning.
  • The reservoir spectral radius must be strictly smaller than 1 for stability, and it is related to the reservoir’s variable memory length.
  • The architecture described for the proposed reservoir consists of two layers.

B. Update Rule Based on Deep ESN

Each UAV updates its deep ESN state from network observations and actions, estimates rewards, and trains output weights by minimizing prediction error.

  • Each UAV uses the deep ESN update phase to store and estimate network information over time.
  • Each UAV minimizes e_j(v_j(t)) = |r_j(v_j(t),z_j(t),z_-j(t)) − y_j(v_j(t),z_j(t))|.
  • The first-layer reservoir state follows a leaky-integrator transition, where the leaking parameter controls response speed to the input.
  • The deep ESN output estimates each UAV’s reward from the current network state and the actions of that UAV and the others.
  • Temporal-difference reinforcement learning trains the output matrix with linear gradient descent using the reward error signal.

C. Proposed Deep RL Algorithm

The proposed deep RL algorithm trains UAV policies through parallel action selection, state and resource updates, reward computation, and ESN weight adaptation, then executes greedy actions during testing. If it converges, its strategy profile corresponds to an SPNE, while its practical training complexity is reduced under fixed altitude.

  • The algorithm combines a deep ESN architecture with multi-agent reinforcement learning to learn an SPNE in behavioral strategies.
  • During training, UAVs balance utility-maximizing actions with exploration using an ϵ-greedy policy.
  • Selected actions update each UAV’s location, cell association, and transmission power, after which UAVs broadcast actions and compute rewards.
  • The training loop updates reservoir states, computes ESN outputs, and adapts output weights using the linear gradient-descent rule.
  • During testing, each UAV selects the action with the highest estimated output for its current state.
  • Analytical convergence guarantees are challenging because convergence depends strongly on training hyperparameters, so the paper assesses convergence through simulations under reasonable choices.
  • If Algorithm 1 converges, its convergence strategy profile corresponds to an SPNE of game G.
  • The training algorithm has complexity O(A3), reducing to O(A2) when UAV altitude is fixed.

V. SIMULATION RESULTS AND ANALYSIS

The simulations show that UAV altitude, network density, mission objectives, and state information jointly shape the tradeoff among latency, ground-UE rate, energy efficiency, and interference. The proposed approach improves this tradeoff over shortest-path planning while using comparable movement steps.

  • Altitude effects: In dense networks, UAVs should use lower altitudes for latency-critical missions and higher altitudes when minimum ground-UE data rates are required.Optimal altitude bounds vary with SINR thresholds, transmit power, network density, and ground-network data requirements.
  • Path planning comparison: The proposed path selects routes away from dense areas while remaining near serving BSs, reducing ground-UE interference and UAV wireless latency.This route also maintains a step count comparable to shortest-path planning.
  • Path planning comparison: 25% increase in average rate per ground UE and 47% decrease in wireless latency compared with shortest-path planning, with the same number of steps.These results are reported for one UAV.
  • Multiple-UAV performance: For 5 UAVs, the proposed scheme achieves a 37% increase in average achievable rate per ground UE, a 62% decrease in average wireless latency per UAV, and a 14% increase in energy efficiency.The comparison is against the shortest-path scheme.
  • Altitude effects: As UAV altitude increases, wireless latency per UAV increases while the average data rate per ground UE also increases across the studied utility functions.Higher altitude increases UAV-BS distance and path loss but decreases interference caused to neighboring BSs.
  • Network densification: As the number of ground BSs increases from 10 to 30, latency rises from 13 ms to 47 ms while ground-UE rate falls from 0.86 Mbps to 0.48 Mbps.This example corresponds to minimizing interference together with energy efficiency.
  • State representation: Increasing the number of nearest BSs in the state definition from 1 to 5 increases average rate per ground UE by 28% for interference-minimizing scenarios.The larger state gives UAVs more information about their surroundings for selecting lower-interference locations.
  • Complexity: A tradeoff exists between ground-UE performance and algorithmic complexity because larger network representations increase neurons and convergence iterations.The proposed algorithm therefore balances performance gains against running complexity.

VI. CONCLUSION

The paper proposes an interference-aware path-planning scheme that models UAV interactions as a noncooperative game and solves it with deep reinforcement learning based on ESN cells. The approach targets interference, latency, and energy-efficiency tradeoffs, reaches an SPNE if it converges, and achieves favorable simulated performance with altitude-dependent effects.

  • The scheme models cellular-connected UAVs as players in a noncooperative game to minimize ground-network interference and wireless transmission latency.Each UAV also seeks to balance these objectives with energy efficiency while transmitting mission-related data.
  • A deep RL algorithm based on ESN cells enables each UAV to autonomously select its next location, transmission power, and cell association vector.The learned policy adapts these decisions to changes in the network.
  • The proposed algorithm is guaranteed to reach an SPNE if it converges.
  • The approach achieves better wireless latency per UAV and rate per ground UE while requiring a number of steps comparable to the shortest-path scheme.
  • UAV altitude plays a vital role in minimizing ground-UE interference and UAV wireless transmission delay, varying with network density, objective, and other UAVs' actions.

APPENDIX

The appendix derives upper and lower altitude bounds for each UAV under network-state, interference, SINR, and delay constraints. These bounds depend on neighboring base stations, propagation distances, interference thresholds, and network-design parameters tied to density, UAV count, and ground-UE rate requirements.

  • The upper altitude bound is derived for a given network state and action under the transmission-link constraint to the serving base station.The derivation assumes negligible base-station altitude and uses serving-base-station distance and interference-plus-noise terms.
  • A delay-only objective derives an altitude condition associated with the UAV's SINR upper limit.The derivation assumes negligible altitude for the neighboring base station and identifies the altitude at which the SINR limit is achieved or the transmission constraint is violated.
  • A lower altitude bound is derived when a UAV minimizes interference alone, requiring interference at neighboring base stations to remain below a predefined value.The bound is evaluated across neighboring base stations and depends on the altitude achieving the prescribed interference level.
  • The neighboring-base-station calculations use Euclidean UAV-to-base-station distances and are solved separately for each neighboring base station.
  • The interference threshold is a network-design parameter dependent on ground-network density, UAV count, and ground-UE data-rate requirements.It is part of an admission-control policy limiting the number of UAVs and their corresponding ground-network interference.
Loading 1801.05500v1…