PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Design and Power Control for Multi-UAV Assisted Wireless Networks: A Machine Learning Approach

Xiao Liu, Yuanwei Liu, Yue Chen, Lajos Hanzo

arXiv:1812.07665v2eess.SP

TL;DR

The paper tackles joint 3D trajectory design and power control for multiple UAVs serving mobile users while maximizing instantaneous sum rate under user-rate requirements. It combines multi-agent Q-learning, Twitter-based mobility data, and ESN prediction to plan UAV placement and movement. The authors report improved ESN prediction with larger reservoirs, about 17% throughput gains, and convergence under mild conditions.

  • Problem

    The problem is advance multi-UAV trajectory and power planning when users move, building on a gap in research that mainly assumes static or current-time user locations.

  • Method

    The method combines initial multi-agent Q-learning placement, Twitter GPS mobility data, ESN user-position prediction, and multi-agent Q-learning for per-slot UAV trajectory and power control.

  • Results

    About 17% throughput gains are achieved, while ESN prediction accuracy improves as reservoir size increases and the multi-agent Q-learning algorithm converges under mild conditions.

  • Takeaways & Limitations

    The framework provides a prediction-based route to planning cooperative UAV deployment and power control for mobile-user wireless service.

Abstract

from arXiv · show

A novel framework is proposed for the trajectory design of multiple unmanned aerial vehicles (UAVs) based on the prediction of users' mobility information. The problem of joint trajectory design and power control is formulated for maximizing the instantaneous sum transmit rate while satisfying the rate requirement of users. In an effort to solve this pertinent problem, a three-step approach is proposed which is based on machine learning techniques to obtain both the position information of users and the trajectory design of UAVs. Firstly, a multi-agent Q-learning based placement algorithm is proposed for determining the optimal positions of the UAVs based on the initial location of the users. Secondly, in an effort to determine the mobility information of users based on a real dataset, their position data is collected from Twitter to describe the anonymous user-trajectories in the physical world. In the meantime, an echo state network (ESN) based prediction algorithm is proposed for predicting the future positions of users based on the real dataset. Thirdly, a multi-agent Q-learning based algorithm is conceived for predicting the position of UAVs in each time slot based on the movement of users. In this algorithm, multiple UAVs act as agents to find optimal actions by interacting with their environment and learn from their mistakes. Additionally, we also prove that the proposed multi-agent Q-learning based trajectory design and power control algorithm can converge under mild conditions. Numerical results are provided to demonstrate that as the size of the reservoir increases, the proposed ESN approach improves the prediction accuracy. Finally, we demonstrate that throughput gains of about 17% are achieved.

I. INTRODUCTION

The paper addresses 3D trajectory design and power control for multiple UAVs serving mobile users, where prior work largely assumes static users or current-time locations. It proposes a machine-learning framework combining user-mobility prediction with cooperative UAV deployment.

  • Pre-deploying UAVs from predicted full user-location information could support advance trajectory design, reduce response time, and reduce interaction with the ground control center.
  • Existing UAV research has mainly studied 2D multi-UAV placement or single-UAV movement with static users.
  • The paper formulates joint trajectory and power-control design for multiple UAVs moving in 3D space to maximize throughput while serving users.
  • The proposed three-step approach uses multi-agent Q-learning for initial placement, ESN prediction for user mobility, and multi-agent Q-learning for UAV trajectory acquisition and power control.
  • Twitter data containing GPS coordinates and timestamps is used with an ESN to acquire users’ mobility information for trajectory planning.
  • The multi-agent formulation supports cooperative UAV deployment, and the proposed algorithms are shown to converge to an optimal state.

D. Organization and Notations

The system model describes multiple UAVs serving clustered users through downlink FDMA, with user mobility informing advance 3D UAV trajectory adjustment. It models air-to-ground propagation, interference, achievable rates, and transmit-power constraints.

  • II. SYSTEM MODEL: Multiple UAVs act as aerial base stations for clustered users in a downlink FDMA network, with each UAV simultaneously serving multiple users.
  • A. Mobility Model: User positions are represented by time-varying two-dimensional coordinates, and Twitter GPS data with timestamps supplies real-time mobility information for UAV planning.
  • A. Mobility Model: UAV trajectories are designed in advance from predicted user movement, with both horizontal position and altitude adjustable during each flight period.
  • B. Transmission Model: The air-to-ground channel randomly encounters LoS and NLoS conditions, with LoS probability depending on elevation angle and environmental constants.
  • B. Transmission Model: The model includes path loss, received SINR, interference from other UAVs, user achievable rate, and overall instantaneous sum rate.
  • B. Transmission Model: Transmit power is bounded by Pmax, uniformly allocated among each UAV’s associated users, and lower-bounded to satisfy every user’s rate requirement.
  • B. Transmission Model: Continuous user roaming can shift users between UAV service areas, increasing interference and reducing received SINR, which makes accurate power control important.

C. Problem Formulation

The paper formulates joint UAV trajectory and transmit-power control to maximize total transmit rate while meeting users’ rate requirements. The problem is non-convex and difficult to solve exhaustively, motivating a low-complexity multi-agent Q-learning approach supported by predicted user movement.

  • The optimization jointly determines each UAV’s trajectory and transmit power at every time slot to maximize total transmit rate under per-user rate requirements.
  • Each user is assigned to one UAV cluster, while altitude, rate, and transmit-power constraints define the feasible solution.
  • Trajectory design seeks optimal UAV positions at each time slot, which improves instantaneous transmit rate.
  • The objective is non-convex in UAV coordinates and altitude, and exhaustive search has excessive complexity.
  • Multi-agent Q-learning is invoked to obtain a high-probability solution while searching only a small fraction of the solution space.

III. ECHO STATE NETWORK ALGORITHM FOR PREDICTION OF USERS’ MOVEMENT

The paper predicts user mobility from real-world Twitter location data rather than relying solely on synthetic mobility models. The collected trajectories provide the basis for estimating future user positions.

  • Synthetic mobility models may distribute movement directions uniformly, so the paper predicts mobility using a real dataset collected from Twitter.
  • Online social-network applications provide geographical data describing anonymous user trajectories in the physical world.
  • The dataset contains 12,000 tweets collected near Oxford Street in London on 14 March 2018, including 50 users who tweeted more than three times.
  • Interpolation records each user’s position every 200 seconds, producing trajectories represented by a user-position vector for each time slot.

B. Echo State Network Algorithm for the Prediction of Users’ Movement

The ESN predicts future user positions from Twitter-derived position vectors using a reservoir-based architecture. Its reservoir size affects prediction accuracy and must be balanced against memory, complexity, and overfitting considerations.

  • The ESN maps Twitter-derived user-position vectors to predicted position vectors through input, reservoir, and output layers.
  • Only the output weight matrix is trainable after the network is established, while the reservoir and its connections remain fixed.
  • Prediction accuracy is evaluated with mean square error, comparing predicted and real user positions.
  • The reservoir is a sparse recurrent network with short-term memory, using randomly generated input and recurrent weight matrices and a leakage rate in [0, 1).
  • Larger reservoirs can improve prediction precision and memory capacity, but they increase computational demands and the probability of overfitting.
  • The ESN procedure trains on 75% of the dataset and tests on the remaining 25%, then returns predicted user coordinates.

IV. JOINT TRAJECTORY DESIGN AND TRANSMIT POWER CONTROL OF UAVS

The proposed control stage uses multi-agent Q-learning to adapt multiple UAVs’ three-dimensional movement and transmit power to clustered users. States encode UAV positions and power, while agents exchange information to encourage cooperative actions.

  • Each cluster is served by one UAV whose flight trajectory and transmit power are adapted to maximize the sum transmit rate in each time slot.
  • Multi-agent Q-learning extends independent UAV agents by incorporating other agents’ states and actions to promote cooperative rewards.
  • Each UAV state includes its three-dimensional position and transmit power, with corresponding state-space bounds for coordinates, altitude, and power.
  • Convergence speed depends on the number of users and UAVs and their initial positions, with closer placement to optimal positions enabling faster convergence.
  • The approach constrains UAV mobility to seven directions even though real UAVs can fly in arbitrary directions.
  • The three power levels trade off system performance and algorithmic complexity, while arbitrary numbers of levels can be accommodated.
  • Actions specify travel direction and candidate transmit power, including seven movement options and power levels of 0.08W, 0.09W, and 0.1W.

C. Reward Function of Multi-agent Q-learning Algorithm

The multi-agent Q-learning reward is tied to users’ instantaneous sum rate, while reward design affects how quickly the algorithm converges rather than its final result.

  • The shared reward or punishment for all agents is directly related to users’ instantaneous sum rate.An action receives a favorable reward when it improves the sum rate.
  • Continuous rewards can accelerate convergence compared with binary rewards, although changing reward values does not alter the algorithm’s final result.

D. Transition of Multi-agent Q-learning Algorithm

The multi-agent transition extends Q-learning to joint agent actions and cooperative state information, with learning across episodes toward optimal UAV actions. The paper states that the learned Q-values converge to an optimal state.

  • Multi-agent Q-learning represents each agent’s Q-function using the joint actions of all agents rather than only its own action.This formulation defines future discounted rewards under the strategies followed by all agents.
  • The multi-agent reward depends on the joint action of all agents, requiring a corresponding update rule.
  • Agents share state-specific Q-table rows and select actions while cooperating during multi-agent training.Training updates one agent at a time while the other agents’ policies remain fixed.
  • Learning proceeds over episodes with state re-initialization, using a learning rate, discount factor, and ε-greedy exploration policy to choose actions each time slot.
  • Theorem 1 states that multi-agent Q-learning converges to an optimal state under the paper’s stated conditions.

E. Complexity of the Algorithm

The algorithm’s complexity includes clustering and multi-agent Q-learning components, while the Q-learning storage requirement grows exponentially with the number of agents. Simulations analyze user prediction, UAV trajectories, and power control.

  • The algorithm’s complexity is driven by the GAK-means clustering procedure and the multi-agent Q-learning trajectory-acquisition and power-control procedure.
  • For Nu users and N clusters, the clustering stage requires O(6KNu) floating-point operations to calculate all Euclidean distances.
  • The multi-agent model maintains N Q-functions, one for each agent, while assuming the learning agent observes other agents’ actions and rewards.
  • The total storage requirement is N|S|·|A|^N, increasing linearly with states, polynomially with actions, and exponentially with agents.
  • Simulations use randomized initial UAV locations and uniform transmit-power allocation to analyze instantaneous rates, user-position prediction, 3D trajectories, and power control.

A. Predicted Users’ Positions

The proposed ESN predicts user positions, while multi-agent Q-learning designs UAV deployment and trajectories with power control. Larger reservoirs improve prediction accuracy but increase overfitting risk, and the joint design adapts UAV movement to user mobility.

  • Predicted Users’ Positions: Increasing the ESN reservoir size reduces the error between real and predicted user tracks.The prediction accuracy is parameterized by reservoir size.
  • Predicted Users’ Positions: Larger neuron reservoirs improve prediction precision but increase the probability of overfitting through higher memory requirements.Reservoir size affects how many user positions the ESN can record.
  • Predicted Users’ Positions: A reservoir size of 1000 attains high prediction accuracy.
  • Predicted Users’ Positions: The ESN with reservoir size 1000 attains lower MSE than the HA and LSTM benchmarks at lower complexity than LSTM.Overall, the proposed ESN algorithm outperforms the benchmarks.
  • Trajectory Design and Power Control of UAVs: Multi-agent Q-learning converges after about 45000 episodes with three UAVs, while four UAVs require 30000 additional training episodes.The agents improve throughput by iteratively acting and learning from mistakes.
  • Trajectory Design and Power Control of UAVs: A learning rate of 0.80 yields higher throughput than 0.60 and 0.70, whereas 0.90 converges faster but is more likely to reach a sub-optimal Q∗ value.The sub-optimal value leads to lower throughput.
  • Trajectory Design and Power Control of UAVs: Re-deploying UAVs according to user movement mitigates throughput decline relative to the static scenario, and power control better maintains service quality.The proposed trajectory-acquisition and power-control algorithm also outperforms GAK-means.
  • Trajectory Design and Power Control of UAVs: Joint trajectory design and power control produce different UAV trajectories while preserving the main flying direction.Power control balances received signal power and interference power, increasing received SINR.

APPENDIX A: PROOF OF LEMMA 1

The appendix proves a bound involving line-of-sight and non-line-of-sight channel terms, using the condition under which the bound becomes equality.

  • APPENDIX A: PROOF OF LEMMA 1: The user rate requirement is imposed as r_kn(t) ≥ r_0 before rewriting the relevant expression.
  • APPENDIX A: PROOF OF LEMMA 1: The weighted channel expression P_LoSμ_LoS + P_NLoSμ_NLoS is bounded above by μ_NLoS.
  • APPENDIX A: PROOF OF LEMMA 1: Equality holds when the probability of a non-line-of-sight connection is 1.
  • APPENDIX A: PROOF OF LEMMA 1: Under the equality condition, each UAV’s maximum transmit rate must obey the resulting constraint.

APPENDIX B: PROOF OF THEOREM 1

The appendix establishes convergence of the multi-agent Q-learning algorithm by first proving single-agent convergence and then extending the result to multiple agents.

  • APPENDIX B: PROOF OF THEOREM 1: The convergence proof proceeds in two steps: single-agent convergence followed by extension to the multi-agent domain.
  • APPENDIX B: PROOF OF THEOREM 1: The Q-learning update rule is analyzed by subtracting the optimal Q value and defining the resulting error term.
  • APPENDIX B: PROOF OF THEOREM 1: The expected error is bounded by β times the current Q-function error because the operator is a β-contraction.
  • APPENDIX B: PROOF OF THEOREM 1: Bounded rewards yield a conditional variance bound, supporting convergence of the single-agent model to the optimal Q-function under the stated learning-rate conditions.
  • APPENDIX B: PROOF OF THEOREM 1: The multi-agent proof initializes the multi-agent Q value consistently and tracks its episode-wise updates.
  • APPENDIX B: PROOF OF THEOREM 1: If the multi-agent Q value converges to its optimal value at episode k + 1, the corresponding multi-agent state equation also converges to its optimum.
Loading 1812.07665v2…