Source-linked AI summary
Multi-Agent Deep Reinforcement Learning Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing
Liang Wang, Kezhi Wang, Cunhua Pan, Wei Xu, Nauman Aslam, Lajos Hanzo
TL;DR
The paper addresses joint optimization of UE fairness, UAV UE-load fairness, and UE energy consumption in a multi-UAV MEC system with mixed integer and continuous decisions. It uses multi-agent deep reinforcement learning with MADDPG for independent UAV trajectory control and a low-complexity offloading optimizer, with simulations showing improved fairness and energy performance over benchmarks.
Problem
The paper seeks to jointly optimize geographical UE fairness, each UAV’s UE-load fairness, and overall UE energy consumption in multi-UAV MEC despite mixed integer and continuous decisions.
Method
The method uses MADDPG-based multi-agent trajectory control, followed by low-complexity optimization of UE offloading decisions for given UAV trajectories.
Results
The proposed MAT achieves the best reported performance among MAT, CIRCLE, and RANDOM for UE fairness and energy consumption, while MAT reaches 0.9 on one fairness metric and RANDOM about 0.5.
Takeaways & Limitations
Simulation results indicate that coordinated UAV movement can improve UE service fairness and reduce energy consumption through increased task offloading.
Abstract
from arXiv · showhide
An unmanned aerial vehicle (UAV)-aided mobile edge computing (MEC) framework is proposed, where several UAVs having different trajectories fly over the target area and support the user equipments (UEs) on the ground. We aim to jointly optimize the geographical fairness among all the UEs, the fairness of each UAV' UE-load and the overall energy consumption of UEs. The above optimization problem includes both integer and continues variables and it is challenging to solve. To address the above problem, a multi-agent deep reinforcement learning based trajectory control algorithm is proposed for managing the trajectory of each UAV independently, where the popular Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is applied. Given the UAVs' trajectories, a low-complexity approach is introduced for optimizing the offloading decisions of UEs. We show that our proposed solution has considerable performance over other traditional algorithms, both in terms of the fairness for serving UEs, fairness of UE-load at each UAV and energy consumption for all the UEs.
I. INTRODUCTION
The introduction motivates UAV-assisted MEC and trajectory planning as a way to support ground users, while highlighting the difficulty of jointly optimizing fairness, UAV load, and UE energy consumption. The paper therefore proposes a multi-agent deep reinforcement learning solution with separate offloading optimization.
- Motivation: UAVs can provide connectivity without infrastructure, complement blocked or damaged base stations, and establish direct line-of-sight links with ground UEs.Their three-dimensional mobility can potentially boost throughput and support seamless connectivity.
- Motivation: UAV-assisted MEC lets resource-limited UEs offload computation-intensive and latency-critical applications to edge servers, potentially reducing UE energy consumption.Prior work investigated integrating UAV mobility with MEC and optimizing UE energy under QoS requirements.
- Research problem: The paper jointly optimizes geographical fairness, each UAV’s UE-load fairness, and overall UE energy consumption through UAV trajectories and UE offloading decisions.Geographical fairness reflects the QoS level of UEs served from the initial time slot through the current slot, while UAV UE-load is defined separately.
- Research problem: Because the optimization contains integer and continuous variables, traditional approaches such as convex optimization and dynamic programming are challenging to apply.The introduction identifies this mixed-variable formulation as the central computational difficulty.
- Proposed approach: The proposed solution applies multi-agent deep reinforcement learning with MADDPG for trajectory control and introduces a low-complexity method for optimizing offloading decisions given the trajectories.The paper reports that its DRL-based algorithm outperforms benchmark algorithms in simulation.
II. SYSTEM MODEL
The system models multiple fixed-altitude UAVs serving ground UEs that execute computational tasks locally or by offloading them over consecutive time slots. It jointly optimizes UAV trajectories, offloading decisions, UE-service fairness, UAV-load fairness, and UE energy consumption.
- System setting: N UEs are randomly distributed over a square area, while M UAVs fly at fixed altitude H across T consecutive time slots.Each UE has one computational task per time slot, and UAVs may be recharged on building roofs.
- Task execution and offloading: Each task is executed either locally or offloaded to one UAV, represented by binary variable z_n,m,t.The index m = 0 denotes local execution, while m > 0 denotes offloading to UAV m.
- UAV mobility: UAV movement is controlled by direction α_m,t and distance d_m,t, subject to area boundaries and a minimum inter-UAV distance R_u.UAV coordinates remain at altitude H, and the minimum separation constraint prevents collisions.
- Communication model: Offloading is possible only when a UE lies within a UAV’s horizontal coverage radius, with data rate determined by bandwidth, UE transmission power, channel gain, and noise power.The model explicitly does not assume a particular modulation and coding scheme.
- Optimization objectives: The optimization minimizes total UE energy while maximizing fairness in UAV UE-load and the number of times UEs are served across time slots.The formulation combines continuous trajectory variables P with discrete offloading variables Z, making traditional approaches challenging.
III. THE PROPOSED ALGORITHM
The proposed algorithm section introduces deep reinforcement learning background and then presents the multi-agent trajectory-control method for solving the formulated optimization problem.
- Algorithm overview: The section first reviews deep reinforcement learning before presenting the MAT algorithm for solving Problem (21).The algorithm is designed for the paper’s multi-agent trajectory-control formulation.
A. Background Knowledge
The section introduces MDP-based reinforcement learning and deep Q-learning, then motivates DDPG for continuous-valued control. It also describes experience replay and target networks for more stable training.
- An MDP models an agent observing a state, taking an allowed action, receiving a reward, and transitioning to a new state.
- DQN uses a neural-network Q-function to approximate the accumulated reward associated with actions, states, and a policy.
- Experience replay randomly samples stored transitions to reduce correlations between consecutive transitions during training.
- Because DQN cannot directly solve continuous-valued control problems, DDPG combines an actor for action selection with a critic for action evaluation.
B. MAT
The proposed multi-agent trajectory-control procedure models UAV coordination as an observable Markov game using MADDPG, with centralized training and decentralized execution. It defines UAV observations, movement actions, collision and boundary penalties, prioritized replay, and a separate low-complexity offloading procedure.
- Multi-agent formulation: MADDPG models the UAV system as an observable Markov game in which each agent acts from private observations within shared state and action sets.Each UAV is controlled by a dedicated agent with actor, critic, target networks, and an experience replay buffer.
- Multi-agent formulation: Centralized training lets agents exchange observations and actions, while execution remains decentralized through each UAV’s dedicated agent.The centralized state contains all agents’ observations and actions, including exchangeable private information such as coordinates.
- State, action, and reward design: Each UAV observes its coordinates, relative distances to other UAVs, cumulative UE service times, and UE-load, then selects flying direction and distance.The reward function includes a penalty when a UAV leaves the target area or violates the minimum collision distance.
- Training procedure: The training procedure uses prioritized experience replay based on absolute TD-error, because high-TD-error transitions can indicate successful attempts.The sampling probability is defined for mini-batch transitions, with β set to 0.6 and µ set to 0.4 in the associated loss and implementation details.
- Offloading optimization: Given UAV trajectories, a low-complexity offloading procedure assigns each UE to the least-energy UAV or keeps execution local.The presented approach omits each UAV’s maximum available computing-resource constraint; adding a matching algorithm is left for future work.
IV. SIMULATION RESULTS
Simulations evaluate MAT against RANDOM and CIRCLE for 3- and 4-UAV settings, showing convergence, cooperative coverage, higher fairness, and lower UE energy consumption.
- Training convergence: MAT’s accumulated reward begins increasing after about 1000 episodes, converges near 300 with 3 UAVs, and saturates around 450 with 4 UAVs.The 3-UAV curve remains under 50 initially, while the 4-UAV curve remains below 200 before increasing.
- UAV trajectories and coverage: With 4 UAVs, the minimum UE-serving occurrences increase from about 2.5 TSs with 3 UAVs to about 6 TSs.The trajectories show UAV cooperation and better coverage as the number of UAVs increases.
- Fairness performance: For 3 UAVs, MAT and CIRCLE approach UE-load fairness of 1, whereas RANDOM reaches only 0.75.MAT and CIRCLE control UAVs to serve similar numbers of UEs.
- Energy consumption: MAT achieves the best UE energy-consumption performance in both UAV settings, followed by CIRCLE and RANDOM.The paper attributes MAT’s lower energy consumption to cooperative UAV service enabling more UEs to offload tasks.
V. CONCLUSIONS
The paper proposes a multi-agent deep reinforcement learning trajectory-control algorithm that jointly optimizes UE fairness, UAV UE-load fairness, and UE energy consumption. Simulations show considerable performance gains over benchmark algorithms.
- The proposed algorithm jointly optimizes fairness among UEs, fairness of each UAV’s UE-load, and UE energy consumption through trajectory and offloading decisions.
- Simulation results show considerable performance gains over the compared benchmark algorithms.