PaperlayerView exact revision on arXiv

Source-linked AI summary

Path Planning for UAV-Mounted Mobile Edge Computing with Deep Reinforcement Learning

Q. Liu, L. Shi, L. Sun, J. Li, M. Ding, F. Shu

arXiv:2001.10268v1cs.ITeess.SP

TL;DR

The paper studies trajectory planning for an energy-limited UAV that serves mobile TUs while maintaining their QoS. It formulates trajectory and association decisions as an MDP and uses a QoS-based policy based on DDQN. The proposed method converges faster, achieves higher sum throughput than conventional algorithms, and provides an almost 99% QoS guarantee rate per TU.

  • Problem

    Mobile TU locations make offline UAV trajectory designs unsuitable for dynamically maintaining QoS in UAV-mounted MEC networks.

  • Method

    The paper formulates trajectory and UAV-TU association optimization as an MDP and develops a QoS-based ǫ-greedy policy based on DDQN.

  • Results

    Almost 99% guarantee rate in each TU’s QoS is achieved, while the proposed algorithm outperforms DQN, DQL, and QL in convergence and sum throughput.

  • Takeaways & Limitations

    The proposed QoS-based DDQN approach supports dynamic UAV service to mobile TUs under energy and QoS constraints.

Abstract

from arXiv · show

In this letter, we study an unmanned aerial vehicle (UAV)-mounted mobile edge computing network, where the UAV executes computational tasks offloaded from mobile terminal users (TUs) and the motion of each TU follows a Gauss-Markov random model. To ensure the quality-of-service (QoS) of each TU, the UAV with limited energy dynamically plans its trajectory according to the locations of mobile TUs. Towards this end, we formulate the problem as a Markov decision process, wherein the UAV trajectory and UAV-TU association are modeled as the parameters to be optimized. To maximize the system reward and meet the QoS constraint, we develop a QoS-based action selection policy in the proposed algorithm based on double deep Q-network. Simulations show that the proposed algorithm converges more quickly and achieves a higher sum throughput than conventional algorithms.

I. INTRODUCTION

The paper addresses online trajectory planning for an energy-limited UAV serving mobile TUs whose locations vary over time. It formulates the problem for DRL-based optimization with QoS-aware action selection and reports faster convergence, higher throughput, and near-99% QoS guarantee.

  • Motivation: MEC uses UAVs as mobile edge servers to provide flexible computing services to mobile terminal users.The UAV offers reliable connectivity with affordable infrastructure investment.
  • Research problem: Offline trajectory designs assume invariant TU locations, whereas mobile TUs require online adaptation to preserve each TU’s QoS.The paper identifies trajectory planning for mobile TUs as challenging.
  • Research problem: DRL is used because time-varying TU locations create huge state and action spaces that make conventional reinforcement learning more difficult.The changing locations lead to innumerable MDP states.
  • Approach: The proposed algorithm models UAV trajectory optimization as an MDP under Gauss-Markov TU motion, limited UAV energy, and per-TU QoS constraints.UAV-TU association is also part of the modeled service setting.
  • Results: 99% guarantee rate in each TU’s QoS is achieved by the QoS-based ǫ-greedy policy, while the algorithm outperforms conventional methods in convergence and throughput.The reported comparison is against conventional RL and DQN algorithms.

A. Movement Model of TUs

The TU movement model initializes users randomly and updates their velocities, directions, and locations over discrete time slots using a Gauss-Markov random model. Shared average speed, user-specific average directions, and Gaussian randomness characterize the motion.

  • Movement Model of TUs: TU locations are randomly initialized, then treated as unchanged during each interval between consecutive time slots.The movement update begins for t ≥ 1.
  • Movement Model of TUs: The Gauss-Markov model updates each TU’s velocity and direction while retaining adjustable effects from its previous state.The parameters κ1 and κ2 control the previous-state influence.
  • Movement Model of TUs: All TUs share the same average speed, while different TUs have distinct average directions.The average direction is indexed by TU.
  • Movement Model of TUs: Independent Gaussian variables represent randomness in different TUs’ movements through their velocity and direction updates.The distributions have different mean-variance pairs.
  • Movement Model of TUs: Each TU’s location is updated from its modeled velocity and direction at the corresponding time slot.The UAV location is separately specified at each FPAP and time slot.

B. Energy Consumption of UAV

The UAV’s energy use combines flying, hovering, and computing consumption. These components account for motion, wireless task offloading, and execution of the offloaded tasks over the operating slots.

  • Energy Consumption of UAV: UAV energy consumption is divided into flying, hovering, and computing categories.The total per-slot consumption is the sum of these three components.
  • Flying Energy Consumption: Flying energy depends on constant UAV speed and flying power, with flying power comprising parasitic and induced power.The components overcome parasitic drag and lift-induced drag.
  • Hovering Energy Consumption: Hovering energy is associated with uploading tasks from the selected TU through the line-of-sight UAV channel.The uploading rate depends on transmission power, noise, channel gain, and UAV altitude.
  • Hovering Energy Consumption: Hovering energy uses the offloaded task amount and bits per task together with the UAV hovering power.The task amount is denoted by µn(t), and the bits-per-task quantity by Nb.
  • Computing Energy Consumption: Computing energy is proportional to offloaded bits and depends on switched capacitance, CPU cycles per bit, and CPU frequency.The paper gives ec(t) = γcC(fc)2µn(t)Nb.

III. MDP MODELING AND PROBLEM FORMULATION

Because TU locations and UAV energy have Markov characteristics, the paper formulates UAV trajectory optimization as an MDP. The objective is long-term system reward maximization under UAV energy and TU QoS constraints.

  • MDP Modeling and Problem Formulation: Markov characteristics of TU locations and UAV energy justify modeling UAV trajectory optimization as an MDP.The formulation represents the evolving system state over time.
  • MDP Modeling and Problem Formulation: The optimization maximizes long-term system reward subject to the UAV energy constraint and the QoS constraint of the TUs.Both resource limitation and service quality are included in the formulation.

A. State, Action, and Reward

The UAV’s state and action design represents time-varying TU service and FPAP selection, while a sigmoidal utility shapes rewards from offloaded tasks.

  • Action: The action indices range over N TUs, M FPAPs, and T time slots.The supplied formulation specifies n ∈ {1, 2, ..., N}, m ∈ {1, 2, ..., M}, and t ∈ {0, 1, ..., T −1}.
  • Action: The UAV chooses one TU to serve and one FPAP to fly to in each time slot.These two decisions constitute the system’s action space.
  • Reward: System utility depends nonlinearly on the number of offloaded tasks µ_n(t) rather than increasing linearly.The paper adopts a sigmoidallike function with η and β adjusting the efficiency of U(µ_n(t)).
  • Reward: The utility rises steeply with µ_n(t) and then becomes steady when the task volume is sufficiently large.This reward shape discourages serving one TU for too long while ignoring other TUs.

B. Problem Formulation

The problem is formulated as finding an optimal policy that maximizes average long-term system reward while respecting UAV energy and TU QoS constraints.

  • Objective: The optimal policy π* is the policy that contributes to the maximal long-term system reward.The objective is to find π* that maximizes the average long-term system reward.
  • Constraints: The UAV’s total energy consumption over T time slots cannot exceed its battery capacity.This is the first stated constraint in the problem formulation.
  • Constraints: The QoS constraint guarantees at least Z offloaded tasks from each TU over T time slots.This is the second stated constraint in the formulation.

IV. PROPOSED ALGORITHM

The proposed algorithm uses reinforcement learning with DDQN and a QoS-based ε-greedy policy to maximize long-term rewards while addressing large state-action spaces and QoS requirements.

  • IV. PROPOSED ALGORITHM: The UAV explores an unknown environment by trying actions, learning from feedback, and reinforcing actions that deliver better results.The learning objective is to maximize long-term system rewards.
  • IV. PROPOSED ALGORITHM: Fig. 2 depicts the DDQN structure with the QoS-based ε-greedy policy.The figure identifies the architecture and action-selection component used by the proposed algorithm.
  • IV. PROPOSED ALGORITHM: DDQN addresses DQN overestimation and the massive state-action space caused by time-varying TU locations.The method is chosen instead of a conventional reinforcement-learning algorithm for these settings.
  • IV. PROPOSED ALGORITHM: The proposed algorithm adds a QoS-based ε-greedy policy to further meet the QoS constraint.The policy modifies action selection within the DDQN-based approach.

A. Deep Q-Network (DQN)

The DQN component evaluates actions with neural-network approximations of Q-values using predicted and target networks.

  • A. Deep Q-Network (DQN): Q(s_t, a_t) evaluates how good action a_t is when performed in state s_t.The Q-value is used to assess action quality in the reinforcement-learning procedure.
  • A. Deep Q-Network (DQN): DQN approximates Q-values with two deep neural networks having the same four fully connected layers but different parameters.The two networks are a predicted network and a target network.
  • A. Deep Q-Network (DQN): The predicted network takes the current state-action pair (s_t, a_t) as input and outputs the predicted Q-value.The passage denotes this output as Q_DQN predicted(s_t, a_t; φ_1).
  • A. Deep Q-Network (DQN): The target network takes the next state s_t+1 as input and outputs the maximum Q-value among next-state action pairs.The resulting output is used to form the target value for (s_t, a_t).
  • A. Deep Q-Network (DQN): a′ denotes the candidate next action in the DQN update.This symbol identifies an action considered for the next state.

B. DDQN with Proposed QoS-Based ǫ-greedy Policy

The proposed DDQN combines two Q-value estimators with a QoS-based ǫ-greedy policy to plan UAV actions under mobile-TU constraints. Training proceeds through replayed transitions, while action selection prioritizes TUs that currently violate QoS requirements.

  • DDQN structure: DDQN uses separate predicted and target networks to address DQN’s Q-value overestimation problem in environments with massive state-action spaces.The predicted network selects the next-state action, while the target network evaluates it; the target network is updated after a fixed interval.
  • DDQN structure: The UAV selects an action from the current state, receives a reward, transitions to the next state, and stores the transition in replay memory.When memory is full, K samples are randomly extracted for network training.
  • Training procedure: The predicted network is updated by gradient descent, the target network periodically copies the predicted-network parameters, and ǫ decreases by δ during training.An episode ends when the UAV’s remaining energy-related quantity b(t) is no longer positive.
  • QoS-based ǫ-greedy policy: When all TUs satisfy QoS, the UAV uses conventional ǫ-greedy action selection; otherwise, it restricts selection toward actions associated with QoS-violating TUs.Candidate actions associated with non-violating TUs are discarded until an action serving a QoS-violating TU is selected.
  • Training procedure: Training is performed offline to obtain the optimal policy π*, which then directs the UAV during online testing to maximize long-term system reward.The algorithm initializes two network parameter sets and iterates over episodes while the UAV has available energy.

V. SIMULATIONS AND RESULTS

Simulations evaluate convergence, throughput, robustness, trajectory planning, and QoS satisfaction. The proposed algorithm converges fastest, achieves the highest throughput across tested conditions, remains robust to speed variation, and improves QoS satisfaction over conventional ϵ-greedy selection.

  • Convergence and reward: The proposed algorithm achieves the largest convergence rate and average reward among the proposed algorithm, DQN, DQL, and QL comparisons.Lower UAV altitude or fewer TUs produces larger average reward; higher altitude increases path loss and hovering energy.
  • Convergence and reward: When Ne > 8000, QL and DQL are hardly implemented because constructing Q-tables with massive state-action spaces is unaffordable.
  • Throughput and robustness: The proposed algorithm achieves the largest sum throughput per episode for every tested average TU speed, while throughput decreases as the number of TUs increases.Throughput increases as average TU speed decreases; both the proposed algorithm and DQN achieve their largest throughput at v̄ = 1m/s.
  • Throughput and robustness: The proposed algorithm converges under average TU speeds of 5, 8, 15, and 20m/s despite DNN training only at v̄ = 1m/s.
  • QoS satisfaction: The QoS-based ϵ-greedy policy significantly outperforms conventional ϵ-greedy policy in QoS satisfaction over 100000 episodes for 15 TUs.

VI. CONCLUSIONS

The proposed DDQN-based QoS-aware policy optimizes UAV trajectories for dynamically serving mobile TUs under QoS and long-term reward objectives. It outperforms DQN, DQL, and QL in convergence and sum throughput while achieving an almost 99% QoS guarantee rate for each TU.

  • The UAV trajectory optimization is formulated as an MDP for dynamically serving mobile TUs whose motion follows the GMRM.
  • The QoS-based ǫ-greedy policy is developed with DDQN to maximize long-term system reward while meeting each TU’s QoS constraint.
  • Almost 99% guarantee rate is achieved for the QoS of each TU.
Loading 2001.10268v1…