PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Freshness and Energy-Efficient UAV Navigation Optimization: A Deep Reinforcement Learning Approach

Sarder Fakhrul Abedin, Md. Shirajum Munir, Nguyen H. Tran, Zhu Han, Choong Seon Hong

arXiv:2003.04816v1eess.SPcs.LGcs.NIcs.SYeess.SYstat.ML

TL;DR

UAV-BS navigation must account for limited energy and timely IoT data delivery in edge computing, where AoI measures freshness. The paper formulates constrained trajectory optimization and solves it with a deep Q-network using experience replay; the replay-based approach outperforms greedy and baseline DQN in average cumulative reward.

  • Problem

    UAV-BS navigation must address limited onboard energy while delivering fresh IoT data for edge applications, a need measured by AoI.

    Source: §I · p. 1

  • Method

    The paper formulates energy-efficient UAV-BS navigation under energy, navigation, and AoI constraints and uses a DQN with experience replay to learn trajectory policies.

    Source: §I · p. 1, §V · p. 5

  • Results

    The replay-memory DQN outperforms greedy and baseline DQN by up to 2.84% and 1.91%, respectively, in average cumulative reward.

    Source: C. Experiment Results · p. 8

  • Takeaways & Limitations

    The paper reports a strong correlation between energy efficiency and AoI thresholds, with appropriate thresholds enhancing energy efficiency and data freshness for computation-oriented applications.

    Source: §VII · p. 10

Abstract

from arXiv · show

In this paper, we design a navigation policy for multiple unmanned aerial vehicles (UAVs) where mobile base stations (BSs) are deployed to improve the data freshness and connectivity to the Internet of Things (IoT) devices. First, we formulate an energy-efficient trajectory optimization problem in which the objective is to maximize the energy efficiency by optimizing the UAV-BS trajectory policy. We also incorporate different contextual information such as energy and age of information (AoI) constraints to ensure the data freshness at the ground BS. Second, we propose an agile deep reinforcement learning with experience replay model to solve the formulated problem concerning the contextual constraints for the UAV-BS navigation. Moreover, the proposed approach is well-suited for solving the problem, since the state space of the problem is extremely large and finding the best trajectory policy with useful contextual features is too complex for the UAV-BSs. By applying the proposed trained model, an effective real-time trajectory policy for the UAV-BSs captures the observable network states over time. Finally, the simulation results illustrate the proposed approach is 3.6% and 3.13% more energy efficient than those of the greedy and baseline deep Q Network (DQN) approaches.

I. INTRODUCTION

UAV base stations can improve coverage and support demanding 5G applications, but their limited energy and the need for fresh data complicate autonomous navigation. This paper formulates energy- and AoI-constrained navigation and applies deep Q-learning with experience replay.

I. INTRODUCTION

  • UAV-BSs offer agile coverage for demanding 5G services, but limited energy capacity makes deploying and autonomously navigating multiple UAV-BSs challenging.The cited services include enhanced mobile broadband, ultra-reliable low-latency communications, and massive machine-type communications.Source: §I · p. 1

A. UAV Navigation

  • Existing UAV-BS navigation research focuses on path planning and placement with communication and energy constraints, while edge applications also require energy efficiency and timely data delivery.The paper motivates AoI as a measure of data freshness for computation-oriented applications.Source: §I · p. 1

I. INTRODUCTION

  • The paper formulates an energy-efficient UAV-BS navigation problem with energy, navigation, and AoI constraints, and shows that it is NP-hard.Its proposed DQN uses experience replay to learn a real-time trajectory policy from observable network states.Source: §I · p. 1
  • The proposed replay-based DQN navigation policy achieves significant energy efficiency and data freshness compared with baseline approaches.The paper also reports simulation analysis to select a discount factor and AoI threshold for the learning model.Source: §I · p. 2

C. Age of Information (AoI)

The paper designs UAV-BS trajectories to trade off energy efficiency and AoI while meeting trajectory constraints for computation-oriented communications. Its system relays IoT updates to a ground BS-MEC, using separate uplink and backhaul links.

C. Age of Information (AoI)

  • The proposed trajectory policy balances energy efficiency and AoI alongside trajectory constraints, and uses mmWave spectrum for UAV-BS-to-ground-BS backhaul updates.Earlier AoI-focused UAV work considered data acquisition, update scheduling, or expired-packet minimization.Source: C. Age of Information (AoI) · pp. 2–3

III. SYSTEM MODEL

  • Battery-powered UAV-BSs relay IoT updates from trajectory points to a ground BS-MEC that acts as the information fusion center.The BS-MEC receives updates when a UAV-BS is within backhaul transmission range; IoT devices use non-mmWave uplink communication.Source: §III · p. 3

A. IoT-to-UAV-BS Communication Model

  • The IoT-to-UAV-BS model accounts for environment-dependent LoS and NLoS path loss, received signal power, interference, and channel capacity.The non-mmWave uplink bandwidth is equally distributed among IoT devices at a trajectory point, and successful transmission requires SINR above a threshold.Source: A. IoT-to-UAV-BS Communication Model · p. 3

B. UAV-BS-to-BS Communication Model

The backhaul model describes UAV-BS communication with the ground BS, including received power, link capacity, a transmission-range condition, and transmission energy.

B. UAV-BS-to-BS Communication Model

  • The received-power model identifies UAV-BS transmit power, UAV-to-ground-BS distance, carrier frequency, and antenna gains as link quantities.The ground BS is the receiver in the backhaul link.Source: B. UAV-BS-to-BS Communication Model · p. 4
  • The backhaul link uses mmWave bandwidth, and the UAV-BS sends an update when its distance to the ground BS is below a threshold.The passage identifies the link bandwidth and distance condition but does not provide a numerical threshold.Source: B. UAV-BS-to-BS Communication Model · p. 4
  • The paper calculates UAV-BS transmission energy for use of the backhaul link.The cited passage frames this calculation in terms of the backhaul channel model.Source: B. UAV-BS-to-BS Communication Model · p. 4

C. UAV-BS Relay Network Energy Efficiency Metric Design

The paper defines UAV-BS mobility and energy-efficiency quantities for relaying data across trajectory points over an observation period. The modeled UAVs fly horizontally at constant altitude.

C. UAV-BS Relay Network Energy Efficiency Metric Design

  • Each UAV-BS covers the area horizontally at constant altitude, with different UAV-BSs allowed to maintain different altitudes.The paper relates this assumption to regulations for small unmanned aircraft operations.Source: C. UAV-BS Relay Network Energy… · p. 4
  • The model defines a time-varying horizontal travel distance and a mobility energy cost for covering that distance.The stated upper bound on propulsion power depends on UAV-BS design parameters.Source: C. UAV-BS Relay Network Energy… · p. 4
  • The paper defines total energy efficiency for a UAV-BS serving IoT devices across trajectory points over time.The provided passage identifies the metric's scope but does not state its full expression.Source: C. UAV-BS Relay Network Energy… · p. 4

D. Age of Information Model for Ground Station

The paper uses AoI to represent information freshness at the ground BS and formulates UAV-BS navigation as a joint energy-efficiency and freshness-constrained problem. Its decision problem is described as NP-hard.

D. Age of Information Model for Ground Station

IV. PROBLEM FORMULATION

  • The optimization assigns non-overlapping trajectory-point subsets to UAV-BSs and seeks a cooperative configuration maximizing total network energy efficiency subject to energy and AoI constraints.The constraints require minimum energy efficiency and average AoI below a threshold.Source: §IV · pp. 4–5
  • The formulation differs from traditional UAV energy-efficiency maximization by jointly accounting for AoI-dependent freshness requirements, and the authors characterize its decision problem as NP-hard.The paper relates the decision problem to vertex cover / maximum clique.Source: §IV · p. 5

V. PROPOSED TRAJECTORY POLICY ALGORITHM BASED ON DEEP Q-LEARNING

The proposed trajectory policy uses a model-free DQN with experience replay to address the constrained navigation problem. Its state, actions, and reward encode network context and energy efficiency for policy learning.

V. PROPOSED TRAJECTORY POLICY ALGORITHM BASED ON DEEP Q-LEARNING

  • The proposed DQN combines a neural network, experience replay, and reinforcement learning to learn a constrained UAV-BS trajectory policy without a network dynamics model.Replay stores observed transitions and breaks temporal dependence among training observations to promote stability.Source: §V · p. 5

A. State and Action Space

  • The four-dimensional state represents UAV-BS positions, target position, energy efficiency, and average AoI, while actions configure feasible navigation trajectories under the problem constraints.Initial UAV-BS positions and IoT-device counts are randomly assigned in each trial.Source: A. State and Action Space · p. 5

B. Reward and Control Policy

  • The immediate reward is based on energy efficiency and applies constraint-dependent outcomes, while the agent seeks to maximize future reward to form its control policy.The reward definition distinguishes satisfying all constraints from violations of the specified constraint groups.Source: B. Reward and Control Policy · p. 5

1 Step 1: Initialization

The algorithm initializes the Q-network and replay memory, then trains with sampled experience transitions and stochastic gradient descent.

1 Step 1: Initialization

  • Initialization sets Q(s, a; θ), replay memory M, target DQN parameters θ−, and constructs the DQN.The supplied algorithm text identifies these as initialization operations.Source: §1 · p. 6

3 Step 2: Training DQN with experience replay

  • During training, the procedure calculates energy efficiency and an instant reward, then selects an action with probability ϵ.The listed steps identify the action selection probability but do not specify its value.Source: §3 · p. 6
  • Training stores observed transitions in replay memory, samples a minibatch, and applies stochastic gradient descent to train the DQN using its loss function.The listed transitions include states, actions, and rewards.Source: §3 · p. 6

15 Store the Q-network

The algorithm uses Q-values to select joint UAV-BS actions and update trajectories, while recursively updating the Q-function to obtain an optimal control policy.

15 Store the Q-network

  • The algorithm retrieves the joint UAV-BS action that maximizes the Q-function for the current state and updates trajectories using the selected joint action.The stated goal is to obtain the best control policy.Source: §18 · p. 6
  • The discounted cumulative state function is part of the policy-optimization description.The supplied passage names this function but provides no further explanation here.Source: §18 · p. 6
  • The Q-function is updated recursively, with the learning rate identified as ψ.The update uses the next time step and next action.Source: §18 · p. 6

C. Training with Experience Replay and Testing UAV-BS Trajectory Policy

The UAV-BS DQN is trained on trajectory observations using a replay memory and a target network, then its stored policy is used for joint navigation in testing.

C. Training with Experience Replay and Testing UAV-BS Trajectory Policy

  • The Q-network is trained against a target value for each trajectory-observation state, and its training loss is introduced using the weights obtained during training.Source: C. Training with Experience Replay and… · p. 6
  • A fixed-size replay memory reuses randomly sampled episode transitions, improving data efficiency and representing the loss with a uniform distribution over the memory.The target DQN uses stored weights θ−.Source: C. Training with Experience Replay and… · p. 6
  • Training uses energy-efficiency rewards under navigation and AoI constraints, updates the DQN with shuffled replay mini-batches and stochastic gradient descent, and stops at the terminal trajectory.The trained network is stored for testing.Source: C. Training with Experience Replay and… · p. 6
  • The stored trajectory-policy network is used in testing to select actions for joint UAV-BS navigation while updating trajectories and DQN target values.Source: C. Training with Experience Replay and… · p. 6

VI. PERFORMANCE EVALUATION

The evaluation compares the replay-based DQN with baseline DQN and greedy approaches, then examines discount-factor and AoI-threshold choices. The reported parameter analysis finds an energy-efficiency peak at γ = 0.7 and selects an AoI threshold to balance reward and freshness.

A. Experiment Setting

  • The evaluation averages normalized results over 100 episodes and compares the proposed DQN with replay memory against baseline DQN without replay and a greedy policy.The greedy policy chooses paths for maximum immediate reward and penalizes constraint violations.Source: §VI · p. 7, A. Experiment Setting · p. 7

B. Finding Appropriate Discount Factor and AoI Threshold

C. Experiment Results

Across trajectory-waypoint settings, the proposed DQN with replay memory improves energy efficiency, AoI, bandwidth efficiency, and UAV resource utilization over the greedy and baseline DQN approaches. Its reported advantages vary with waypoint density and AoI thresholds.

C. Experiment Results

  • At |P| = 14, the proposed approach outperforms the greedy and baseline DQN in average cumulative reward by 5.08% and 4.02%, respectively.For up to |P| = 8, the approaches' performance is not distinct; the reported gaps increase as waypoint count rises.Source: C. Experiment Results · p. 8
  • At |P| = 14, the proposed approach reduces average AoI by 0.72% and 1.29% compared with the greedy and baseline DQN, respectively.The passage also reports reductions of up to 1.21% and 1.17%, respectively; results fluctuate slightly at |P| = 10.Source: Figure 3 · p. 8, C. Experiment Results · p. 8
  • At |P| = 14, the proposed approach is 3.6% and 3.13% more energy efficient than the greedy and baseline DQN, respectively.At |P| = 10, the corresponding gains are 0.32% and 1.97%.Source: C. Experiment Results · p. 8, Figure 4 · p. 9
  • With |P| = 10, the proposed approach outperforms the greedy and baseline DQN by 2.41% and 2.87%, respectively, in bandwidth efficiency.The passage reports a slight bandwidth-efficiency decrease across approaches as IoT-device count increases in this setting.Source: Figure 5 · p. 9, C. Experiment Results · p. 9
  • The proposed approach uses UAV network resources 9.26% and 4.71% more efficiently than the greedy and baseline DQN, respectively.For a fixed |U| = 3, the passage says IoT devices tend to use the maximum network resource as waypoint count increases.Source: Figure 6 · p. 9, C. Experiment Results · p. 9
  • At AoI threshold b = 0.7, the proposed approach is 1.63% and 0.95% more energy efficient than the greedy and baseline DQN, respectively.The passage attributes the relatively close gap with baseline DQN to the shared discount factor γ = 0.7.Source: Figure 7 · p. 9, C. Experiment Results · p. 9

D. Discussion

The discussion attributes improved trajectory decisions to replaying sampled state transitions and identifies discount factor and AoI threshold settings as important. The conclusion reports a relationship between AoI thresholds, energy efficiency, and data freshness, and says the approach was effective under different network conditions.

D. Discussion

  • Replay memory reuses randomly sampled environment-state transitions, stabilizing and improving DQN training for trajectory policies that consider energy consumption, data freshness, and bandwidth use.The conclusion also describes the model as suited to navigation under trajectory and AoI constraints.Source: D. Discussion · p. 10, §VII · p. 10
  • The UAV-BS objective depends substantially on the discount factor and AoI threshold, and an appropriate discount factor can improve DQN convergence.The discussion relates this observation to DQN training with replay memory.Source: D. Discussion · p. 10
  • In large networks, the proposed DQN with replay memory significantly outperforms the greedy approach, although greedy can perform better in small networks.The authors say the greedy advantage is limited to small networks.Source: D. Discussion · p. 10

VII. CONCLUSION

  • The conclusion reports a strong correlation between energy efficiency and AoI thresholds, with proper threshold values able to enhance energy efficiency and data freshness for COC applications.The authors also report effectiveness under different network conditions.Source: §VII · p. 10
Loading 2003.04816v1…