Source-linked AI summary
Deep Reinforcement Learning for Minimizing Age-of-Information in UAV-assisted Networks
Mohamed A. Abd-Elmagid, Aidin Ferdowsi, Harpreet S. Dhillon, Walid Saad
TL;DR
Real-time applications can depend on the freshness of information collected by UAVs, but prior UAV-assisted AoI work did not jointly optimize flight trajectory and update scheduling. This paper uses deep reinforcement learning to find a policy that jointly optimizes both, and reports lower sum-AoI than distance-based and random-walk baselines, including reductions of approximately 50% and 75% in a studied time-constraint scenario.
Problem
Prior UAV-assisted AoI studies were limited to relay scenarios and did not jointly optimize UAV trajectory and update scheduling, although freshness can constrain QoS for real-time applications.
Source: §I · p. 1Method
The paper formulates joint trajectory and update-scheduling optimization as a finite-horizon MDP and uses deep reinforcement learning to address its extremely large state space.
Source: §I · p. 1Results
The proposed policy significantly outperforms distance-based and random-walk policies in achievable sum-AoI per process; in scenario 4, average sum-AoI is approximately 50% and 75% smaller, respectively.
Source: C. Effects of System Parameters on the… · p. 5, §V · p. 6Takeaways & Limitations
Simulations show that achievable sum-AoI varies with energy availability, time constraint, and ground-node spacing, with larger node spacing increasing average sum-AoI per process.
Source: C. Effects of System Parameters on the… · pp. 5–6
Abstract
from arXiv · showhide
Unmanned aerial vehicles (UAVs) are expected to be a key component of the next-generation wireless systems. Due to their deployment flexibility, UAVs are being considered as an efficient solution for collecting information data from ground nodes and transmitting it wirelessly to the network. In this paper, a UAV-assisted wireless network is studied, in which energy-constrained ground nodes are deployed to observe different physical processes. In this network, a UAV that has a time constraint for its operation due to its limited battery, moves towards the ground nodes to receive status update packets about their observed processes. The flight trajectory of the UAV and scheduling of status update packets are jointly optimized with the objective of achieving the minimum weighted sum for the age-of-information (AoI) values of different processes at the UAV, referred to as weighted sum-AoI. The problem is modeled as a finite-horizon Markov decision process (MDP) with finite state and action spaces. Since the state space is extremely large, a deep reinforcement learning (RL) algorithm is proposed to obtain the optimal policy that minimizes the weighted sum-AoI, referred to as the age-optimal policy. Several simulation scenarios are considered to showcase the convergence of the proposed deep RL algorithm. Moreover, the results also demonstrate that the proposed deep RL approach can significantly improve the achievable sum-AoI per process compared to the baseline policies, such as the distance-based and random walk policies. The impact of various system design parameters on the optimal achievable sum-AoI per process is also shown through extensive simulations.
I. INTRODUCTION
The paper targets information freshness in UAV-assisted networks, where real-time application QoS can depend on the freshness of collected updates. It jointly optimizes the UAV’s trajectory and update scheduling using deep reinforcement learning to minimize weighted sum-AoI.
I. INTRODUCTION
- Freshness of information collected by the UAV constrains QoS for real-time applications such as human safety applications.The paper motivates designing both flight trajectory and transmission scheduling to preserve freshness.Source: §I · p. 1
Related works
- Prior UAV-assisted AoI studies considered relay scenarios and did not jointly optimize trajectory and update scheduling across ground nodes.The paper identifies this as a limitation of the cited prior work and.Source: §I · p. 1
Contributions.
- The paper jointly optimizes UAV flight trajectory and ground-node update scheduling to minimize weighted sum-AoI.It models the problem as a finite-horizon MDP and proposes deep RL because the state space makes standard dynamic programming computationally impractical.Source: §I · p. 1
A. Network Model
The network comprises ground nodes observing physical processes and a UAV that collects their status updates while operating within battery and time constraints. The model discretizes the UAV’s flight region and represents ground-node batteries in energy quanta.
A. Network Model
- Ground nodes observe potentially different physical processes, while the UAV collects their status updates during its operation.The network uses uplink transmissions from nodes to the UAV.Source: A. Network Model · pp. 1–2
- Each ground node has a finite-capacity battery divided into energy quanta, and time is modeled in unit-length slots.The battery level is represented at the beginning of each slot.Source: A. Network Model · p. 2
- The UAV flies at fixed height over a region partitioned into equal-area cells, with its position mapped to a cell each time slot.Its operation is limited to a finite time interval between initial and final locations.Source: A. Network Model · p. 2
- The channel model assumes line-of-sight links between the UAV and ground nodes.The channel power gain depends on the UAV-node distance and a reference-distance channel gain.Source: A. Network Model · p. 2
B. State and Action Spaces
The system state tracks each ground node’s battery and process AoI together with the UAV’s location and remaining time. At each slot, the system chooses both a movement action and an update-scheduling action, subject to packet-energy requirements.
B. State and Action Spaces
- Each ground-node state comprises its battery level and the AoI of its observed process, while the UAV state includes its location and remaining time.Together these components form the system state.Source: B. State and Action Spaces · p. 2
- The UAV can stay in place or move to an adjacent cell, and the scheduling action selects a ground node or no transmission.Movement beyond the region boundary leaves the UAV in its current cell.Source: B. State and Action Spaces · p. 2
- A scheduled node must have enough battery energy to transmit, and its process AoI resets to one; otherwise, AoI increases by one up to its maximum.The packet’s required energy is determined from its size, bandwidth, and noise power.Source: B. State and Action Spaces · pp. 2–3
A. Problem Formulation
The objective is to find a finite-horizon policy that minimizes expected weighted sum-AoI. The paper models the system as a finite-state MDP, but its large state space makes standard dynamic programming infeasible.
A. Problem Formulation
- The age-optimal policy minimizes expected total weighted AoI cost over the finite horizon.The policy specifies action probabilities conditioned on the state history.Source: A. Problem Formulation · p. 3
- For identical ground nodes under the theorem’s stated conditions, minimum total expected cost is attained by updating a node with the maximum current AoI at every slot.The theorem assumes equal maximum AoI values and initial AoI of one for every node.Source: A. Problem Formulation · p. 3
- The system is a finite-horizon MDP with finite state and action spaces, but its extremely large state space makes standard finite-horizon dynamic programming computationally infeasible.The paper therefore proposes deep reinforcement learning to solve the optimization problem.Source: A. Problem Formulation · p. 3
B. Deep Reinforcement Learning Algorithm
The algorithm uses Q-learning to minimize expected weighted sum-AoI, with a deep Q network approximating the state-action value function because the state space is too large for iterative tabular updates. An ANN, replay memory, and ε-greedy exploration support learning across episodes.
B. Deep Reinforcement Learning Algorithm
- Q-learning estimates expected system cost from a state-action pair and uses a Bellman update to seek the policy minimizing future cost.The instantaneous cost is the weighted sum of process AoI values; the episodic task uses γ = 1.Source: B. Deep Reinforcement Learning Algorithm · pp. 3–4
- The UAV selects a random action with probability ε and can reduce ε toward zero as learning proceeds.Otherwise it selects the action with the lowest estimated Q-value.Source: B. Deep Reinforcement Learning Algorithm · p. 4
- The deep Q network approximates the Q-function with an ANN because the extremely large state space makes iterative updates impractical and unable to generalize to unvisited states.A fully connected layer extracts an abstraction of the state space.Source: B. Deep Reinforcement Learning Algorithm · p. 4
- After each episode, the algorithm samples past state, action, cost, and next-state experiences from replay memory to update ANN weights.It trains on a batch of b experiences using the gradient of the loss function.Source: B. Deep Reinforcement Learning Algorithm · p. 4
- The algorithm repeats action selection, cost and next-state observation, experience storage, and network training until convergence to a sum-AoI.The network and replay memory are initialized before episodic learning begins.Source: B. Deep Reinforcement Learning Algorithm · p. 4
IV. SIMULATION RESULTS
The simulations use a discretized 100-meter grid over an approximately 1.1-kilometer square area. Reported sum-AoI per process values are averaged over 1,000 episodes.
IV. SIMULATION RESULTS
- The simulated region spans coordinates from −50 to 1,050 meters on each axis and is discretized into 11-by-11 cells of 100 meters per side.Cell indices correspond to cell-center coordinates divided by 100.Source: §IV · p. 4
- The simulations use B = 1 MHz bandwidth, 20 Mbits packet size, −100 dBm noise power, and a UAV height of 100 meters.The passage also specifies 1 mJ energy quanta and a maximum AoI of 50.Source: §IV · p. 4
- Reported sum-AoI per process values are averages over 1,000 episodes, using a one-layer ANN with 200 hidden nodes for training.Training used Tensorflow-Agents and a single NVIDIA P100 GPU.Source: §IV · p. 4
A. Convergence Analysis
In scenario 1, the study examines convergence for a single ground node at varying positions and reports that average sum-AoI per process is lower for nodes closer to the direct UAV route. The authors attribute this pattern to limited energy for updating distant nodes.
A. Convergence Analysis
- Scenario 1 sets the direct trip between initial and final cells to 10 time slots and uses emax = 26 for transmission from the furthest node at (5, 10).The candidate node positions range from (5, 5) to (5, 10).Source: A. Convergence Analysis · p. 4
- Average sum-AoI per process is smaller for ground nodes closer to the straight line between the initial and final cells.The reported convergence is after 50,000 training episodes.Source: A. Convergence Analysis · pp. 4–5, Figure 4 · p. 5
- The authors attribute the proximity pattern to the UAV's straight-line travel and insufficient energy to update far-away nodes, while closer nodes can be updated several times.The scenario uses one ground node at one of cells (5, 5) through (5, 10), with τ = 10 and emax = 26.Source: A. Convergence Analysis · pp. 4–5
B. Trajectory Optimization
In scenario 2, the UAV has two ground nodes and enough time for a three-slot detour to update them before returning to its straight route. The deep RL algorithm finds the best path and scheduling strategy in this setup.
B. Trajectory Optimization
- In scenario 2, two ground nodes are located at (2, 10) and (8, 10), and the UAV can receive only one packet from either node.The setup chooses emax and τ to permit reception when the UAV is as close as possible to a node.Source: B. Trajectory Optimization · p. 5
- With τ = 16, the UAV has six extra slots beyond the direct route, enough for a three-slot northward detour, updates, and a return.The setup uses emax = 5, the energy required to transmit a packet when the UAV is two cells away.Source: B. Trajectory Optimization · p. 5
- The proposed deep RL algorithm optimally finds the path and scheduling strategy in the described scenario 2 setup.Cross markers identify the UAV locations where it receives packets from the indexed ground nodes.Source: B. Trajectory Optimization · p. 5
C. Effects of System Parameters on the Minimum Sum-AoI
Across energy, time, and node-density scenarios, the deep RL policy achieves lower average sum-AoI per process than the distance-based and random-walk baselines. The baselines’ relative performance depends on the available energy and time, while lower spatial density raises average sum-AoI.
C. Effects of System Parameters on the Minimum Sum-AoI
- Higher maximum energy quanta lowers average sum-AoI; distance-based policy performance stays constant after emax = 25, while the stronger baseline varies with energy.Distance-based policy is more effective than random walk at smaller emax, while random walk is more effective at larger emax.Source: C. Effects of System Parameters on the… · p. 5, Figure 5 · p. 6
- The deep RL policy achieves approximately 50% and 75% smaller average sum-AoI than the distance-based and random-walk policies, respectively, as UAV time constraint varies.For time constraints below 50, random walk is more effective than the distance-based policy; at larger constraints, the distance-based policy can outperform random walk.Source: C. Effects of System Parameters on the… · p. 5, Figure 5 · p. 6
- Lower spatial density of ground nodes increases average sum-AoI, and deep RL achieves lower average sum-AoI than the baseline policies.The authors attribute the increase to greater node separation: limited UAV time means updates arrive from farther away and less frequently.Source: C. Effects of System Parameters on the… · p. 6, Figure 5 · p. 6
V. CONCLUSION
The paper develops a deep RL algorithm to jointly optimize UAV trajectory and status-update scheduling for minimizing weighted sum-AoI. It reports that the policy significantly outperforms distance-based and random-walk baselines in achievable sum-AoI per process.
V. CONCLUSION
- The deep RL algorithm characterizes an age-optimal policy that jointly optimizes UAV trajectory and update scheduling, significantly outperforming distance-based and random-walk baselines in achievable sum-AoI per process.The algorithm addresses the curse of dimensionality of the original MDP; simulations report sum-AoI improving with UAV time constraint and ground-node spatial density, and decreasing with node battery sizes.Source: §V · p. 6