Source-linked AI summary
Age of Information-Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective
Xianfu Chen, Celimuge Wu, Tao Chen, Honggang Zhang, Zhi Liu, Yan Zhang, Mehdi Bennis
TL;DR
The paper studies long-term AoI-aware RRM for a mobile Manhattan-grid V2V network with difficult partial observability and high-dimensional local states. It decomposes the MDP and uses LSTM-based deep reinforcement learning for decentralized allocation and scheduling, with experiments showing significant gains over four baselines.
Problem
Long-term AoI-aware RRM is challenging because vehicle mobility and changing traffic information create partial observability and high-dimensional local network states.
Method
The paper linearly decomposes the single-agent MDP and proposes an LSTM- and DQN-based proactive algorithm for decentralized frequency-band allocation and packet scheduling.
Results
Numerical experiments show significant gains in average utility performance over four state-of-the-art baselines.
Takeaways & Limitations
The proposed approach supports frequency-band allocation and packet scheduling using only local partial observations and no a priori network-dynamics statistics.
Abstract
from arXiv · showhide
In this paper, we investigate the problem of age of information (AoI)-aware radio resource management for expected long-term performance optimization in a Manhattan grid vehicle-to-vehicle network. With the observation of global network state at each scheduling slot, the roadside unit (RSU) allocates the frequency bands and schedules packet transmissions for all vehicle user equipment-pairs (VUE-pairs). We model the stochastic decision-making procedure as a discrete-time single-agent Markov decision process (MDP). The technical challenges in solving the optimal control policy originate from high spatial mobility and temporally varying traffic information arrivals of the VUE-pairs. To make the problem solving tractable, we first decompose the original MDP into a series of per-VUE-pair MDPs. Then we propose a proactive algorithm based on long short-term memory and deep reinforcement learning techniques to address the partial observability and the curse of high dimensionality in local network state space faced by each VUE-pair. With the proposed algorithm, the RSU makes the optimal frequency band allocation and packet scheduling decision at each scheduling slot in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical experiments validate the theoretical analysis and demonstrate the significant performance improvements from the proposed algorithm.
I. INTRODUCTION
The paper addresses long-term AoI-aware radio resource management in a mobile Manhattan-grid V2V network, where changing channel and traffic conditions complicate centralized control. It decomposes the problem and combines LSTM with deep reinforcement learning to enable decentralized decisions from partial observations.
- Most existing RRM efforts optimize instantaneous performance while overlooking temporal and spatial variations in communication quality and traffic information.
- Vehicle mobility affects channel quality and creates opportunities for frequency sharing among different VUE-pair groups, making efficient RRM challenging.
- The paper formulates AoI-aware RRM in a Manhattan-grid V2V network as a single-agent MDP optimizing expected long-term performance for all VUE-pairs.
- Because joint frequency-band allocation and packet-scheduling decisions grow exponentially with the number of VUE-pairs, the MDP is linearly decomposed into per-VUE-pair decision processes.
- An LSTM- and DRL-based proactive algorithm addresses partial observability and high-dimensional local state spaces, using centralized offline training and decentralized online testing.
- Numerical experiments report that the proposed algorithm outperforms four state-of-the-art baseline algorithms in average utility performance.
II. SYSTEM DESCRIPTIONS
The system models Manhattan-grid V2V communication under vehicle mobility, channel-state variation, traffic arrivals, and shared frequency bands. VUE-pair locations and channel conditions determine communication behavior within a single RSU-covered area.
- System scope: The RSU serves the covered roads and coordinates VUE-pair communications through the shared resource model.The system description includes network, channel, and notation components for the subsequent resource-allocation problem.
- Network model: The network contains K VUE-pairs sharing B orthogonal frequency bands over equal-duration scheduling slots.The coverage area is represented in a two-dimensional Euclidean space and is served by one RSU.
- Mobility model: Each VUE-pair consists of a transmitter and receiver whose mobility follows a Manhattan model while maintaining a fixed separation.The transmitter and receiver positions are represented by geographical coordinates.
- Channel model: The channel state varies with whether the transmitter and receiver occupy the same or perpendicular lanes relative to an intersection.The three channel categories are LOS, WLOS, and NLOS, with fast fading included in the channel state.
- Channel model: The model uses a 5.9 GHz urban-area channel with finite channel-state space and path-loss parameters constrained by the intersection distance.The fast-fading component is averaged out in the analysis.
B. AoI Evolution
Traffic updates arrive at slot boundaries, packets must be delivered within their arrival slot, and AoI tracks the elapsed time since the latest successful transmission. Frequency allocation and transmission decisions determine packet delivery and freshness evolution.
- Transmission model: The RSU allocates frequency bands and schedules transmissions, while transmit power and received interference determine achievable delivery.The model includes bandwidth, background noise, interference, packet size, and maximum transmit power.
- Frequency allocation: Each VUE-pair can receive at most one frequency band per scheduling slot, represented by a binary allocation indicator.The frequency allocation vector records which band is assigned to a VUE-pair.
- Traffic arrivals: Traffic information updates arrive at each VUE-pair transmitter only at the beginning of a scheduling slot.Packets associated with arrivals are independently distributed across VUE-pairs and identically distributed across slots with mean rate λ.
- Packet delivery: Packets arriving in a slot must be delivered by that slot’s end; otherwise, they are dropped.The number of packet arrivals is denoted by X_j_k, and dropped packets are tracked after deadline expiry.
- AoI evolution: AoI is the time elapsed since the most recently successful packet transmission and evolves with the scheduling interval τ.When packets are delivered during a slot, the next-slot AoI is reset to τ; otherwise, it increases by τ.
C. VUE-pair Clustering
The paper clusters geographically related VUE-pairs to reduce mutual interference and enable frequency reuse across groups. Spectral clustering produces disjoint groups whose members share exclusive band allocations within each group.
- VUE-pair clustering: The RSU clusters VUE-pairs into geographically separated groups to alleviate mutual interference and improve frequency-band utilization.The grouping is based on the VUE-pairs’ geographical locations.
- Frequency reuse: Frequency bands are exclusively allocated among VUE-pairs within a group and reused by transmitters in different groups.This grouping pattern adds a constraint to band allocation during each scheduling slot.
- Clustering procedure: Spectral clustering is applied to construct G disjoint groups covering all VUE-pairs.The algorithm clusters normalized row vectors derived from the similarity representation.
- Clustering procedure: A distance-based Gaussian similarity measure controls neighborhood influence through a neighborhood-size parameter ζ.The resulting similarity structure is used to form the VUE-pair groups.
- Problem formulation: The clustered formulation rewrites transmit-power consumption by approximating interference from VUE-pairs in other groups.The paper then formulates the AoI-aware resource-management problem as a discrete-time single-agent MDP.
A. AoI-Aware RRM
The AoI-aware resource-management problem is formulated as an infinite-horizon discounted single-agent MDP in which the RSU observes global state and jointly allocates bands and schedules packets. The objective aggregates utility across VUE-pairs while accounting for power, packet drops, and AoI.
- MDP formulation: The global state comprises local VUE-pair states together with the states of all other VUE-pairs.The local state includes quantities associated with packet arrivals and AoI.
- Control policy: At each slot, the RSU observes the global state and selects frequency-band allocation and packet-scheduling actions.The control policy maps the observed state to the corresponding allocation and scheduling decisions.
- Utility: The immediate utility measures QoE using transmit power consumption, packet drops, and AoI with non-negative weighting constants.The exponential utility is chosen to stabilize offline training and avoid dramatic divergence.
- MDP formulation: Vehicle mobility, packet arrivals, and AoI evolution make the global network-state sequence Markovian under the control policy.The resulting objective uses an infinite-horizon discounted criterion with discount factor γ ∈ [0, 1).
- Optimization objective: The RSU optimizes expected long-term discounted utility, accumulated from immediate utility across all VUE-pairs.As γ approaches 1, the discounted objective also approximates expected long-term undiscounted utility.
- Optimization objective: The AoI-aware RRM problem is formally posed as a single-agent MDP subject to the frequency-allocation and scheduling constraints.The formulation defines both per-VUE-pair value functions and a global state value function.
B. General Solution
The paper models AoI-aware radio resource management as an infinite-horizon discounted single-agent MDP, then develops a tractable decentralized solution for its large state and decision spaces.
- The AoI-aware RRM problem is formulated as a discrete-time single-agent MDP with a discounted criterion.
- Conventional dynamic programming is practically challenging because it requires complete knowledge of the network dynamics.
- The global network state space is extremely large, while the number of RSU decision combinations grows exponentially with the number of VUE-pairs.
- The solution decomposes the original MDP into per-VUE-pair MDPs solved through decentralized SARSA, followed by an LSTM- and DRL-based proactive algorithm.
- The proactive algorithm uses centralized offline training at the RSU and decentralized online testing at the VUE-pairs.
A. Linear Q-function Decomposition
Linear Q-function decomposition separates the centralized control problem into per-VUE-pair components, simplifying decisions while preserving optimal expected long-term performance under the stated assumptions.
- The RSU combines per-VUE-pair Q-function values to allocate frequency bands and schedule packet transmissions under the global network state.
- The decomposition reduces the RSU’s decision-making burden by shifting local frequency-allocation and packet-scheduling choices to VUE-pairs.
- Theorem 1 states that linear Q-function decomposition guarantees the optimal control policy and optimal expected long-term performance for all VUE-pairs.
- The decentralized learning process converges to the per-VUE-pair Q-functions when every state-action pair is visited infinitely often.
- The convergence argument assumes finite global-state and decision spaces, bounded per-VUE-pair Q-functions, and exploration through an ε-greedy policy.
B. Proactive DRL for Optimal Control Policy
The proactive DRL design addresses partial observability and large local observation spaces by using recurrent Q-networks trained centrally and deployed for decentralized control.
- During online control, the trained DRQN outputs frequency-band and packet-scheduling decisions for all VUE-pairs while satisfying the stated constraints.
- Local partial observations leave the decentralized Q-function state space huge, making tabular SARSA impractical.
- The method replaces the first fully connected DQN layer with an LSTM, producing a deep recurrent Q-network for partial observability.
- The DRQN takes the N most recent local partial observations as input to predict the current global network state more precisely and proactively.
- The RSU trains the DRQN offline using replay memory, observation pools, mini-batches, and target-network updates.
V. NUMERICAL EXPERIMENTS
The numerical experiments use TensorFlow to validate the theoretical analysis and evaluate the proposed proactive algorithm under online decentralized control.
- The experiments validate the theoretical studies for AoI-aware RRM in V2V communications.
- They evaluate the performance achieved by the proposed proactive algorithm.
- The DRQN is first trained offline at the RSU, after which online frequency-band allocation and packet-scheduling decisions follow the specified policy across scheduling slots.
A. General Setups
The experiments use a Manhattan-grid V2V network with Poisson packet arrivals and specified vehicle, distance, and neural-network settings. Performance is compared against four channel-, packet-, AoI-, and random-aware baselines.
- The network model uses a Manhattan grid with nine intersections, 250 × 250 m2 area, two lanes per road, and 60 km/h average vehicle speed.
- Update arrivals at each VUE-pair follow a Poisson process.
- The DRQN uses an LSTM layer followed by two fully connected layers, with 32 neurons in each layer, ReLU activation, and Adam optimization.
- Four baselines are considered: Channel-Aware, Packet-Aware, AoI-Aware, and Random.
- The proposed implementation centralizes frequency-band allocation at the RSU before VUE-pairs transmit their feasible packet counts.
B. Experiment Results
Experiments evaluate offline-training convergence and average power, packet drops, AoI, and utility under changing band counts and VUE-pair distances. The proposed algorithm achieves the best utility trade-off among the compared algorithms.
- 1) Efficiency of Offline Training: Offline training converges within 1.2·10^4 scheduling slots, while the slow convergence makes the training costly and motivates performing it at the RSU.
- 2) Performance with Changing Number of Bands: As frequency bands increase, transmit power consumption rises, while packet drops and AoI decrease.
- 2) Performance with Changing Number of Bands: Average utility improves for all algorithms as the number of frequency bands increases because packet drops and AoI jointly dominate the utility function.
- The proposed algorithm outperforms all baselines in average utility, indicating a better trade-off among transmit power, packet drops, and AoI.
- 3) Performance with Changing VUE-Pair Distance: Increasing VUE-pair distance worsens channel quality, increases power requirements, and produces lower average utility.
- 3) Performance with Changing VUE-Pair Distance: For Packet-Aware and AoI-Aware baselines, transmit power first increases and then decreases as distance grows because poor channels can prevent delivery even at maximum power.
4) Performance of Different Number of VUE-Pairs:
With more VUE-pairs, transmission opportunities decrease and packet drops increase, affecting the evaluated performance metrics. Across the experiments, the proactive algorithm provides better average utility than the four baselines.
- 4) Performance of Different Number of VUE-Pairs:: As the number of VUE-pairs increases, each pair has fewer opportunities to receive a frequency band for scheduled transmission.
- 4) Performance of Different Number of VUE-Pairs:: With more VUE-pairs, all algorithms show lower average transmit power consumption and more average packet drops.
- 4) Performance of Different Number of VUE-Pairs:: The proposed proactive algorithm ensures better average utility performance than the other four baseline algorithms across the experiments.
- The paper decomposes the global single-agent MDP into per-VUE-pair MDPs to simplify decision making under high-dimensional local states.
- The LSTM- and DQN-based proactive algorithm addresses partial observability and high-dimensional local state spaces.
- Frequency-band allocation and packet scheduling use local partial observations without a priori knowledge of network dynamics.