Source-linked AI summary
Deep Reinforcement Learning for Resource Allocation in V2V Communications
Hao Ye, Geoffrey Ye Li
TL;DR
V2V safety links must meet stringent latency and reliability requirements while sharing spectrum with V2I communications. The paper uses decentralized multi-agent deep reinforcement learning, with each V2V link selecting sub-bands and power levels from local observations. Simulations show that agents can satisfy V2V constraints while reducing interference to V2I communications.
Problem
V2V resource allocation must handle stringent latency and reliability requirements, but prior methods have difficulty modeling latency directly and centralized schemes require costly global information.
Method
Each V2V link acts as an agent using deep Q-learning and a neural-network Q-function to select transmission sub-bands and power levels from observed state information.
Results
The proposed method achieves better V2I-rate performance, dynamically allocates resources toward latency-violating links, and learns to satisfy V2V constraints while minimizing V2I interference.
Takeaways & Limitations
Decentralized decisions avoid requiring global network information, keeping transmission overhead small while jointly addressing V2V constraints and V2I interference.
Abstract
from arXiv · showhide
In this article, we develop a decentralized resource allocation mechanism for vehicle-to-vehicle (V2V) communication systems based on deep reinforcement learning. Each V2V link is considered as an agent, making its own decisions to find optimal sub-band and power level for transmission. Since the proposed method is decentralized, the global information is not required for each agent to make its decisions, hence the transmission overhead is small. From the simulation results, each agent can learn how to satisfy the V2V constraints while minimizing the interference to vehicle-to-infrastructure (V2I) communications.
I. INTRODUCTION
V2V communications require stringent latency and reliability, while existing centralized or traditional allocation methods face scalability and modeling challenges. The paper applies multi-agent deep reinforcement learning to decentralized spectrum and power allocation, aiming to satisfy V2V constraints while limiting V2I interference.
- V2V links require millisecond-scale end-to-end latency and nearly 100% reliability for traffic-safety applications.
- High vehicle mobility causes wireless channels to change rapidly over time, creating resource-management challenges for V2V systems.
- Centralized allocation requires global network information, while the resulting optimization problems are usually NP-hard and difficult to solve.
- Centralized control also incurs transmission overhead that grows linearly with mobile speed and quadratically with the number of vehicles.
- The proposed multi-agent deep reinforcement learning mechanism lets each V2V link learn spectrum and power decisions while directly addressing latency constraints.
- Simulation results indicate that agents can share channels and generate low interference to V2I channels.
II. SYSTEM MODEL
The system reuses orthogonal V2I uplink spectrum for V2V links, creating interference-management and constraint-design requirements. The proposed objective emphasizes satisfying V2V latency and reliability while preserving V2I throughput.
- The network contains M cellular users with V2I links and K D2D-user pairs requiring V2V links for traffic-safety information exchange.
- Orthogonal V2I uplink spectrum is reused by V2V links to improve spectrum utilization efficiency.
- V2I interference comprises background noise and signals from V2V links sharing the same sub-band.
- Latency and reliability constraints are modeled directly in the reward, with a lower reward when V2V constraints are violated.
- The reward also reflects V2I sum-rate maximization, whereas the main autonomous-allocation goal is satisfying each V2V link's latency constraint.
III. DEEP REINFORCEMENT LEARNING FOR RESOURCE ALLOCATION
The paper frames decentralized V2V resource allocation as a deep reinforcement learning problem and introduces the framework used to represent and train the agents.
- The deep reinforcement learning framework represents key components of V2V resource allocation and trains deep Q-networks.
- The approach is designed for decentralized resource allocation in V2V communications.
- The framework focuses on learning resource-allocation decisions rather than relying solely on predefined optimization procedures.
A. Reinforcement Learning
Each V2V link acts as an agent that observes local state, selects a sub-channel and power level, and receives a reward tied to V2I capacity and V2V latency constraints.
- In decentralized settings, each V2V link is an agent interacting with an environment that includes the uncontrollable actions of other V2V links.
- The agent observes state and selects a sub-band and transmission power according to a deep-learning-derived Q-function policy.
- Transmission power is discretized into three levels, giving an action-space dimension of 3 × N_RB for N_RB resource blocks.
- The reward is positive when V2V constraints are satisfied and becomes a negative penalty r_N when they are violated.
- State transitions and rewards follow a Markov decision process, with the agent controlling only its own actions.
B. Q-Learning
Q-learning maps observed states to spectrum-and-power actions through a policy, selecting actions that maximize long-term discounted rewards without requiring known system dynamics.
- The policy maps each state st to an action at, where the action selects transmission power and spectrum.
- Q-learning constructs an optimal resource-allocation policy by maximizing long-term expected accumulated discounted rewards.
- The agent selects the action with the maximum Q-value for the observed state.
- Q-learning can converge to the optimal Q∗ under infinite exploration of every state-action pair and an appropriately decaying learning rate.
C. Deep Q Networks
Deep Q-networks extend Q-learning to large state-action spaces by approximating the Q-function with a deep neural network trained to minimize a loss over stored data.
- Deep Q-networks replace lookup tables with a deep neural network that approximates the Q-function for large state-action spaces.The network learns mappings from channel information to desired outputs used to determine Q-values.
- The Q-network updates its weights θ at each iteration to minimize a loss derived from old network weights over data set D.
- The corresponding reward is denoted rt in the loss formulation.
D. Training and Testing Algorithms
Training uses simulated interaction and experience replay to learn a deep Q-network, while testing loads the network and evaluates greedy actions in the simulator.
- Training: Training and testing data are generated by an environment simulator containing DUEs, CUEs, vehicles, and their wireless channels.
- Training: Experience replay stores generated state, action, reward, and post-state data in memory, then samples mini-batches for Q-network updates.Sampling from memory suppresses temporal correlation in the training data.
- Training: The training procedure repeatedly samples V2V links, generates data under policy π, updates the network, and chooses the action with maximum Q-value.
- Testing: Testing loads the Q-network, samples V2V links, selects largest-Q actions, updates the simulator, and returns evaluation results.
- Testing: Evaluation results include average V2I capacity and the probability of successful DUEs.
IV. SIMULATIONS
Simulations evaluate the proposed deep Q-network in a 2 GHz Manhattan outdoor system and compare it with random allocation and a position-and-load-based iterative scheme.
- The simulation uses a single-cell outdoor system at 2 GHz with nine Manhattan blocks and both LOS and NLOS channels.Vehicles are randomly placed according to a spatial Poisson process, and each communicates with three nearby vehicles.
- The deep Q-network is a five-layer fully connected network with hidden-layer sizes 500, 250, and 120, using ReLU activation.
- The training setup uses an ε-greedy policy for exploration-exploitation balance and Adam for optimization.
- The proposed method is compared with random resource allocation and a method that groups vehicles before iterative sub-band allocation and adjustment.
A. V2I Capacity
The proposed method achieves much better mitigation of V2V interference to V2I communications than the compared approaches as the number of V2V links varies.
- The proposed method has much better performance in mitigating V2V interference to V2I communications.
B. V2V Latency
The proposed method increases the probability that V2V links satisfy latency constraints as the number of links changes. It dynamically adjusts transmission power and sub-band allocation to provide more resources to links likely to violate latency requirements.
- The proposed method gives DUEs a larger probability of satisfying the latency constraint.
- It dynamically adjusts transmission power and sub-band allocation for transmission.Links likely to violate the latency constraint receive more resources.
- The mechanism is decentralized, with each V2V link making its own sub-band and power-level decisions.Global information is not required for each agent's decisions, keeping transmission overhead small.