Source-linked AI summary
Deep Reinforcement Learning based Resource Allocation for V2V Communications
Hao Ye, Geoffrey Ye Li, Biing-Hwang Fred Juang
TL;DR
V2V resource allocation must handle stringent latency requirements, rapidly changing channels, and interference without relying on centralized global information. The paper develops decentralized multi-agent deep reinforcement learning for unicast and broadcast scenarios, and simulations show that agents learn to satisfy V2V latency constraints while minimizing interference to V2I communications.
Problem
Rapidly changing channels, centralized overhead, and difficult-to-formulate latency constraints complicate V2V resource allocation.
Method
The paper develops a decentralized multi-agent deep reinforcement learning mechanism in which V2V links or vehicles select resources for unicast and broadcast communications.
Results
Simulations show that agents learn to satisfy V2V latency constraints while minimizing interference to V2I communications.
Takeaways & Limitations
The mechanism operates without global information for each agent and incurs small transmission overhead.
Abstract
from arXiv · showhide
In this paper, we develop a decentralized resource allocation mechanism for vehicle-to-vehicle (V2V) communications based on deep reinforcement learning, which can be applied to both unicast and broadcast scenarios. According to the decentralized resource allocation mechanism, an autonomous agent', a V2V link or a vehicle, makes its decisions to find the optimal sub-band and power level for transmission without requiring or having to wait for global information. Since the proposed method is decentralized, it incurs only limited transmission overhead. From the simulation results, each agent can effectively learn to satisfy the stringent latency constraints on V2V links while minimizing the interference to vehicle-to-infrastructure (V2I) communications.
I. INTRODUCTION
V2V communications require stringent latency and reliability, while high mobility makes full-CSI resource management difficult. The paper therefore develops decentralized multi-agent deep reinforcement learning for unicast and broadcast resource allocation.
- High mobility causes rapidly changing wireless channels, limiting traditional V2V resource-management methods that assume full CSI.
- Centralized V2V resource allocation requires vehicles to report local channel and interference information, creating large overhead that grows with network size.
- Broadcast communication is relevant when surrounding vehicles are the destinations, but blind broadcasting can cause packet collisions through broadcast storms.
- Latency constraints have not been thoroughly incorporated into prior V2V optimization because they are difficult to formulate directly.
- The framework maps local observations, including channel and interference information, to resource allocation and scheduling decisions.
- The proposed decentralized mechanism uses multi-agent deep reinforcement learning to address V2V latency constraints in both unicast and broadcast scenarios.
II. SYSTEM MODEL FOR UNICAST COMMUNICATION
The unicast system shares uplink V2I spectrum with V2V links, requiring resource allocation to satisfy V2V latency while limiting interference to V2I and other V2V links. Agents use channel, interference, and neighbor information to make decentralized decisions.
- The network contains M cellular users requesting V2I links and K V2V-user pairs sharing orthogonally allocated V2I uplink spectrum.
- V2V safety communication prioritizes stringent latency and reliability requirements, while data rate is less important.
- The resource-management objective is to satisfy V2V latency constraints while minimizing V2V interference to V2I links.
- Each V2V link selects a resource block and transmission power without requiring the base station to manage V2V resources.
- Local observations include instantaneous V2V and V2I channel information, previous interference, and neighbors’ prior sub-channel selections.
III. DEEP REINFORCEMENT LEARNING FOR UNICAST RESOURCE ALLOCATION
The unicast framework models each V2V link as an agent that observes local state and selects a sub-channel and power level. Deep Q-learning uses rewards combining link capacities with a latency penalty and optimizes discounted future rewards.
- Each V2V link acts as an agent interacting with an environment representing conditions beyond that link in the decentralized resource-allocation setting.
- At each time step, the agent observes a state and selects a sub-channel and transmission power according to a policy approximated by deep learning.
- The state includes channel and interference information, neighbor sub-channel selections, remaining transmission load, and remaining latency time.
- The action space has dimension 3×N_RB because transmission power is discretized into three levels across N_RB resource blocks.
- The reward combines V2I capacity, V2V capacity, and a penalty for violating or approaching the latency constraint.
- The learning objective maximizes expected cumulative discounted rewards, with discount factor β ∈ [0, 1].
B. Deep Q-Learning
The method uses a policy mapping from observed states to actions, with Q-values estimating long-term discounted rewards and deep networks approximating the Q-function for large state-action spaces.
- The policy maps each state to an action selecting a transmission power level and spectrum subband for a V2V link.
- Q-learning evaluates state-action pairs by their expected accumulated discounted rewards and uses the highest Q-value to construct an improved policy.
- The optimal policy can be obtained without prior knowledge of system dynamics through the Q-value update process.
- Under infinite exploration and an appropriately decaying learning rate, Q-values converge with probability 1 to the optimal Q∗.
- After training, the learned policy selects spectrum and power to maximize overall capacity while satisfying V2V latency constraints.
- Classic Q-learning becomes impractical for the large V2V state-action space, motivating deep Q-networks that combine neural networks with Q-learning.
C. Training and Testing Algorithms
Training and testing use an environment simulator and interacting agents, while asynchronous updates expose environmental changes caused by other V2V links.
- Training and testing data are generated through interactions between an environment simulator and the agents.
- Each training sample contains the state, next state, action, and reward used to optimize the deep Q-network.
- Asynchronous action updates mitigate the problem that simultaneous independent decisions hide other V2V links’ actions from each agent.
- The training procedure initializes the policy and simulator, selects spectrum and power, generates states and rewards, and stores experience in memory.
- Testing loads the Q-network, selects the maximum-Q action for each V2V link, updates the simulator, and records evaluation results.
IV. RESOURCE ALLOCATION FOR BROADCAST SYSTEM
The paper extends its deep-reinforcement-learning resource allocation scheme from unicast to broadcast V2V communications and formulates the associated learning framework and training algorithms.
- The resource allocation scheme based on deep reinforcement learning is extended to the broadcast V2V communication scenario.
- The broadcast-system section formulates key reinforcement-learning elements and presents algorithms for training deep Q-networks.
A. Broadcast System
The broadcast system models vehicles transmitting messages to groups of surrounding receivers while decentralized decisions use local information to meet latency constraints and minimize V2I interference.
- Broadcast V2V communication has one transmitter and a group of surrounding receivers for each message, alongside CUEs demanding V2I links.
- V2V links reuse uplink spectrum allocated to V2I links because uplink resources are less intensively used and base-station interference is more controllable.
- Broadcast reliability requires rebroadcasting received messages, but dense networks can create broadcast storms through excessive redundant transmissions.
- The V2I capacity model accounts for noise and signals from VUEs sharing the same subband, with transmission powers, channel gains, and allocation indicators defining interference.
- For broadcast receivers, capacity is based on the receiver’s SINR and includes interference from CUEs and other VUEs.
- Each vehicle independently selects messages to broadcast and subchannels using local observations, with latency satisfaction and minimized V2I interference as goals.
- Broadcast-specific observations include how often a message was received and the minimum distance to vehicles that broadcast it.
B. Reinforcement Learning for Broadcast
The broadcast framework lets each vehicle select messages and sub-channels from local observations using a trained deep Q-network. Simulations evaluate V2I capacity and successful V2V delivery against random and grouped allocation methods.
- Framework: Each vehicle acts as an agent that selects broadcast messages and a transmission sub-channel according to its policy.The action space provides one action per resource block plus an action for withholding transmission.
- State representation: Broadcast agents observe interference, V2I channel information, neighbors’ prior sub-channel use, remaining latency, reception counts, and sender distance.The state is summarized as st = [It−1, Ht, Nt−1, Ut, Ot, Dt].
- Objective: The reward balances V2I capacity, V2V capacity, and latency satisfaction while limiting interference to V2I links and other V2V links.The same objective applies to selecting broadcast channels and messages under V2V latency constraints.
- Training and testing: Deep Q-network training uses simulated states, rewards, and actions stored in memory, then mini-batch updates to improve the policy.Testing selects actions with the largest Q-value and reports average V2I capacity and successful V2V delivery probability.
- Evaluation: The proposed method is compared with random sub-band selection and an iterative grouped allocation method.The grouped method allocates and adjusts sub-bands among vehicles grouped by similarity.
- Results: The proposed method achieves much better V2I rate and a much larger probability of satisfying V2V latency constraints than the comparison methods.The V2I-rate comparison attributes the advantage to better mitigation of V2V interference to V2I communications.
2) V2V Latency:
The evaluation measures how often V2V links satisfy their latency constraint as vehicle density increases. The proposed method maintains a much larger satisfaction probability by dynamically reallocating transmission resources.
- V2V Latency: The figure plots the probability that V2V links satisfy the latency constraint against the number of vehicles.This directly evaluates latency satisfaction under increasing vehicle density.
- V2V Latency: The proposed method has a much larger probability of satisfying the V2V latency constraint.The passage reports this comparison directly from the figure.
- V2V Latency: Dynamic power and sub-band adjustment gives links likely to violate latency constraints more resources.This is the reported mechanism behind the higher satisfaction probability.
B. Broadcast
In broadcast V2V communications, the deep reinforcement learning method jointly optimizes scheduling and channel selection, improving latency satisfaction and mitigating interference to V2I links as vehicle count changes.
- Broadcast: A broadcast V2V transmission is successful only when all targeted receivers successfully receive the message.
- Broadcast: The method jointly optimizes scheduling and channel selection for broadcast V2V communications.
- Broadcast: The proposed method yields a larger probability that VUEs satisfy latency constraints as the number of vehicles varies.
- Broadcast: The proposed method achieves better performance in mitigating V2V interference to V2I communications as the number of vehicles varies.
- Broadcast: The decentralized approach enables agents to satisfy V2V constraints while minimizing interference to V2I communications.