Source-linked AI summary
Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning
Le Liang, Hao Ye, Geoffrey Ye Li
TL;DR
The paper addresses spectrum sharing between multiple V2V links and V2I links in high-mobility vehicular networks, where centralized control is hindered by rapidly changing channels. It models sharing as multi-agent RL and uses fingerprint-based DQNs with centralized training and distributed implementation. The learned agents cooperate locally to improve both V2I capacity and V2V payload delivery, closely approaching centralized benchmarks in the reported tests.
Problem
High mobility prevents accurate full CSI at a central controller, while coordinating distributed V2V decisions and sequential packet delivery remains challenging.
Method
The paper uses fingerprint-based multi-agent DQNs with a common reward, experience replay, centralized training, and distributed implementation.
Results
The proposed MARL outperforms distributed baselines across tested V2V payload sizes, reaches 100% delivery probability for B = 1060 and B = 2×1060 bytes, and stays close to centralized maxV2V.
Takeaways & Limitations
V2V agents can learn cooperative spectrum and power allocation from local information while improving system-level V2I and V2V performance.
Abstract
from arXiv · showhide
This paper investigates the spectrum sharing problem in vehicular networks based on multi-agent reinforcement learning, where multiple vehicle-to-vehicle (V2V) links reuse the frequency spectrum preoccupied by vehicle-to-infrastructure (V2I) links. Fast channel variations in high mobility vehicular environments preclude the possibility of collecting accurate instantaneous channel state information at the base station for centralized resource management. In response, we model the resource sharing as a multi-agent reinforcement learning problem, which is then solved using a fingerprint-based deep Q-network method that is amenable to a distributed implementation. The V2V links, each acting as an agent, collectively interact with the communication environment, receive distinctive observations yet a common reward, and learn to improve spectrum and power allocation through updating Q-networks using the gained experiences. We demonstrate that with a proper reward design and training mechanism, the multiple V2V agents successfully learn to cooperate in a distributed way to simultaneously improve the sum capacity of V2I links and payload delivery rate of V2V links.
I. INTRODUCTION
The paper studies spectrum sharing between V2V and V2I links in high-mobility vehicular networks, where diverse service requirements and rapidly changing channels complicate conventional resource allocation. It motivates a distributed multi-agent RL approach that supports local decision-making and cooperation among V2V links.
- V2I links support cellular connectivity, while V2V links provide direct communication among neighboring vehicles.
- V2V links reuse V2I spectrum to improve utilization, requiring coordinated sub-band selection and transmission-power control.
- Fast-changing vehicular channels create uncertainty because centralized resource allocation may rely on inaccurate channel state information.
- The work targets simultaneous improvement of V2I capacity and V2V payload delivery reliability under diverse service requirements.
- A multi-agent RL approach is proposed so V2V links can cooperate while making distributed spectrum-access decisions using local information.
C. Contribution
The paper contributes a distributed multi-agent RL solution for jointly improving V2I and V2V performance during spectrum sharing. Its design explicitly addresses V2V reliability and enables cooperative local decisions.
- The paper models multiple V2V spectrum-access decisions as a multi-agent problem and develops distributed spectrum and power allocation.
- The method directly treats reliable delivery of periodic V2V safety messages under a time budget while adapting to small-scale channel fading.
- A proper reward and training mechanism enables V2V transmitters to learn cooperative system-level strategies from local information.
II. SYSTEM MODEL
The system contains fixed-power, orthogonally assigned V2I links and V2V links that select shared spectrum and transmission power. The optimization jointly targets V2I sum capacity and V2V packet delivery within a time constraint, while high mobility motivates distributed control.
- The model includes M V2I links for high-data-rate services and K V2V links for periodic safety-message dissemination.
- V2I links occupy preassigned orthogonal sub-bands with fixed power, leaving V2V links to select sub-bands and control transmission power.
- Each V2V link accesses one sub-band through a binary allocation indicator, while channel fading is modeled as approximately constant within a sub-band.
- The objectives are to maximize V2I sum capacity and the rate of delivering V2V packets of size B within time budget T.
- High mobility prevents accurate full CSI collection at a central controller, and sequential V2V decisions across coherence slots challenge conventional optimization.
III. MULTI-AGENT RL BASED RESOURCE ALLOCATION
The resource-sharing problem is formulated as multi-agent RL, with each V2V link interacting with the environment and refining local allocation policies. A common reward converts the agents’ interaction into a cooperative learning problem, using centralized training and distributed implementation.
- Each V2V link acts as an agent that observes the communication environment and learns spectrum allocation and power-control strategies from experience.
- The agents receive a common reward so the resource-sharing problem is treated as fully cooperative with respect to global network performance.
- The approach separates centralized learning, where agents update DQNs using a system-level reward, from distributed implementation using local observations.
A. State and Observation Space
Each V2V link acts as an agent that receives a local observation, selects actions, and shares a common reward while interacting with the evolving environment. The fingerprint-based method addresses nonstationarity caused by concurrently learning agents.
- State and Observation Space: V2V links independently observe the environment and jointly form the system action at each coherence time step.The environment then transitions and returns a reward before the next observations arrive.
- State and Observation Space: All V2V agents share the same reward, encouraging cooperative behavior.
- State and Observation Space: Each agent’s observation includes local channel information for its own link and relevant interference channels.These include V2V interference, interference to the BS, and V2I interference channels.
- State and Observation Space: The true global environment state is unknown to individual agents, which access it through an observation function.
- State and Observation Space: Fingerprint-based learning conditions the action-value function on estimates of other agents’ policies to reduce nonstationarity.Naive independent Q-learning with experience replay can use experiences that no longer reflect current dynamics.
B. Action Space
Resource sharing requires each V2V link to select a spectrum sub-band and transmission power. The paper discretizes power into four levels, producing an action space with 4 × M possible combinations.
- Action Space: Each V2V action combines selection of one of M disjoint spectrum sub-bands with a transmission-power choice.
- Action Space: The available V2V power levels are 23, 10, 5, and −100 dBm.The −100 dBm option represents zero V2V transmission power.
- Action Space: The resulting action-space dimension is 4 × M, with each action representing a spectrum-power combination.
C. Reward Design
The reward combines instantaneous aggregate V2I capacity with V2V delivery progress, using a terminal bonus to encourage payload completion within the time constraint. The design balances the two objectives while preserving learning efficiency.
- Reward Design: The reward design targets both maximizing sum V2I capacity and increasing successful V2V payload delivery within time T.
- Reward Design: Instantaneous sum capacity of all V2I links is included in the reward at each time step.
- Reward Design: Before delivery, each V2V agent receives its effective transmission rate; after delivery, it receives β greater than the largest possible V2V rate.
- Reward Design: With γ = 1, maximizing cumulative rewards encourages more V2V data transmission before completion and more successful payload deliveries.
- Reward Design: β is tuned empirically because a purely terminal reward can hinder learning efficiency.The tuning procedure sets β above the largest V2V rate observed under several random resource-allocation steps.
- Reward Design: The V2I and V2V reward components are balanced using positive weights λc and λd.
D. Learning Algorithm
The learning process uses episodic interaction over the V2V delivery deadline, with states evolving as channel fading changes and agents adapting their actions. Episodes begin from randomized environment states with full payloads.
- Learning Algorithm: Each episode spans the V2V payload delivery time constraint T.
- Learning Algorithm: Episodes start from a randomly initialized environment state and a full V2V payload of size B.
- Learning Algorithm: Small-scale channel-fading changes trigger state transitions and cause each V2V agent to adjust its action.
1) Training Procedure:
The training procedure equips each V2V agent with a DQN and uses exploration, replayed experiences, and target networks to learn spectrum-access policies.
- Deep Q-learning trains multiple V2V agents to learn effective spectrum-access policies.
- Each V2V agent has a dedicated DQN that maps its current observation to action-value estimates.
- Agents explore the state-action space with soft policies such as ε-greedy during repeated training episodes.The maximal-value action is selected with probability 1−ε, while a random action is selected with probability ε.
- Experience replay updates Q-network parameters from uniformly sampled mini-batches of stored experiences.This improves sample efficiency, reduces correlation between successive updates, and stabilizes learning.
- A target Q-network is periodically duplicated from the training network and held fixed for several updates.
2) Distributed Implementation:
During deployment, each V2V agent uses local channel estimates and its trained Q-network to select spectrum and power actions, while training remains offline.
- Each V2V agent estimates local channels and compiles a local environment observation at every implementation time step.
- Each agent selects the action with the maximum action value according to its trained Q-network.
- All V2V links transmit using the power level and frequency sub-band determined by their selected actions.
- The trained DQNs are updated only after significant changes in environment characteristics, potentially once a week or month.Computation-intensive training can run offline, whereas the inexpensive implementation procedure runs online.
IV. SIMULATION RESULTS
Simulations evaluate distributed MARL against distributed baselines and idealized upper bounds under varying V2V payload sizes. The proposed method improves V2I capacity and V2V delivery, learns cooperation during training, and has payload-size limitations.
- Setup and baselines: The simulator follows the urban-case evaluation methodology in 3GPP TR 36.885, with figure-specific settings taking precedence over default parameters.
- Setup and baselines: MARL is compared with SARL and random distributed baselines, alongside idealized upper bounds for V2I and V2V performance.The upper bounds are not implementable but indicate how closely MARL approaches ideal performance.
- V2I capacity: 14% degradation is the proposed method’s worst-case distance from the V2I upper bound at a V2V payload of 6 × 10^60 bytes.MARL outperforms the two distributed baselines across tested payload sizes despite training with a fixed 2 × 10^60-byte payload.
- V2V delivery: 100% V2V payload delivery probability is achieved for payloads of 10^60 and 2 × 10^60 bytes, with near-perfect performance at 3 × 10^60 and 4 × 10^60 bytes.Success probabilities decline as payload size grows for the distributed algorithms, while centralized maxV2V remains at 100% in the tested cases.
- Robustness boundary: Payload variation constrains robustness: performance noticeably degrades beyond 4 × 10^60 bytes, and maintaining delivery above 95% requires packets no larger than 5 × 10^60 bytes in the current setting.The authors caution that these thresholds depend on system parameters and should not be generalized directly.
- Training convergence: Around 2,000 training episodes, cumulative reward gradually converges despite mobility-induced channel-fading fluctuations.Each agent is trained for 3,000 episodes for the evaluations in Figs. 3 and 4.
- Learned cooperation: MARL coordinates V2V links by exploiting good channels, protecting vulnerable links, and sequencing transmissions so all links can deliver payloads within the time constraint.In the illustrated episode, MARL delivers the payload for all links, whereas the random baseline leaves V2V Link 2 unsuccessful.
V. CONCLUSION
The paper develops a distributed multi-agent reinforcement-learning resource-sharing scheme for vehicular networks and uses fingerprinting to address nonstationarity in independent Q-learning with DQN experience replay. The authors report that centralized training followed by distributed implementation encourages cooperation among locally acting V2V transmitters, while identifying robustness analysis and MIMO extensions as future work.
- The proposed scheme uses fingerprint-based multi-agent reinforcement learning to address nonstationarity when independent Q-learning is combined with DQN experience replay.
- Centralized training followed by distributed implementation encourages cooperation among V2V links while each transmitter makes local decisions.
- Fig. 7 compares V2V transmission rates for MARL and random resource allocation during the first 30 ms of an episode with initial payload B = 2,120 bytes.
- Future work includes analyzing robustness and update requirements for single-agent versus multi-agent algorithms and extending resource allocation to MIMO and millimeter MIMO scenarios.