PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning Based Mode Selection and Resource Allocation for Cellular V2X Communications

Xinran Zhang, Mugen Peng, Shi Yan, Yaohua Sun

arXiv:2002.05485v1cs.NIeess.SP

TL;DR

Unreliable V2V links and centralized allocation overhead complicate cellular V2X support for heterogeneous QoS requirements. The paper uses decentralized DRL for joint mode selection and resource allocation, supplemented by two-timescale federated learning, and reports stronger decentralized performance plus robust adaptation for newly activated pairs.

  • Problem

    Unreliable V2V links, heterogeneous QoS requirements, and centralized training or allocation overhead motivate joint decentralized optimization for cellular V2X.

  • Method

    The paper models joint mode selection and resource allocation as an MDP with local DRL agents, then combines graph-based large-timescale clustering with small-timescale federated DRL.

  • Results

    The proposed DRL algorithm outperforms other decentralized algorithms, while federated DRL achieves similar final performance with negligible training time for newly activated V2V pairs.

  • Takeaways & Limitations

    Federated global DRL models provide robust and stable performance for newly activated V2V pairs in the evaluated cellular V2X setting.

Abstract

from arXiv · show

Cellular vehicle-to-everything (V2X) communication is crucial to support future diverse vehicular applications. However, for safety-critical applications, unstable vehicle-to-vehicle (V2V) links and high signalling overhead of centralized resource allocation approaches become bottlenecks. In this paper, we investigate a joint optimization problem of transmission mode selection and resource allocation for cellular V2X communications. In particular, the problem is formulated as a Markov decision process, and a deep reinforcement learning (DRL) based decentralized algorithm is proposed to maximize the sum capacity of vehicle-to-infrastructure users while meeting the latency and reliability requirements of V2V pairs. Moreover, considering training limitation of local DRL models, a two-timescale federated DRL algorithm is developed to help obtain robust model. Wherein, the graph theory based vehicle clustering algorithm is executed on a large timescale and in turn the federated learning algorithm is conducted on a small timescale. Simulation results show that the proposed DRL-based algorithm outperforms other decentralized baselines, and validate the superiority of the two-timescale federated DRL algorithm for newly activated V2V pairs.

I. INTRODUCTION

Cellular V2X must support high-capacity V2I traffic alongside safety-critical V2V messages with stringent latency and reliability requirements. The paper jointly optimizes transmission mode, resource allocation, and power using decentralized DRL, then adds two-timescale federated learning to improve robustness and newly activated-pair performance.

  • Motivation: Safety-critical V2V messages may require 5 ms maximum latency and 99.999% reliability, creating QoS demands that centralized allocation approaches struggle to guarantee.The challenge is intensified by unstable V2V links and diverse application requirements.
  • Motivation: Blockage makes V2V links less reliable, motivating V2I forwarding and joint optimization of mode selection and resource allocation.V2I forwarding improves reliability but can increase relay latency and reduce spectrum utilization.
  • Joint optimization: Each V2V pair selects V2V or V2I mode, an RB, and transmit power while the objective maximizes V2I sum capacity under V2V latency and reliability constraints.Resource sharing among V2V pairs in different transmission modes is included.
  • DRL-based decentralized approach: The paper formulates the joint problem as an MDP and assigns each V2V pair a DRL agent that makes decisions from local observations.The decentralized method addresses large continuous-value state spaces and complex interference control.
  • Federated DRL: A two-timescale federated DRL algorithm combines large-timescale graph-based vehicle clustering with small-timescale federated learning.Vehicles in the same cluster cooperate to train robust global DRL models, which can be downloaded by newly activated pairs.
  • Results: Simulation results show the proposed DRL algorithm outperforms decentralized algorithms and performs competitively against a centralized algorithm.The study also examines vehicular density and outage-threshold effects.

II. SYSTEM MODEL AND PROBLEM FORMULATION

The system comprises a base station, V2I users, and V2V pairs sharing resource blocks under mobility, interference, and channel-blockage constraints. The formulation represents RB allocation, channel conditions, SINR, and achievable V2I rates for the optimization problem.

  • II. SYSTEM MODEL AND PROBLEM FORMULATION: The vehicular network contains one centrally located BS and multiple single-antenna VUEs distributed along a crossroad.Active VUEs are divided into M V2I users and K V2V pairs.
  • II. SYSTEM MODEL AND PROBLEM FORMULATION: The total bandwidth is divided into F RBs, with each I-VUE occupying one RB and remaining RBs available for V2V allocation.V2V pairs may select allocated or unused RBs, and multiple V2V pairs may share an RB.
  • II. SYSTEM MODEL AND PROBLEM FORMULATION: The indicator 1m,f equals one when RB f is allocated to I-VUE m, specifically when m=f.Otherwise, the indicator equals zero.
  • II. SYSTEM MODEL AND PROBLEM FORMULATION: The V2V allocation indicator ak,f records whether RB f is allocated to V2V pair k.When ak,f=1, RB f is allocated to pair k.
  • II. SYSTEM MODEL AND PROBLEM FORMULATION: Because of high mobility, the BS and vehicles are assumed to know only large-scale channel gains, including path loss and shadowing.Blockage from nearby vehicles and buildings can place channels in line-of-sight or non-line-of-sight conditions.
  • 1) I-VUEs:: For I-VUE uplinks, the SINR is determined by the desired I-VUE signal, noise, and interference from V2V pairs reusing the same RB.This SINR underlies the I-VUE rate calculation.
  • 1) I-VUEs:: The achievable I-VUE data rate is obtained from its SINR and the bandwidth W assigned to each RB.The rate expression is introduced after defining the per-RB bandwidth.

2) V2V pairs:

Each V2V pair chooses direct V2V communication or indirect V2I forwarding through the BS. The two modes differ in interference structure, RB eligibility, and the SINR-based rate expressions used to evaluate their performance.

  • 2) V2V pairs:: A V2V pair selects V2V or V2I mode according to its individual channel quality.The mode variable sk equals one for V2I mode and zero for V2V mode.
  • V2V mode: In V2V mode, each transmitter communicates directly with its receiver, while interference comes from co-channel I-VUEs and V2V pairs.The mode-specific receiver SINR is defined on RB f.
  • V2V mode: The achievable V2V-mode rate is expressed from the V2V receiver’s SINR on the selected RB.This rate accounts for the direct-link interference environment.
  • V2I mode: In V2I mode, safety-critical data is uploaded to the BS and then forwarded through downlink to the V2V receiver.The paper assumes uplink SINR is lower than downlink SINR, so V2I performance is bounded by the uplink.
  • V2I mode: Only unused RBs can be assigned to V2I-mode V2V pairs, and each such RB can serve at most one V2I-mode pair.Interference comes from V2V-mode pairs sharing the same RB.
  • V2I mode: The achievable V2I-mode rate is calculated from the mode-specific uplink SINR.This reflects the assumption that uplink performance limits the two-hop forwarding mode.

B. QoS Requirements of I-VUEs and V2V Pairs

I-VUEs require minimum capacity to support bandwidth-demanding applications, while V2V pairs must satisfy latency and reliability constraints for safety-critical messages. The paper assumes common capacity, packet-size, latency, and outage-probability requirements within each corresponding user group.

  • I-VUEs have minimum capacity requirements intended to support bandwidth-demanding entertainment and traffic applications.
  • V2V pairs have latency and reliability requirements because they distribute safety-critical messages in real time.
  • Latency is constrained by the transmission time needed to deliver a V2V message within its maximum tolerable latency.
  • Reliability is represented by outage probability, constrained using an outage threshold and tolerable outage probability.
  • Under Rayleigh fading, the reliability constraint can be transformed using an effective outage threshold.
  • The model assumes identical packet sizes, maximum tolerable latencies, and tolerable outage probabilities for all V2V pairs.

C. Problem Formulation

The paper formulates joint mode selection, resource allocation, and power control as a constrained optimization problem, then recasts it as an MDP for decentralized DRL. Each V2V pair observes local state and selects a joint action to balance V2I capacity with V2V QoS.

  • C. Problem Formulation: The objective maximizes total I-VUE capacity while satisfying V2V latency and reliability requirements through joint mode, RB, and power decisions.
  • C. Problem Formulation: The problem is a mixed integer nonlinear program because binary mode and allocation variables combine with non-convex power optimization.
  • C. Problem Formulation: Centralized methods face global-CSI acquisition and high computational complexity, motivating decentralized approaches for dynamic large-scale vehicular networks.
  • B. DRL-based Decentralized Algorithm: The optimization is formulated as an MDP whose DRL agents interact with the cellular V2X environment through states, actions, and rewards.
  • A. Basics of Deep Reinforcement Learning: A DQN-based framework is used because the problem has continuous state spaces and a discrete action space.
  • B. DRL-based Decentralized Algorithm: Each V2V pair observes local interference, channel, load, and timing information before selecting a joint mode, RB, and power action.
  • B. DRL-based Decentralized Algorithm: The discrete action space has size 2FNp, combining F RB choices, two communication modes, and Np power levels.
  • B. DRL-based Decentralized Algorithm: The reward combines I-VUE sum-capacity revenue with penalties for capacity, reliability, and latency violations, weighted by c1, c2, c3, and c4.

IV. FEDERATED DRL-BASED SEMI-DECENTRALIZED ALGORITHM

Local DRL training is challenged by stringent latency requirements, limited training data, newly activated V2V pairs, and vehicle mobility. The proposed two-timescale federated DRL framework trains robust models and improves newly activated-pair performance.

  • Stringent latency requirements and limited training data make accurate local DRL training difficult.
  • Poorly trained local models can cause newly activated V2V pairs to make inferior decisions and degrade global performance.
  • Vehicle mobility can make well-trained DRL models outdated.
  • The proposed two-timescale federated DRL framework trains robust DRL models and improves newly activated V2V-pair performance.

A. Two-Timescale Federated DRL Framework

The framework combines federated learning with centralized vehicle clustering across two timescales. It addresses centralized-training privacy and communication costs while recognizing that local device-side training can be time-consuming and imprecise.

  • The framework uses centralized VUE clustering on a large timescale and federated DRL on a small timescale.
  • Federated learning is introduced because devices may avoid sharing private data due to privacy and communication-cost concerns.
  • Local training and analysis on devices can be time-consuming and imprecise.

1) Basics of federated learning:

Federated DRL combines local model training with periodic model averaging, using large-timescale graph clustering to organize vehicles and reduce resource conflicts. Clusters are formed from large-scale channel gains, and candidate RB groups are assigned accordingly.

  • Federated learning separates local training on raw device data from infrequent centralized averaging of local models.This can enhance distributed DNN and DRL training while incurring negligible uploading overhead compared with centralized learning.
  • The proposed framework operates through graph-based clustering on a large timescale and federated DRL procedures on a small timescale.The BS periodically constructs clusters and candidate RB groups before local models are averaged within clusters.
  • Graph construction: Each graph vertex represents a V2V pair or I-VUE, with edges weighted by large-scale channel gains rather than Euclidean distances.Large-scale gains are used because nearby VUE links can be unreliable under blockage.
  • Graph clustering: The clustering task partitions the graph to maximize within-cluster edge weights under disjoint cluster-covering constraints.The resulting graph-partitioning problem is NP-hard, so spectral clustering is adopted instead of Euclidean-distance methods or randomly initialized heuristics.
  • RB assignment: Each cluster receives candidate RBs excluding resources allocated to its member I-VUEs, while same-cluster VUEs use orthogonal resources to mitigate interference.The candidate RB assignment is derived from the clustering results.

C. Federated DRL on a Small Timescale

On the small timescale, V2V pairs asynchronously train local DRL models and periodically aggregate them through cluster-specific federated averaging. Newly activated pairs receive an existing cluster model to avoid time-consuming local training.

  • At each coordination round, the BS distributes a pretrained or averaged model, and V2V pairs train local models using their local data.Local training follows the DRL procedure within each cluster.
  • Federated averaging: Local Q-network updates are aggregated into a cluster global model through mini-batch-based federated averaging.The update weights depend on the total batch size and each V2V pair’s training batch size.
  • Asynchronous decisions: Asynchronous action selection assigns each V2V pair in a cluster to a specific subframe, reducing resource collisions caused by incomplete local observations.Each subframe block contains as many subframes as V2V pairs in the cluster.
  • Newly activated pairs: Newly activated V2V pairs request cluster assignment and download the corresponding global DRL model and network parameters.This avoids the time-consuming training of a local DRL model from scratch.

V. SIMULATION AND NUMERICAL RESULTS

The simulation evaluates the proposed DRL methods in a 1 km × 1 km crossroad scenario with randomly deployed I-VUEs and V2V links. Comparisons include centralized, decentralized DRL without mode selection, and random-selection algorithms.

  • Simulation setting: The evaluation uses a crossroad with a centrally located BS, vehicles generated by a spatial Poisson process, and urban-street channel parameters.The crossroad measures 1 km × 1 km, with two lanes in each direction.
  • Simulation setting: Five active I-VUEs and K active V2V transmitters are selected, with each transmitter paired to the farthest vehicle within its broadcast range.LOS status, path loss, shadowing, and fast fading follow the 3GPP urban street scenario.
  • DRL configuration: The DQN uses a fully connected network with one hidden layer of 256 neurons, ReLU activation, and adaptive moment estimation.The parameters were selected through multiple simulation tests to balance DRL complexity and performance.
  • Baselines: The proposed methods are compared with a centralized algorithm, decentralized DRL without mode selection, and random RB selection.The centralized method uses global CSI, while the decentralized baselines rely on local decisions or random candidate-RB selection.

A. Network Performance versus the Number of V2V pairs

As V2V-pair density increases, the proposed DRL algorithm outperforms decentralized baselines on I-VUE sum capacity and V2V satisfied rate, while approaching centralized performance. As outage thresholds increase, both metrics decline for most evaluated methods, but adaptive mode selection remains advantageous.

  • Number of V2V pairs: The proposed DRL algorithm outperforms other decentralized algorithms in I-VUE sum capacity and V2V satisfied rate as the number of V2V pairs varies.Its advantage grows with more V2V pairs because it identifies unstable NLOS links and selects transmission modes using local observations.
  • Number of V2V pairs: The proposed DRL algorithm achieves performance close to the centralized algorithm while making decisions from local observations.The centralized approach requires global CSI, high computational complexity, and frequent rescheduling for newly activated pairs.
  • Outage threshold: As the outage threshold increases, I-VUE sum capacity and V2V satisfied rate decline for the centralized, proposed DRL, and DRL-without-mode-selection algorithms.Higher outage thresholds lead V2V pairs to use higher transmission power, increasing interference.
  • Outage threshold: The proposed DRL algorithm outperforms other decentralized algorithms across outage thresholds by adaptively selecting transmission modes to alleviate interference.The random-selection algorithm produces the same I-VUE sum capacity for different outage thresholds because it ignores I-VUE capacity when choosing RBs.

C. Effectiveness of the Federated DRL Algorithm

The federated DRL approach combines graph-based vehicle clustering with federated learning to train robust models and support newly activated V2V pairs. It converges toward centralized performance and provides stable performance with negligible training time for newly activated pairs.

  • Vehicle clustering: Graph-based clustering groups neighboring V2V pairs and I-VUEs with similar line-of-sight states into the same cluster.The method uses large-scale channel gain as graph-edge weights and applies spectral clustering.
  • Federated learning: The federated DRL model’s average reward increases with training epochs before reaching a relatively stable value.This behavior demonstrates convergence of the federated DRL learning process.
  • Federated learning: The converged federated DRL reward is close to the centralized algorithm’s optimal reward.The simulation uses 10 V2V pairs for this learning-process evaluation.
  • Newly activated V2V pairs: For a newly activated V2V pair, the federated DRL algorithm downloads the global model, while comparison DRL methods use transfer learning or train locally from scratch.The federated model is formed by averaging local models within the same cluster.
  • Newly activated V2V pairs: The federated DRL algorithm achieves similar final performance with negligible training time and more stable behavior than the two comparison DRL algorithms.The comparison methods show greater fluctuation because of exploration and nonideal DRL models.
Loading 2002.05485v1…