PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning Based Mode Selection and Resource Allocation for Cellular V2X Communications

Xinran Zhang, Mugen Peng, Shi Yan, Yaohua Sun

arXiv:2002.05485v1cs.NIeess.SP

TL;DR

Unreliable V2V links and centralized allocation overhead complicate cellular V2X support for heterogeneous QoS requirements. The paper uses decentralized DRL for joint mode selection and resource allocation, supplemented by two-timescale federated learning, and reports stronger decentralized performance plus robust adaptation for newly activated pairs.

Abstract

from arXiv · show

Cellular vehicle-to-everything (V2X) communication is crucial to support future diverse vehicular applications. However, for safety-critical applications, unstable vehicle-to-vehicle (V2V) links and high signalling overhead of centralized resource allocation approaches become bottlenecks. In this paper, we investigate a joint optimization problem of transmission mode selection and resource allocation for cellular V2X communications. In particular, the problem is formulated as a Markov decision process, and a deep reinforcement learning (DRL) based decentralized algorithm is proposed to maximize the sum capacity of vehicle-to-infrastructure users while meeting the latency and reliability requirements of V2V pairs. Moreover, considering training limitation of local DRL models, a two-timescale federated DRL algorithm is developed to help obtain robust model. Wherein, the graph theory based vehicle clustering algorithm is executed on a large timescale and in turn the federated learning algorithm is conducted on a small timescale. Simulation results show that the proposed DRL-based algorithm outperforms other decentralized baselines, and validate the superiority of the two-timescale federated DRL algorithm for newly activated V2V pairs.

I. INTRODUCTION

Cellular V2X must support high-capacity V2I traffic alongside safety-critical V2V messages with stringent latency and reliability requirements. The paper jointly optimizes transmission mode, resource allocation, and power using decentralized DRL, then adds two-timescale federated learning to improve robustness and newly activated-pair performance.

Motivation

  • Safety-critical V2V messages may require 5 ms maximum latency and 99.999% reliability, creating QoS demands that centralized allocation approaches struggle to guarantee.The challenge is intensified by unstable V2V links and diverse application requirements.Source: §I · p. 1
  • Blockage makes V2V links less reliable, motivating V2I forwarding and joint optimization of mode selection and resource allocation.V2I forwarding improves reliability but can increase relay latency and reduce spectrum utilization.Source: A. Related Works and Chanllenges · p. 2

Joint optimization

  • Each V2V pair selects V2V or V2I mode, an RB, and transmit power while the objective maximizes V2I sum capacity under V2V latency and reliability constraints.Resource sharing among V2V pairs in different transmission modes is included.Source: B. Contributions and Organization · p. 2

DRL-based decentralized approach

  • The paper formulates the joint problem as an MDP and assigns each V2V pair a DRL agent that makes decisions from local observations.The decentralized method addresses large continuous-value state spaces and complex interference control.Source: B. Contributions and Organization · p. 2

Federated DRL

  • A two-timescale federated DRL algorithm combines large-timescale graph-based vehicle clustering with small-timescale federated learning.Vehicles in the same cluster cooperate to train robust global DRL models, which can be downloaded by newly activated pairs.Source: B. Contributions and Organization · p. 2

Results

  • Simulation results show the proposed DRL algorithm outperforms decentralized algorithms and performs competitively against a centralized algorithm.The study also examines vehicular density and outage-threshold effects.Source: B. Contributions and Organization · pp. 2–3

II. SYSTEM MODEL AND PROBLEM FORMULATION

The system comprises a base station, V2I users, and V2V pairs sharing resource blocks under mobility, interference, and channel-blockage constraints. The formulation represents RB allocation, channel conditions, SINR, and achievable V2I rates for the optimization problem.

II. SYSTEM MODEL AND PROBLEM FORMULATION

  • The vehicular network contains one centrally located BS and multiple single-antenna VUEs distributed along a crossroad.Active VUEs are divided into M V2I users and K V2V pairs.Source: §II · p. 3
  • The total bandwidth is divided into F RBs, with each I-VUE occupying one RB and remaining RBs available for V2V allocation.V2V pairs may select allocated or unused RBs, and multiple V2V pairs may share an RB.Source: §II · p. 3
  • The indicator 1m,f equals one when RB f is allocated to I-VUE m, specifically when m=f.Otherwise, the indicator equals zero.Source: §II · p. 3
  • The V2V allocation indicator ak,f records whether RB f is allocated to V2V pair k.When ak,f=1, RB f is allocated to pair k.Source: §II · p. 3
  • Because of high mobility, the BS and vehicles are assumed to know only large-scale channel gains, including path loss and shadowing.Blockage from nearby vehicles and buildings can place channels in line-of-sight or non-line-of-sight conditions.Source: §II · p. 3

1) I-VUEs:

  • For I-VUE uplinks, the SINR is determined by the desired I-VUE signal, noise, and interference from V2V pairs reusing the same RB.This SINR underlies the I-VUE rate calculation.Source: 1) I-VUEs · p. 3
  • The achievable I-VUE data rate is obtained from its SINR and the bandwidth W assigned to each RB.The rate expression is introduced after defining the per-RB bandwidth.Source: 1) I-VUEs · p. 3

2) V2V pairs:

Each V2V pair chooses direct V2V communication or indirect V2I forwarding through the BS. The two modes differ in interference structure, RB eligibility, and the SINR-based rate expressions used to evaluate their performance.

2) V2V pairs:

  • A V2V pair selects V2V or V2I mode according to its individual channel quality.The mode variable sk equals one for V2I mode and zero for V2V mode.Source: 2) V2V pairs · p. 3

V2V mode

  • In V2V mode, each transmitter communicates directly with its receiver, while interference comes from co-channel I-VUEs and V2V pairs.The mode-specific receiver SINR is defined on RB f.Source: 2) V2V pairs · p. 3
  • The achievable V2V-mode rate is expressed from the V2V receiver’s SINR on the selected RB.This rate accounts for the direct-link interference environment.Source: 2) V2V pairs · p. 3

V2I mode

  • In V2I mode, safety-critical data is uploaded to the BS and then forwarded through downlink to the V2V receiver.The paper assumes uplink SINR is lower than downlink SINR, so V2I performance is bounded by the uplink.Source: 2) V2V pairs · pp. 3–4
  • Only unused RBs can be assigned to V2I-mode V2V pairs, and each such RB can serve at most one V2I-mode pair.Interference comes from V2V-mode pairs sharing the same RB.Source: 2) V2V pairs · p. 4
  • The achievable V2I-mode rate is calculated from the mode-specific uplink SINR.This reflects the assumption that uplink performance limits the two-hop forwarding mode.Source: 2) V2V pairs · p. 4

B. QoS Requirements of I-VUEs and V2V Pairs

I-VUEs require minimum capacity to support bandwidth-demanding applications, while V2V pairs must satisfy latency and reliability constraints for safety-critical messages. The paper assumes common capacity, packet-size, latency, and outage-probability requirements within each corresponding user group.

C. Problem Formulation

The paper formulates joint mode selection, resource allocation, and power control as a constrained optimization problem, then recasts it as an MDP for decentralized DRL. Each V2V pair observes local state and selects a joint action to balance V2I capacity with V2V QoS.

C. Problem Formulation

  • The objective maximizes total I-VUE capacity while satisfying V2V latency and reliability requirements through joint mode, RB, and power decisions.Source: C. Problem Formulation · p. 4
  • The problem is a mixed integer nonlinear program because binary mode and allocation variables combine with non-convex power optimization.Source: C. Problem Formulation · p. 4
  • Centralized methods face global-CSI acquisition and high computational complexity, motivating decentralized approaches for dynamic large-scale vehicular networks.Source: C. Problem Formulation · p. 4

B. DRL-based Decentralized Algorithm

A. Basics of Deep Reinforcement Learning

B. DRL-based Decentralized Algorithm

IV. FEDERATED DRL-BASED SEMI-DECENTRALIZED ALGORITHM

Local DRL training is challenged by stringent latency requirements, limited training data, newly activated V2V pairs, and vehicle mobility. The proposed two-timescale federated DRL framework trains robust models and improves newly activated-pair performance.

  • Stringent latency requirements and limited training data make accurate local DRL training difficult.Source: §IV · p. 6
  • Poorly trained local models can cause newly activated V2V pairs to make inferior decisions and degrade global performance.Source: §IV · p. 6
  • Vehicle mobility can make well-trained DRL models outdated.Source: §IV · p. 6
  • The proposed two-timescale federated DRL framework trains robust DRL models and improves newly activated V2V-pair performance.Source: §IV · p. 6

A. Two-Timescale Federated DRL Framework

The framework combines federated learning with centralized vehicle clustering across two timescales. It addresses centralized-training privacy and communication costs while recognizing that local device-side training can be time-consuming and imprecise.

1) Basics of federated learning:

Federated DRL combines local model training with periodic model averaging, using large-timescale graph clustering to organize vehicles and reduce resource conflicts. Clusters are formed from large-scale channel gains, and candidate RB groups are assigned accordingly.

  • Federated learning separates local training on raw device data from infrequent centralized averaging of local models.This can enhance distributed DNN and DRL training while incurring negligible uploading overhead compared with centralized learning.Source: 1) Basics of federated learning · p. 6
  • The proposed framework operates through graph-based clustering on a large timescale and federated DRL procedures on a small timescale.The BS periodically constructs clusters and candidate RB groups before local models are averaged within clusters.Source: 2) Two-timescale federated DRL framework · pp. 6–7

Graph construction

  • Each graph vertex represents a V2V pair or I-VUE, with edges weighted by large-scale channel gains rather than Euclidean distances.Large-scale gains are used because nearby VUE links can be unreliable under blockage.Source: B. Centralized VUE Clustering on a… · p. 7

Graph clustering

  • The clustering task partitions the graph to maximize within-cluster edge weights under disjoint cluster-covering constraints.The resulting graph-partitioning problem is NP-hard, so spectral clustering is adopted instead of Euclidean-distance methods or randomly initialized heuristics.Source: B. Centralized VUE Clustering on a… · p. 7

RB assignment

  • Each cluster receives candidate RBs excluding resources allocated to its member I-VUEs, while same-cluster VUEs use orthogonal resources to mitigate interference.The candidate RB assignment is derived from the clustering results.Source: B. Centralized VUE Clustering on a… · p. 7

C. Federated DRL on a Small Timescale

On the small timescale, V2V pairs asynchronously train local DRL models and periodically aggregate them through cluster-specific federated averaging. Newly activated pairs receive an existing cluster model to avoid time-consuming local training.

  • At each coordination round, the BS distributes a pretrained or averaged model, and V2V pairs train local models using their local data.Local training follows the DRL procedure within each cluster.Source: C. Federated DRL on a Small Timescale · p. 7

Federated averaging

  • Local Q-network updates are aggregated into a cluster global model through mini-batch-based federated averaging.The update weights depend on the total batch size and each V2V pair’s training batch size.Source: C. Federated DRL on a Small Timescale · pp. 7–8

Asynchronous decisions

  • Asynchronous action selection assigns each V2V pair in a cluster to a specific subframe, reducing resource collisions caused by incomplete local observations.Each subframe block contains as many subframes as V2V pairs in the cluster.Source: C. Federated DRL on a Small Timescale · p. 8

Newly activated pairs

  • Newly activated V2V pairs request cluster assignment and download the corresponding global DRL model and network parameters.This avoids the time-consuming training of a local DRL model from scratch.Source: C. Federated DRL on a Small Timescale · p. 8

V. SIMULATION AND NUMERICAL RESULTS

The simulation evaluates the proposed DRL methods in a 1 km × 1 km crossroad scenario with randomly deployed I-VUEs and V2V links. Comparisons include centralized, decentralized DRL without mode selection, and random-selection algorithms.

Simulation setting

  • The evaluation uses a crossroad with a centrally located BS, vehicles generated by a spatial Poisson process, and urban-street channel parameters.The crossroad measures 1 km × 1 km, with two lanes in each direction.Source: §V · p. 8
  • Five active I-VUEs and K active V2V transmitters are selected, with each transmitter paired to the farthest vehicle within its broadcast range.LOS status, path loss, shadowing, and fast fading follow the 3GPP urban street scenario.Source: §V · p. 8

DRL configuration

  • The DQN uses a fully connected network with one hidden layer of 256 neurons, ReLU activation, and adaptive moment estimation.The parameters were selected through multiple simulation tests to balance DRL complexity and performance.Source: §V · p. 9

Baselines

  • The proposed methods are compared with a centralized algorithm, decentralized DRL without mode selection, and random RB selection.The centralized method uses global CSI, while the decentralized baselines rely on local decisions or random candidate-RB selection.Source: §V · p. 9

A. Network Performance versus the Number of V2V pairs

As V2V-pair density increases, the proposed DRL algorithm outperforms decentralized baselines on I-VUE sum capacity and V2V satisfied rate, while approaching centralized performance. As outage thresholds increase, both metrics decline for most evaluated methods, but adaptive mode selection remains advantageous.

Number of V2V pairs

  • The proposed DRL algorithm outperforms other decentralized algorithms in I-VUE sum capacity and V2V satisfied rate as the number of V2V pairs varies.Its advantage grows with more V2V pairs because it identifies unstable NLOS links and selects transmission modes using local observations.Source: A. Network Performance versus the… · p. 9
  • The proposed DRL algorithm achieves performance close to the centralized algorithm while making decisions from local observations.The centralized approach requires global CSI, high computational complexity, and frequent rescheduling for newly activated pairs.Source: A. Network Performance versus the… · p. 9

Outage threshold

  • As the outage threshold increases, I-VUE sum capacity and V2V satisfied rate decline for the centralized, proposed DRL, and DRL-without-mode-selection algorithms.Higher outage thresholds lead V2V pairs to use higher transmission power, increasing interference.Source: B. Network Performance versus Outage… · p. 9
  • The proposed DRL algorithm outperforms other decentralized algorithms across outage thresholds by adaptively selecting transmission modes to alleviate interference.The random-selection algorithm produces the same I-VUE sum capacity for different outage thresholds because it ignores I-VUE capacity when choosing RBs.Source: B. Network Performance versus Outage… · pp. 9–10

C. Effectiveness of the Federated DRL Algorithm

The federated DRL approach combines graph-based vehicle clustering with federated learning to train robust models and support newly activated V2V pairs. It converges toward centralized performance and provides stable performance with negligible training time for newly activated pairs.

Vehicle clustering

  • Graph-based clustering groups neighboring V2V pairs and I-VUEs with similar line-of-sight states into the same cluster.The method uses large-scale channel gain as graph-edge weights and applies spectral clustering.Source: C. Effectiveness of the Federated DRL… · p. 10

Federated learning

Newly activated V2V pairs

Loading 2002.05485v1…