PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Reinforcement Learning Based Resource Allocation for Dynamic V2X Communications

Yi Yuan, Gan Zheng, Kai-Kit Wong, Khaled B. Letaief

arXiv:2110.07734v1cs.ITmath.IT

TL;DR

Shared-spectrum V2I/V2V allocation is difficult when vehicular environments change and existing DRL quantizes continuous power. The paper combines DQN, DDPG, and meta-learning; simulations report improved V2I sum-rate performance over quantized-power DQN and fast adaptation to changed environments.

  • Problem

    Existing V2X resource-allocation methods face quantization of continuous actions and train–test mismatch as vehicular environments change.

    Source: §I · pp. 1–2, B. DRL-based Decentralized Algorithm · p. 5

  • Method

    The paper combines DQN for sub-band assignment with DDPG for continuous power allocation, then adds MAML-based meta-training for adaptation to new environments.

    Source: §I · p. 2, B. Meta-training Stage · p. 7

  • Results

    Simulations report that proposed DRL achieves better V2I sum-rate performance than quantized-power DQN, while meta-RL adapts efficiently and quickly when the environment changes.

    Source: A. Highway case · p. 11, B. Rural case · p. 11

  • Takeaways & Limitations

    The proposed DRL approach addresses continuous power allocation, and meta-based DRL is reported to support fast adaptation in new environments.

    Source: §I · pp. 2–3

Abstract

from arXiv · show

This paper studies the allocation of shared resources between vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) links in vehicle-to-everything (V2X) communications. In existing algorithms, dynamic vehicular environments and quantization of continuous power become the bottlenecks for providing an effective and timely resource allocation policy. In this paper, we develop two algorithms to deal with these difficulties. First, we propose a deep reinforcement learning (DRL)-based resource allocation algorithm to improve the performance of both V2I and V2V links. Specifically, the algorithm uses deep Q-network (DQN) to solve the sub-band assignment and deep deterministic policy-gradient (DDPG) to solve the continuous power allocation problem. Second, we propose a meta-based DRL algorithm to enhance the fast adaptability of the resource allocation policy in the dynamic environment. Numerical results demonstrate that the proposed DRL-based algorithm can significantly improve the performance compared to the DQN-based algorithm that quantizes continuous power. In addition, the proposed meta-based DRL algorithm can achieve the required fast adaptation in the new environment with limited experiences.

I. INTRODUCTION

The introduction motivates shared-spectrum resource allocation for V2I and V2V links in dynamic vehicular environments, then presents DRL approaches and formulates the allocation objective.

I. INTRODUCTION

  • Traditional optimization struggles with rapidly changing channels, low-overhead CSI acquisition, fast decisions, and mathematically exact V2V reliability requirements.Existing DRL approaches also quantize continuous actions, causing quantization error and increased computational complexity, and assume training and testing environments do not change.Source: §I · pp. 1–2
  • The proposed joint DRL algorithm uses DQN for discrete sub-band assignment and DDPG for continuous power allocation on shared V2I/V2V resources.The V2V links share sub-bands preassigned to V2I links.Source: §I · p. 2
  • The meta-based DRL algorithm incorporates MAML and uses two-level meta-training to learn a generalizing policy that adapts to new environments in a few time steps.The proposed approach targets the off-policy setting.Source: §I · p. 2

II. SYSTEM MODEL AND PROBLEM FORMULATION

  • The system model has M uplink V2I links and K V2V links; V2I links carry high-data-rate information while V2V links deliver safety-critical messages.Each V2I link has a preassigned orthogonal sub-band, and V2V links may share those bands.Source: §II · p. 3
  • The resource-allocation problem jointly maximizes V2I sum rate and reliable V2V payload delivery across sequential time slots, and is described as NP-hard.The paper formulates it as a multi-objective optimization problem and uses DRL to address continuous power constraints and environment mismatch.Source: §II · p. 4

III. DRL-BASED RESOURCE ALLOCATION ALGORITHM

The algorithm formulates resource management in a stationary environment as a DRL problem. Each V2V pair acts as an agent making sub-band and power-allocation decisions.

  • The proposed DRL algorithm targets resource management in a stationary environment, focusing on V2V sub-band assignment and power allocation.Source: §III · p. 4
  • Fig. 2 is identified as the basic DRL architecture for V2X communications.Source: Figure 2 · p. 4
  • Each V2V pair acts as an agent that makes sub-band and power-allocation decisions at each time slot.Source: §III · p. 4

A. Key Elements for MDP

The MDP represents each V2V agent's environment state, actions, and reward. Its reward combines link-rate and transmission-time considerations, while cumulative reward discounts future rewards.

A. Key Elements for MDP

  • Each V2V agent's state includes channel information, received interference, neighbors' sub-band use, remaining load, and remaining time.Source: A. Key Elements for MDP · pp. 4–5
  • Agents choose sub-band assignments and transmission power, with each V2V link limited to one sub-band at a time.Source: A. Key Elements for MDP · p. 5
  • The immediate reward combines V2I and V2V sum rates with a transmission-time penalty tied to the latency limit.Source: A. Key Elements for MDP · p. 5
  • The cumulative discounted reward balances future and current rewards using a discount factor, and equals the immediate reward when γi = 0.Source: A. Key Elements for MDP · p. 5

B. DRL-based Decentralized Algorithm

The decentralized algorithm handles each V2V agent's discrete sub-band and continuous power actions with separate DRL methods. Agents interact with the environment to receive rewards and new states.

B. DRL-based Decentralized Algorithm

1) DQN for Sub-band Assignment:

DQN handles the discrete sub-band assignment by estimating action values. Its target network supports learning stability, and an ϵ-greedy policy balances exploration and exploitation.

1) DQN for Sub-band Assignment:

2) DDPG for Power Allocation:

DDPG handles continuous power allocation after a V2V agent selects a sub-band. Network updates use replayed experiences to train the DQN and DDPG units and maintain target networks.

2) DDPG for Power Allocation:

  • DDPG uses an actor network to generate a deterministic continuous power action from the agent’s observed state after its sub-band is selected.A critic evaluates the state-action reward, and stochastic noise is introduced to balance exploration and exploitation.Source: 2) DDPG for Power Allocation · p. 6

3) Network Update for Both Units:

  • Random mini-batches from replay memory support training with independently and identically distributed data, while gradient descent updates the Q, critic, and actor networks.The Q-network target is periodically copied, while DDPG target networks use a fractional weight update.Source: 3) Network Update for Both Units · pp. 6–7
  • Algorithm 1 repeats action selection, reward and next-state collection, experience storage, and network updates for each V2V agent.Its convergence depends on offline training.Source: 3) Network Update for Both Units · p. 7

IV. META-REINFORCEMENT LEARNING

The stationary-system DRL approach assumes the environment and QoS requirements stay unchanged between training and testing, an assumption the paper describes as impractical for real-time V2X. The proposed MAML-based meta-RL approach targets generalization to new environments and tasks in an off-policy resource-allocation setting.

IV. META-REINFORCEMENT LEARNING

  • The earlier DRL algorithm addresses sub-band assignment and power allocation in a stationary V2X system, but assumes training and testing conditions remain the same.The paper notes that this assumption is impractical in real-time V2X, where a different testing distribution can create mismatch.Source: §IV · p. 7
  • The proposed meta-RL algorithm incorporates MAML into the joint RL framework to provide a model intended to generalize to new environments and tasks.The paper adapts the MAML idea to its off-policy resource-allocation problem because the cited MAML-based RL approach focuses on on-policy MDPs.Source: §IV · p. 7

A. Definitions

The meta-learning setup treats different vehicle initial positions as distinct resource-allocation tasks. Meta-training combines task-specific learning from a shared initialization with global updates informed by task query sets.

A. Definitions

  • The meta task set contains NT tasks, each modeled as an MDP for resource allocation in an environment with different vehicle initial positions.Each task has a replay buffer, and its support and query sets are used for global-network and task-network weight updates, respectively.Source: A. Definitions · p. 7

B. Meta-training Stage

  • Meta-training uses individual-level task updates and periodic global-level updates to learn an initialization that can adapt quickly to a new task.Each task updates its own parameters from the globally shared initialization using the DQN, actor, and critic networks.Source: B. Meta-training Stage · p. 7
  • Each task’s individual update samples replay experiences and optimizes its DQN, critic, and actor parameters independently from other tasks.At the first iteration, task parameters start from the corresponding global parameters, then use their own previous-step parameters.Source: B. Meta-training Stage · p. 8
  • The global update aggregates task adaptation losses evaluated on query sets to optimize global parameters, requiring an additional backward pass relative to the DRL algorithm.The task gradients are combined through the chain rule.Source: B. Meta-training Stage · p. 8

C. Meta-adaptation Stage

Meta-adaptation initializes a new task from trained global parameters and updates them using experiences generated in the new environment. The resulting policy is evaluated in testing, while convergence depends on offline training.

C. Meta-adaptation Stage

  • The new-task parameters are initialized from trained global parameters and adapted in a few steps using experiences generated in the new environment.The experiences are stored in an adaptation replay memory, and the adapted policy is then evaluated in testing.Source: C. Meta-adaptation Stage · p. 8
  • Algorithm 2 alternates meta-training over sampled tasks with an adaptation phase that starts from the learned global parameters.The adaptation phase runs for TAp time slots, and the algorithm’s convergence depends on offline training.Source: C. Meta-adaptation Stage · pp. 8–9

V. SIMULATION RESULTS

The simulations use an urban-trained single-cell V2X setup and evaluate resource-allocation solutions across urban, highway, and rural scenarios.

V. SIMULATION RESULTS

  • The simulator follows a 3GPP V2X setup and uses five-layer DQN and DDPG networks with three hidden layers of 500, 250, and 120 neurons.The reported setup includes four V2I links, 2-GHz carrier frequency, and a 100-ms maximum tolerated V2V latency.Source: §V · p. 9
  • The urban scenario is the training baseline; highway tests use a 120-km/h LOS freeway, while rural tests use WINNER II propagation and 108-km/h vehicles.The urban setup uses a Manhattan layout with LOS and NLOS fading and vehicles at 50 km/h.Source: §V · p. 9
  • The proposed DRL solution is the matched-environment upper bound, while meta-DRL measures fast adaptation and no-adaptation tests urban-trained policies in new environments.The comparison also includes DQN with five power quantization levels and a solution that randomly selects sub-band and transmit power.Source: §V · pp. 9–10
  • Fig. 4 tracks adaptive-sample performance by V2I sum rate and V2V transmission-failure probability.The benchmark comparison solutions include DQN, no-adaptation, and random policies.Source: §V · p. 9, Figure 4 · p. 10

A. Highway case

In the highway test, meta-DRL approaches matched-environment DRL performance with far fewer adaptation samples; performance also varies with user load and metric.

A. Highway case

  • With 40 adaptation samples, meta-DRL generates V2I sum rate very close to matched-environment DRL, versus more than 3000 samples for training from scratch.Meta-DRL also significantly reduces V2V failure probability compared with no-adaptation; the proposed DRL outperforms DQN and random solutions on both links.Source: Figure 5 · p. 10, A. Highway case · p. 11
  • V2I sum rate rises and V2V failure probability falls as adaptive samples increase; meta-RL beats no-adaptation above 30 and 40 samples, respectively.The simulations use 40 samples to balance overhead and performance.Source: A. Highway case · p. 10
  • As user count increases, V2I sum rate decreases and V2V transmission-failure probability increases across all compared solutions.The passage attributes this pattern to more V2V links sharing fixed sub-bands and causing greater interference.Source: A. Highway case · p. 11

B. Rural case

The proposed DRL algorithm outperforms DQN for both V2I and V2V links, while meta-based DRL generalizes and adapts quickly in changing environments. The rural and urban results examine performance across scenarios, vehicle counts, and V2V payload sizes.

B. Rural case

  • In the rural case, the proposed RL algorithm achieves the best V2I sum rate, and DRL and meta-DRL reduce V2V failure probability, especially with many vehicles.Meta-DRL provides fast adaptation when the communications environment changes; random resource selection performs worst due to increased interference.Source: B. Rural case · p. 11, Figure 6 · p. 11

C. Urban case

  • As vehicle counts increase in the urban case, V2I sum rate falls across solutions, while meta-DRL results remain closer to DRL than non-adaptation results.The passage reports the same expected pattern for V2V performance; the baseline is trained with four vehicles.Source: C. Urban case · p. 11, Figure 7 · p. 12
  • As V2V payload grows in the urban case, V2I sum rate falls and V2V failure probability rises; DRL retains higher V2I sum rate and lower V2V failure probability than DQN.The passage attributes the trend to longer transmission duration and higher V2V transmit power, which increase interference; meta-DRL also shows effectiveness under mismatch.Source: C. Urban case · p. 12, Figure 8 · p. 12

VI. CONCLUSIONS

  • The DRL algorithm outperforms DQN on both V2I and V2V links, while meta-based DRL generalizes and adapts quickly to new environments with limited experience.The conclusion identifies multi-antenna V2X and imperfect CSI as future directions, and proposes the algorithms as a framework for other mismatch-prone communication scenarios.Source: §VI · pp. 12–13
Loading 2110.07734v1…