Source-linked AI summary
Decentralized Power Allocation for MIMO-NOMA Vehicular Edge Computing Based on Deep Reinforcement Learning
Hongbiao Zhu, Qiong Wu, Xiaojun Wu, Qiang Fan, Pingyi Fan, Jiangzhou Wang
TL;DR
Vehicular edge computing must allocate power between offloading and local execution under stochastic task arrivals and uncertain MIMO-NOMA channels. The paper uses decentralized DRL with local observations and DDPG to learn power allocation. Simulations report that the learned policy outperforms existing policies.
Problem
Stochastic task arrivals and MIMO-NOMA channel uncertainty from interference and vehicle mobility affect task-processing power consumption and latency.
Source: §I · p. 2Method
The paper formulates decentralized DRL power allocation using local observations as state and applies DDPG to learn the policy.
Source: §I · p. 3Results
Simulations show that the learned policy outperforms existing policies in power consumption and task-processing latency.
Source: §I · p. 3, §VII · p. 23Takeaways & Limitations
The learned policy emphasizes power consumption while compromising buffer length relative to GD-Local and GD-Offload policies across different task-arrival rates.
Source: §VII · pp. 23–24
Abstract
from arXiv · showhide
Vehicular edge computing (VEC) is envisioned as a promising approach to process the explosive computation tasks of vehicular user (VU). In the VEC system, each VU allocates power to process partial tasks through offloading and the remaining tasks through local execution. During the offloading, each VU adopts the multi-input multi-out and non-orthogonal multiple access (MIMO-NOMA) channel to improve the channel spectrum efficiency and capacity. However, the channel condition is uncertain due to the channel interference among VUs caused by the MIMO-NOMA channel and the time-varying path-loss caused by the mobility of each VU. In addition, the task arrival of each VU is stochastic in the real world. The stochastic task arrival and uncertain channel condition affect greatly on the power consumption and latency of tasks for each VU. It is critical to design an optimal power allocation scheme considering the stochastic task arrival and channel variation to optimize the long-term reward including the power consumption and latency in the MIMO-NOMA VEC. Different from the traditional centralized deep reinforcement learning (DRL)-based scheme, this paper constructs a decentralized DRL framework to formulate the power allocation optimization problem, where the local observations are selected as the state. The deep deterministic policy gradient (DDPG) algorithm is adopted to learn the optimal power allocation scheme based on the decentralized DRL framework. Simulation results demonstrate that our proposed power allocation scheme outperforms the existing schemes.
I. INTRODUCTION
The paper addresses power allocation in MIMO-NOMA vehicular edge computing under stochastic task arrivals and changing channel conditions. It proposes decentralized deep reinforcement learning and reports better power-consumption and latency performance than existing policies.
I. INTRODUCTION
- MIMO-NOMA lets vehicles share the full spectrum, but interference, mobility-driven path-loss, and stochastic task arrivals affect processing power and latency.The introduction motivates optimizing both metrics because vehicle equipment has limited energy and some applications have tight processing deadlines.Source: §I · pp. 2–3
- The formulation defines state, action, and reward from local observations, and DDPG learns each vehicle user's power-allocation decision.The framework targets the long-term reward involving power consumption and latency.Source: §I · p. 3
- The proposed decentralized DRL power-allocation policy outperforms existing policies in power consumption and task-processing latency.It targets long-term reward under stochastic arrivals and channel uncertainty caused by MIMO-NOMA interference and vehicle mobility.Source: §I · p. 3
A. Offloading in MIMO or NOMA MEC
- Prior MIMO- or NOMA-based MEC studies optimized energy, latency, or system cost, but did not address vehicular scenarios.The cited examples use optimization methods for NOMA and MU-MIMO resource allocation.Source: A. Offloading in MIMO or NOMA MEC · p. 4
B. DRL-based Offloading in VEC
- Earlier decentralized DRL VEC studies used orthogonal SISO channels and did not consider channel variation from MIMO-NOMA interference and vehicle mobility.The paper identifies the joint setting of stochastic arrivals and varying MIMO-NOMA channels as unaddressed in decentralized DRL power allocation.Source: B. DRL-based Offloading in VEC · p. 5
III. SYSTEM MODEL
The model places resource-limited vehicle users on a multi-lane road served by a roadside base station and edge server. It represents mobility, channel variation, and local power allocation using per-slot observations.
III. SYSTEM MODEL
- A multi-antenna base station with an attached VEC server serves single-antenna vehicle users moving across a J-lane road in equal-duration slots.Each user can process buffered tasks locally or offload them to the server, and chooses power allocation from local information.Source: §III · pp. 5, 7
A. Mobility model
- Vehicle position is represented by its longitudinal distance from the base station and lane, with position treated as approximately constant within each slot.Vehicles move at constant lane-specific velocity, and longitudinal distance is updated each slot as a local mobility observation.Source: A. Mobility model · pp. 7–8
B. Network model
- The MIMO-NOMA channel combines vehicle-to-base-station channel vectors, and the base station uses a zero-forcing detector to obtain each user's SINR.Each user receives its SINR at the next slot, making the previous-slot SINR a local observation of channel variation.Source: B. Network model · pp. 8–10
- The network model includes path-loss and mobility effects in the channel, while each user's offloading power is bounded by a maximum.The channel incorporates small-scale fading and path-loss; the base station observes simultaneous vehicle transmissions.Source: B. Network model · pp. 8–9
C. Computation model
The computation model tracks each vehicle user's queued tasks from slot to slot as new tasks arrive and local or offloaded work departs.
C. Computation model
- The buffer at the next slot equals remaining tasks after local and offloaded processing, floored at zero, plus newly arrived tasks.The combined amount processed locally and offloaded cannot exceed the previous buffer length.Source: C. Computation model · p. 10
1) Local execution:
Local execution converts allocated power into CPU frequency and determines how much queued task data can be processed.
1) Local execution:
- Local processing capacity depends on task computation intensity and CPU frequency, which is determined by local execution power.The model bounds local power by its maximum and CPU frequency by a corresponding maximum.Source: 1) Local execution · p. 11
2) Offloading:
Offloading assumes sufficient VEC-server resources and negligible processing and feedback delays, leaving slot duration as transmission delay. The VU uses bandwidth and SINR to determine offloaded task amount and buffer observations.
2) Offloading:
- The VEC server is assumed to have sufficient computation resources, making its processing latency negligible.Feedback delay is also ignored, so transmission delay is one slot of duration τ0.Source: 2) Offloading · p. 11
- Offloaded task amount depends on bandwidth W and the VU’s SINR γm(t −1).The SINR is supplied by the BS, and the VU uses it with its allocated offloading power and task arrival to calculate its buffer observation.Source: 2) Offloading · p. 11
- The buffer length Bm(t) serves as a local observation reflecting both stochastic task arrivals and uncertain channel conditions.The VU calculates it using prior task-arrival and SINR information.Source: 2) Offloading · p. 11
IV. PROBLEM FORMULATION
The formulation treats each VU’s power allocation as a local-state decision under unknown stochastic task arrivals and channel conditions. Its state captures buffer, channel, and mobility information, while its action allocates local and offloading power.
IV. PROBLEM FORMULATION
- Each VU observes a local state, chooses an action under policy μ, receives a reward, and transitions to a next state.The stochastic task arrivals and uncertain channel conditions are unknown to each VU.Source: §IV · p. 11
A. State
- The local state is selected to reflect stochastic task arrivals, uncertain channel conditions, and VU mobility affecting power consumption and delay.This distinguishes the decentralized decision from centralized schemes that use global state.Source: A. State · p. 12
- The VU’s distance from the BS antennas along the x-axis represents its position and mobility.Source: A. State · p. 12
- The VU’s SINR reflects uncertain channel conditions and depends on its channel vector.Source: A. State · p. 12
- Buffer length Bm(t) reflects both task arrivals and channel uncertainty because it depends on prior arrival am(t−1) and SINR γm(t−1).Source: A. State · p. 12
- The state space is continuous because the task-arrival and channel-vector values are continuous random variables.Source: A. State · p. 12
B. Action
- Each VU’s action consists of its local-execution power and offloading power, chosen from its locally observed state.Source: B. Action · p. 12
- For fine-grained computation applications, local-execution and offloading power vary continuously within [0, Pm,l] and [0, Pm,o], respectively.The resulting action space is continuous.Source: B. Action · p. 12
C. Reward function
The reward formulation weights power consumption and buffer length as a proxy for delay, with the objective of maximizing expected long-term discounted reward. The solution uses DDPG to train a local policy through replayed transitions and then tests the learned policy.
C. Reward function
- The reward penalizes weighted power consumption and buffer length, linking the objective to power use and buffer delay.The server-processing and feedback delays are treated as negligible, while transmission delay is constant; buffer delay is proportional to average buffer length by Little’s Theorem.Source: C. Reward function · pp. 12–13
- Each VU seeks to maximize its expected long-term discounted reward over the slots it moves along its lane.The discount factor γ lies in [0, 1], and Nj is the upper slot-index limit for lane j.Source: C. Reward function · p. 13
- At each slot, a VU observes the changed local network state before choosing its action, supporting operation under a dynamic VEC network.Source: C. Reward function · p. 13
V. SOLUTION
- DDPG is used to learn the policy because the state and action spaces are continuous.The paper first describes training and then tests performance under the learned policy.Source: A. Training stage · p. 13, §V · p. 13
A. Training stage
- DDPG’s actor improves the policy and its critic evaluates it, with neural networks approximating the policy and action-value function.The training account uses target actor and critic networks to support algorithm stability.Source: A. Training stage · pp. 13–14
- Training stores state-action-reward transitions in a replay buffer and samples mini-batches to update the critic and actor.The algorithm also updates target networks; the described loop runs through episodes and slots until its stopping limits.Source: A. Training stage · pp. 14, 15, 17
- The critic approximates the action-value function with a neural network because the continuous action space prevents calculating it by the Bellman equation.The actor parameters are updated using a policy gradient based on the critic’s approximation.Source: A. Training stage · p. 15
B. Testing stage
Testing uses the optimized policy while omitting the critic and both target networks used during training.
B. Testing stage
- The testing stage applies the optimized policy parameter θm∗ without the critic, target actor, or target critic networks.Algorithm 2 provides the testing-stage pseudocode.Source: B. Testing stage · p. 17
C. Complexity Analysis
The analysis gives the DDPG framework’s computational complexity and outlines the testing-stage loop for applying the learned policy.
C. Complexity Analysis
- In testing, the VU receives an observation, generates power with the learned policy, executes the action, and observes reward and a new state.The algorithm resets the VEC simulation parameters at the start of testing.Source: C. Complexity Analysis · p. 18
- Per training slot, DDPG complexity is O(GA + GC + 2UA + 2UC), accounting for actor and critic gradients and parameter updates.Target networks update parameters without computing gradients.Source: C. Complexity Analysis · p. 18
- Across training, DDPG complexity is O((Kmax·Nj − I)(GA + GC + 2UA + 2UC)) after replay-buffer contents exceed I.Training and parameter updates are not activated until the stored tuples exceed I.Source: C. Complexity Analysis · p. 18
VI. SIMULATION RESULTS AND ANALYSIS
The simulation section specifies the DDPG experiment setup, including network architecture, optimization, exploration, task arrivals, and vehicle motion.
VI. SIMULATION RESULTS AND ANALYSIS
- Both actor and critic use four-layer fully connected DNNs with two hidden layers of 400 and 300 neurons.Adam updates both networks, with learning rates 10^-3 for the critic and 10^-4 for the actor.Source: §VI · p. 18
- Task arrivals at each slot follow a Poisson distribution with mean rate λm, and exploration noise follows an Ornstein-Uhlenbeck process.The OU process uses decay-rate 0.15 and variation 0.02.Source: §VI · p. 18
- The target VU moves on lane 2, while three other VUs on each of three lanes drive into BS coverage when dm(t) = 0.The small-scale fading initialization is hs_m(0) ∼ CN(0, I_N).Source: §VI · p. 18
- Table II lists the experiment parameter values, and Fig. 3 is identified as the learning curve.The supplied passages do not state Fig. 3’s plotted values or trend.Source: Table II · p. 19, Figure 3 · p. 19
A. Training Stage
The learned policy’s testing performance is compared with greedy local-execution-first and offload-first policies, following a training stage in which reward stabilizes.
A. Training Stage
- During training, average reward rises rapidly through episode 10, increases more slowly through episode 600, then stabilizes with slight jitter.The paper attributes the small jitter to exploration noise that slightly adjusts the policy.Source: A. Training Stage · p. 19
B. Testing Stage
- Across distance, the optimal policy shifts from local execution toward offloading as the VU approaches the BS, then shifts back when interference degrades the channel.Three other vehicles enter BS coverage at dm(t) = 0; the paper attributes the resulting interference to a sharp increase in local execution power.Source: B. Testing Stage · pp. 20–21, Figure 4 · p. 20
- At the interference event, GD-Offload’s buffer grows, whereas the optimal policy’s buffer fluctuates around mean task arrivals and its reward is usually higher.The paper attributes the optimal policy’s reward advantage to adaptive power allocation.Source: Figure 5 · p. 20, B. Testing Stage · p. 21
- Average power consumption falls 44.2% versus GD-Local and 63.0% versus GD-Offload under the optimal policy.Testing performance is averaged over 100 episodes; Fig. 6 summarizes average power and reward alongside buffer length.Source: B. Testing Stage · pp. 20, 22, Figure 6 · p. 22
- As task arrivals increase, the optimal policy outperforms both greedy policies in power consumption and long-term discounted reward, while its buffer length is slightly higher.All three policies’ discounted rewards decline as arrival rates rise; the optimal policy trades buffer length against power consumption in maximizing reward.Source: B. Testing Stage · p. 22, Figure 7 · p. 23
VII. CONCLUSIONS
The paper concludes that its decentralized DRL power-allocation policy outperforms existing policies in simulations under stochastic task arrivals and uncertain channels. The conclusions also describe how allocation shifts with vehicle distance from the BS and how the optimal policy balances power consumption against buffer length.
VII. CONCLUSIONS
- The proposed decentralized DRL policy outperforms the other existing policies in simulations.It maximizes long-term reward including power consumption and delay under stochastic task arrivals and channel uncertainty from MIMO-NOMA interference and VU mobility.Source: §VII · p. 23
- Closer to the BS, VUs use more offloading power and less local execution power; farther away, they use less offloading power and more local execution power.The conclusions associate this shift with processing more tasks as channel conditions vary with VU mobility.Source: §VII · p. 23
- Across different task arrival rates, the optimal policy emphasizes power consumption and compromises buffer length compared with GD-Local and GD-Offload.Source: §VII · pp. 23–24
- Future work will consider data freshness when designing VEC offloading schemes.Source: §VII · p. 24