PaperlayerView exact revision on arXiv

Source-linked AI summary

Decentralized Computation Offloading for Multi-User Mobile Edge Computing: A Deep Reinforcement Learning Approach

Zhao Chen, Xiaodong Wang

arXiv:1812.07394v1cs.LGeess.SPmath.OCstat.ML

TL;DR

Multi-user MEC offloading must minimize long-term computation cost despite stochastic tasks and changing wireless channels. The paper proposes decentralized DDPG policies that allocate local-execution and offloading power from local observations, and reports lower computation cost than DQN and greedy strategies.

  • Problem

    The paper addresses decentralized dynamic offloading for multi-user MEC, where stochastic tasks and time-varying channels make minimizing long-term power consumption and buffering delay challenging.

  • Method

    It uses continuous-action DDPG to let each mobile user independently learn dynamic power-allocation policies from local MEC observations.

  • Results

    The learned decentralized DDPG strategy achieves reduced computation cost and outperforms conventional DQN-based discrete power control and greedy strategies.

  • Takeaways & Limitations

    Efficient decentralized offloading policies can be learned independently at mobile users, with power-delay tradeoffs analyzable for DDPG and DQN.

Abstract

from arXiv · show

Mobile edge computing (MEC) emerges recently as a promising solution to relieve resource-limited mobile devices from computation-intensive tasks, which enables devices to offload workloads to nearby MEC servers and improve the quality of computation experience. Nevertheless, by considering a MEC system consisting of multiple mobile users with stochastic task arrivals and wireless channels in this paper, the design of computation offloading policies is challenging to minimize the long-term average computation cost in terms of power consumption and buffering delay. A deep reinforcement learning (DRL) based decentralized dynamic computation offloading strategy is investigated to build a scalable MEC system with limited feedback. Specifically, a continuous action space-based DRL approach named deep deterministic policy gradient (DDPG) is adopted to learn efficient computation offloading policies independently at each mobile user. Thus, powers of both local execution and task offloading can be adaptively allocated by the learned policies from each user's local observation of the MEC system. Numerical results are illustrated to demonstrate that efficient policies can be learned at each user, and performance of the proposed DDPG based decentralized strategy outperforms the conventional deep Q-network (DQN) based discrete power control strategy and some other greedy strategies with reduced computation cost. Besides, the power-delay tradeoff is also analyzed for both the DDPG based and DQN based strategies.

I. INTRODUCTION

The paper studies decentralized dynamic computation offloading for multi-user MEC under stochastic task arrivals and time-varying wireless channels. It uses DDPG to learn continuous local and offloading power policies from local observations, reducing computation cost relative to DQN and greedy strategies.

  • Problem: Multi-user MEC offloading must jointly manage radio and computation resources under stochastic arrivals and time-varying wireless channels.These dynamics make long-term control challenging, especially when users have limited resources and wireless transmission affects offloading efficiency.
  • Research gap: Existing work mainly studies centralized DRL, while decentralized DRL for dynamic offloading in multi-user MEC remains unknown.This gap motivates independently learned policies that avoid requiring a centralized controller.
  • System and learning setting: Each mobile user independently learns a dynamic offloading policy from local MEC observations without prior network-statistics knowledge.The considered system has one BS with an attached MEC server, multiple users, stochastic task arrivals, and time-varying channel conditions.
  • Proposed approach: DDPG provides continuous-action control for adaptive power allocation between local execution and computation offloading.The approach is designed for continuous power control rather than the discrete action spaces used by other DRL policies.
  • Evaluation: The decentralized DDPG strategy learns efficient policies and outperforms conventional DQN-based discrete power control and greedy strategies with reduced computation cost.The study also analyzes the power-delay tradeoff for DDPG and DQN strategies.

II. PRELIMINARIES ON DEEP REINFORCEMENT LEARNING

This section introduces reinforcement-learning preliminaries and motivates DDPG as a DRL extension for continuous action spaces.

  • Preliminaries: The section reviews MDPs, reinforcement learning, and deep reinforcement learning before presenting DDPG.It positions DDPG as the continuous-action component of the paper’s learning framework.

A. MDP

An MDP models sequential interaction between an agent and an environment, while RL learns policies from experience when transition dynamics are unknown. Value functions and Bellman relations provide the basis for policy evaluation and improvement.

  • MDP components: An MDP comprises an agent, environment, state space, action space, reward function, and transition dynamics.At each time step, the agent observes a state, selects an action, receives a reward, and transitions to a subsequent state.
  • Policies and returns: A policy maps states to actions or action distributions, while value and state-action functions quantify expected discounted returns.The discount factor γ weights future rewards in the return objective.
  • Bellman relations: Bellman equations express recursive relationships for value functions and state-action functions, including the Bellman optimality equation.These recursions support deriving optimal policies from value estimates.
  • DP versus RL: Dynamic programming can obtain optimal value functions when the environment model is known, whereas RL learns through interactions without an explicit model.RL adapts behavior using observed outcomes when transition probabilities are unknown or non-stationary.
  • Q-learning: Q-learning updates state-action estimates from experience using a temporal-difference error and can derive an optimal policy from the learned Q-values.The update uses experience tuples containing the current state, action, reward, and next state.

C. DRL

DRL methods use deep neural networks to handle high-dimensional reinforcement-learning problems, while DDPG extends this framework to continuous action spaces through an actor–critic design. DDPG learns deterministic continuous actions and supports exploration through added noise.

  • DQN: DQN uses a deep neural network to approximate Q-values, with experience replay improving stability during function-approximation-based reinforcement learning.Experiences are stored in a replay buffer and sampled uniformly for mini-batch updates.
  • Stabilization: Target networks in DQN and DDPG track learned networks through soft updates to improve reinforcement-learning stability.The DQN target parameters use θ′ ←τθ + (1 −τ)θ′ with τ ≪1; DDPG uses corresponding actor and critic target networks.
  • DDPG: DDPG extends deep reinforcement learning to continuous action spaces using separate actor and critic neural networks.The critic approximates Q(s, a|θQ), while the actor deterministically maps state s to a continuous action.
  • DDPG: The DDPG critic follows DQN-style updates, while the actor is optimized using the policy gradient of the expected return.Actor and critic parameters are updated with separate learning rates.
  • Exploration: DDPG separates exploration from learning by adding noise to the deterministic actor policy, addressing exploration challenges in continuous action spaces.Temporally correlated noise can preserve momentum during exploration.

III. DYNAMIC COMPUTATION OFFLOADING FOR MOBILE EDGE COMPUTING

The paper models a multi-user MEC system with time-varying channels and task arrivals, where each user balances local execution and offloading to manage energy consumption and processing delay. The uplink uses multi-user MIMO with zero-forcing detection, and increasing user count worsens each user's SINR.

  • System model: The MEC system contains a base station, an MEC server, and multiple mobile users with computation-intensive tasks.The server is deployed near the base station so users can offload computation through wireless links.
  • Dynamic offloading: The system uses a slotted-time model with equal slot length, and decentralized scheduling becomes more favorable as the number of users increases.The stated motivation is to reduce system overhead and improve MEC scalability.
  • Dynamic offloading: Each user chooses the ratio of local execution to computation offloading in every slot to balance average energy consumption and task-processing delay.Channel conditions and task arrivals vary across slots, motivating dynamic decisions.
  • Network model: The base station uses linear zero-forcing detection for uplink transmissions from multiple single-antenna users.The model assumes more base-station antennas than users, and ZF approaches MMSE performance as antenna count increases.
  • Channel model: The channel model uses Gaussian Markov block fading with temporal correlation represented by ρm and complex Gaussian innovation noise.The correlation coefficient is related to Doppler frequency through Jake's fading spectrum.
  • Network model: Each user's SINR becomes worse as the number of users M increases, requiring more offloading power to maintain transmission performance.Users adapt to the environment using SINR feedback.

B. Computation Model

The computation model lets each user split fine-grained tasks between local execution and MEC-server processing. Queue evolution accounts for processed bits and new arrivals, while local and offloaded processing depend on allocated powers and system assumptions.

  • Task model: Task arrivals a_m(t) are i.i.d. across time with mean rate λ_m = E[a_m(t)].Arrivals during slot t can be processed starting from slot t + 1.
  • Task model: Fine-grained tasks can be divided between locally processed bits d_l,m(t) and bits d_o,m(t) offloaded to the MEC server.This split enables each user to combine local execution with edge computing.
  • Task model: The task-buffer queue evolves by subtracting local and offloaded processed bits from the previous backlog and adding new arrivals.The recurrence is B_m(t + 1) = [B_m(t) −(d_l,m(t) + d_o,m(t))]+ + a_m(t).
  • Local computing: Local processing is determined by allocated local execution power p_l,m(t) within [0, P_l,m], CPU cycles per task bit L_m, and DVFS-based CPU frequency control.The effective switched capacitance κ depends on chip architecture.
  • Edge computing: The MEC server is assumed to have sufficient resources, allowing applications to be processed in parallel with negligible processing latency and ignored feedback delay.Under this assumption, all task bits offloaded through the base station are processed.
  • Edge computing: Offloaded data bits depend on the user's uplink transmission power p_o,m(t), system bandwidth W, and channel-related SINR γ_m(t).The offloading amount is derived from the uplink transmission model.

IV. DRL BASED DECENTRALIZED DYNAMIC COMPUTATION OFFLOADING

The paper develops a decentralized, model-free DDPG offloading framework in which each user independently allocates local-execution and offloading power from local observations. The reward minimizes a weighted combination of energy consumption and buffering delay while avoiding centralized system-state collection.

  • Decentralized DRL framework: DDPG independently learns each user’s dynamic offloading policy from local observations, without prior knowledge of user counts, task arrivals, or wireless-channel statistics.The resulting online learning process is model-free and decentralized.
  • State space: Local observations include queue lengths, channel information, and feedback such as the previous-slot SINR and upcoming channel estimate.Zero-forcing detection also supplies a projected received-power ratio for the user’s uplink signal.
  • Action space: Each user selects local-execution and computation-offloading powers as its action at every time slot.The action is based on the user’s observed system state.
  • Action space: DDPG optimizes both power allocations in a continuous action space rather than selecting from predefined discrete power levels.This significantly reduces the high dimension of discrete action spaces.
  • Reward function: Adjusting the reward weights creates a tradeoff between energy consumption and buffering delay.The weighting factors can be tuned to emphasize either objective.
  • Reward function: The reward penalizes instantaneous total power consumption and task-buffer queue length through nonnegative weights.By Little’s Theorem, average queue length is proportional to buffering delay.

B. Training and Testing

The training and testing framework uses a simulated MEC environment and independently trained DDPG agents. Agents interact through bounded episodes during training, store experiences in local replay buffers, and later load learned actor parameters for testing.

  • Training: Because the interaction is a continuing reinforcement-learning task, training episodes begin from random states and terminate after a predefined maximum number of steps.This provides episode boundaries for exploration.
  • Framework: The framework consists of a simulated environment and a group of user agents for training and testing the decentralized policies.The environment generates data that mimics agent–MEC interaction.
  • Framework: The environment accepts each agent’s decision and returns CSI and SINR feedback.This feedback supports subsequent local observations and actions.
  • Training: Each agent stores state–action–reward–next-state tuples in its own replay buffer and updates its actor and critic networks using sampled mini-batches.After training, the policy is learned independently at each user agent.
  • Testing: During testing, each agent loads its trained actor parameters, starts with an empty data buffer, and interacts with the environment using local observations.Testing follows the learned policy rather than continuing the training procedure.

A. Simulation Setup

The simulations specify channel, computing, neural-network, exploration, and reward-weight settings for evaluating decentralized DDPG offloading. Training uses 2000 episodes of 200 steps, with DQN and greedy strategies included as baselines.

  • System parameters: The simulation uses 1 ms time slots, a 1 MHz system bandwidth, and maximum offloading power of 2 W.The channel model includes temporal correlation coefficient ρ_m = 0.95 and Doppler frequency f_d,m = 70 Hz.
  • System parameters: Local execution is parameterized by κ = 10^-27, 500 CPU cycles per bit, and maximum CPU frequency 1.26 GHz.These settings imply a maximum local-execution power of 2 W.
  • DDPG configuration: Each DDPG actor and critic is a four-layer fully connected network with hidden-layer widths 400 and 300.Hidden layers use ReLU activations, while the actor output uses a sigmoid layer to bound actions.
  • Reward configuration: The reward tradeoff is controlled by w_m ∈ [0,1], with w_m,1 = 10^w_m and w_m,2 = 1 − w_m.This single factor adjusts the relative emphasis on energy consumption and buffering delay.

1) Greedy Local Execution First (GD-Local):

The evaluation compares greedy execution priorities with DDPG and DQN-based dynamic offloading in single-user simulations. Both DRL methods learn from interaction without prior knowledge, while DDPG performs better across the tested scenarios.

  • Greedy Local Execution First (GD-Local): The GD-Local strategy executes as many buffered task bits locally as possible before offloading the remainder to the MEC.It prioritizes local execution within each time slot.
  • Greedy baselines: The comparison also considers a strategy that first offloads as much data as possible and then locally executes the remaining buffered bits.This reverses the priority used by GD-Local.
  • DRL baseline: DQN is implemented as a discrete-action DRL baseline with L = 8 power levels for both local execution and offloading.Each user selects from the Cartesian product of the two power-level sets.
  • Training results: In single-user tests, average reward increases with continued interaction for both DDPG and DQN, indicating that efficient policies can be learned without prior knowledge.The experiments average results over 10 simulation runs.
  • Training results: Both learned policies become stable after about 1500 episodes.This stabilization is reported for the tested task-arrival scenarios.
  • Performance comparison: DDPG’s learned policy is always better than DQN’s across the tested scenarios, demonstrating more efficient exploration for continuous control.The comparison is reported for different task-arrival-rate cases.

2) Testing:

Testing evaluates learned policies across task arrival rates and power-delay weights. DDPG generally achieves the strongest reward and balances lower power consumption against buffering delay.

  • Testing results: Average reward increases as task arrival rate grows, indicating higher computation cost under greater demand.The increased cost results from higher power consumption and longer buffering delay.
  • Testing results: For w1 = 0.5, DDPG outperforms both greedy strategies but slightly compromises buffering delay to achieve the lowest energy consumption.DQN exceeds the greedy strategies in average reward, partly because its discrete power levels are limited.
  • Testing results: For w1 = 0.8, DDPG outperforms all other strategies, with a larger reward gap than at w1 = 0.5.The larger weight imposes more penalty on power consumption in the reward function.
  • Power-delay tradeoff: Increasing w1 reduces power consumption while sacrificing delay performance, enabling tuning for minimum power under a delay constraint.Across each w1 value, DDPG achieves better power consumption and buffering delay than the compared strategies.

1) Training:

Training shows that decentralized policies can be learned for heterogeneous multi-user MEC systems. The learned DDPG strategy reduces computation cost through continuous power control, while preserving a power-delay tradeoff.

  • Training: For each mobile user, average reward increases gradually as interaction with the MEC system continues over more episodes.This indicates that decentralized policies can be learned for heterogeneous users with different computation demands.
  • Training: Higher computation demand requires higher computation cost, while multi-user rewards are lower than in the single-user scenario.Serving more users degrades spectral efficiency and requires more power.
  • Multi-user evaluation: For wm = 0.5, DDPG gives users 2 and 3 better average rewards than the other strategies, but is slightly worse than GD-Local for user 1.The authors identify insufficient exploration at small allocated power as an area for improvement.
  • Multi-user evaluation: With wm = 0.8, DDPG obtains the best average reward for each user and widens its gap over greedy strategies.The stronger power penalty lowers consumed power but moderately increases buffering delay compared with wm = 0.5.
  • Multi-user evaluation: Selecting a proper wm can minimize multi-user power consumption while maintaining a satisfactory average buffering delay.DDPG-based strategies outperform DQN-based strategies in average reward for all users.
  • Conclusion: The study targets long-term computation cost in stochastic, time-varying multi-user MEC systems using decentralized dynamic offloading.DDPG learns per-user policies that adapt local-execution and offloading powers from local observations.
  • Conclusion: Numerical results report lower computation cost for DDPG than conventional DQN discrete power control and greedy strategies, alongside power-delay analysis.The comparison concerns the proposed decentralized strategy and the conventional discrete-power baseline.
Loading 1812.07394v1…