PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance Optimization in Mobile-Edge Computing via Deep Reinforcement Learning

Xianfu Chen, Honggang Zhang, Celimuge Wu, Shiwen Mao, Yusheng Ji, Mehdi Bennis

arXiv:1804.00514v1cs.NI

TL;DR

Resource-constrained mobile devices need offloading decisions that adapt to changing network conditions, especially when multiple BSs create a large state space. The paper models the decision as an MDP and uses an online DQN-based policy. Experiments report minimum long-term cost versus three baselines, with up to 56% performance improvement.

  • Problem

    Resource-constrained mobile devices face a difficult choice between local execution and offloading as channel, energy, and task conditions change, while multiple BSs make the state space large.

    Source: §I · p. 1

  • Method

    The paper formulates offloading as an MDP and proposes an online DQN-based policy that learns without a priori knowledge of dynamic statistics.

    Source: §I · p. 1, §VI · p. 6

  • Results

    Compared with three baselines, the proposed algorithm achieves minimum long-term cost, with up to 56% performance improvement.

    Source: §VI · p. 6

  • Takeaways & Limitations

    The reported result indicates a tradeoff among computation execution delay, handover delay, and task-dropping cost.

    Source: §VI · p. 6

Abstract

from arXiv · show

To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is emerging as a promising paradigm by providing computing capabilities within radio access networks in close proximity. Nevertheless, the design of computation offloading policies for a MEC system remains challenging. Specifically, whether to execute an arriving computation task at local mobile device or to offload a task for cloud execution should adapt to the environmental dynamics in a smarter manner. In this paper, we consider MEC for a representative mobile user in an ultra dense network, where one of multiple base stations (BSs) can be selected for computation offloading. The problem of solving an optimal computation offloading policy is modelled as a Markov decision process, where our objective is to minimize the long-term cost and an offloading decision is made based on the channel qualities between the mobile user and the BSs, the energy queue state as well as the task queue state. To break the curse of high dimensionality in state space, we propose a deep $Q$-network-based strategic computation offloading algorithm to learn the optimal policy without having a priori knowledge of the dynamic statistics. Numerical experiments provided in this paper show that our proposed algorithm achieves a significant improvement in average cost compared with baseline policies.

I. INTRODUCTION

The paper addresses offloading decisions for resource-constrained mobile devices in dynamic, ultra-dense MEC networks. It models task execution and energy allocation and proposes an online DQN policy to handle the resulting state-space explosion.

I. INTRODUCTION

  • Resource-constrained batteries and CPUs make satisfactory QoS and QoE difficult for increasingly computation-intensive mobile applications.MEC places computing capabilities in radio access networks close to mobile users.Source: §I · p. 1
  • Earlier offloading policies relied on one-shot optimization or environmental statistics, while Lyapunov optimization produced an approximately optimal solution.The cited reinforcement-learning approach did not require a priori statistical knowledge, but the paper identifies conventional reinforcement learning as infeasible in ultra-dense networks.Source: §I · p. 1
  • The proposed online DQN-based policy addresses state-space explosion in ultra-dense MEC without requiring a priori knowledge of dynamic statistics.The setting includes multiple BSs with different transmission qualities and wireless-charging-enabled mobile devices.Source: §I · p. 1

II. SYSTEM MODEL AND ASSUMPTIONS

  • At each epoch, the mobile user can drop a task, execute it locally, or offload it through a BS, while allocating energy.The model considers time-varying channel quality, task arrivals, and energy status; it assumes energy is evenly assigned to the input bits for offloading and neglects return-transmission time.Source: §II · pp. 2–3

III. PROBLEM FORMULATION

The formulation represents the mobile user's task, energy, and channel conditions as a network state, then chooses offloading and energy-allocation actions. It minimizes expected discounted long-term cost, combining execution delay, handover delay, and task-dropping cost.

III. PROBLEM FORMULATION

  • The network state includes task-arrival, energy-queue, and BS channel conditions, and actions jointly select task execution and energy allocation.The execution choice may be dropping, local execution, or offloading through a BS.Source: §III · p. 3
  • Per-epoch task cost is a weighted sum of execution delay, handover delay, and task-dropping cost.The model adds handover delay when the mobile user switches to a different BS.Source: §III · p. 3
  • The objective is to choose a policy that minimizes expected discounted long-term cost from a given initial network state.The cost is evaluated over randomized network states and actions induced by the policy.Source: §III · p. 3

IV. SOLVING THE OPTIMAL CONTROL POLICY

The offloading optimization is a single-agent, infinite-horizon MDP with a discounted-cost objective. The section first considers the conventional MDP solution, then proposes a deep reinforcement-learning scheme using limited network-statistics information.

IV. SOLVING THE OPTIMAL CONTROL POLICY

  • The computation offloading optimization is a single-agent, infinite-horizon MDP with a discounted-cost criterion.The section proceeds from the conventional MDP framework to a deep reinforcement-learning scheme.Source: §IV · p. 3
  • The section first investigates the optimal solution within the conventional MDP framework.Source: §IV · p. 3
  • It then proposes a deep reinforcement-learning scheme with limited network-statistics information.Source: §IV · p. 3

A. Optimal MDP Solution

The optimal control problem is characterized through Bellman’s optimality equation, but conventional solution methods face exponential state-space growth and require complete network statistics. The section motivates a practically efficient scheme to approach the optimal policy.

A. Optimal MDP Solution

  • The optimal state-value function satisfies Bellman’s optimality equation, with the right-hand side of equation (15) defined as the optimal action-value function.For each state, the value equals the minimum action value.Source: A. Optimal MDP Solution · p. 3
  • Traditional value or policy iteration requires complete channel-transition, task-arrival, and energy-arrival statistics and has exponential computation complexity.The section therefore motivates a practically efficient scheme to approach the optimal policy.Source: A. Optimal MDP Solution · pp. 3–4
  • With 6 BSs, 8 channel-gain states per BS, and H = 4, the local network state space contains 2,621,440 states.The state space grows exponentially with the number of BSs.Source: A. Optimal MDP Solution · pp. 3–4

B. Deep Reinforcement Learning

The section first uses model-free Q-learning to avoid requiring dynamic network statistics, then adopts a DQN to address the large state space. Replay-based training and exploration support online learning of the offloading policy.

B. Deep Reinforcement Learning

  • Model-free Q-learning learns the optimal control policy without information about dynamic network statistics, but it still faces the huge state space.The section identifies state-space size as the remaining challenge for Q-learning.Source: B. Deep Reinforcement Learning · p. 4
  • The Q-learning update uses observed state, action, execution cost, arrivals, and the resulting next state to update the Q-function online.The learning rate α_k is time-varying.Source: B. Deep Reinforcement Learning · p. 4
  • The algorithm uses an ǫ-greedy strategy to balance exploration and exploitation, while Q-learning convergence requires all state-action pairs to be visited infinitely often.The text says this visitation condition can be satisfied when every action has non-zero selection probability in every network state.Source: B. Deep Reinforcement Learning · p. 4
  • The DQN approximates the Q-function with a neural network and trains from randomly sampled mini-batches stored in a finite replay memory.A second weight set evaluates action values and is periodically updated from the main weights.Source: B. Deep Reinforcement Learning · p. 4

V. NUMERICAL EXPERIMENTS

The numerical experiments quantify the performance of the proposed DQN-based online strategic computation offloading method.

V. NUMERICAL EXPERIMENTS

  • The numerical experiments quantify the performance of the proposed DQN-based online strategic computation offloading method.Source: §V · p. 4

A. General Setup

The experiments use a six-BS MEC system with specified channel, energy, and computing parameters, and compare the proposed approach with local, cloud, and greedy policies.

A. General Setup

  • The MEC setup uses N = 6 BSs and channel gains from {−18, −16, −14, −12, −10, −8, −6, −4} dB, with transitions following randomly generated matrices.Source: A. General Setup · pp. 4–5
  • The DQN replay memory capacity is U = 5000 and the mini-batch size is S = 100; the experiments set γ = 0.9, W = 10^6 Hz, and δ = 10^-3 second.Each energy unit corresponds to 5 × 10−5 J, and harvested energy follows a Poisson arrival process with average rate λ(e).Source: A. General Setup · pp. 4–5
  • The comparison policies are Local, which executes tasks on-device; Cloud, which offloads via the BS with the best channel quality; and Greedy, which minimizes current delay when both queues are nonempty.Source: A. General Setup · p. 5

B. Experimental Results

Numerical experiments under various settings are used to validate the proposed work.

B. Experimental Results

1) Experiment 1 – Convergence performance:

The first experiment assesses convergence of the proposed DQN-based online offloading algorithm by tracking its loss function. The reported convergence supports using 9 × 10^5 epochs for each subsequent experiment configuration.

1) Experiment 1 – Convergence performance:

2) Experiment 2 – Performance under different DQN structures:

The experiments compare DQN structures and evaluate the proposed algorithm as energy arrivals change. The proposed algorithm improves average cost against three baselines, while wider, shallower DQNs perform better in the tested structure comparison.

2) Experiment 2 – Performance under different DQN structures:

  • Average cost is worse with deeper DQNs but improves with more neurons in a one-hidden-layer DQN.The authors link deeper networks to higher training errors over the limited time horizon and conclude that a wider, not deeper, DQN better approximates the Q-function in this MEC scenario.Source: Figure 4 · p. 5, 2) Experiment 2 – Performance under… · p. 5

3) Experiment 3 – Performance with changing λ(t) and λ(e):

VI. CONCLUSIONS

The paper applies a DQN-based online offloading algorithm to an ultra-dense network with dynamic channels, harvested energy, and task arrivals. Experiments report up to 56% improvement over three baselines, with a tradeoff across execution delay, handover delay, and task-dropping cost.

VI. CONCLUSIONS

  • The setting accounts for time-varying channel qualities, harvested energy units, and task arrivals.Source: §VI · p. 6
  • The proposed DQN-based online algorithm addresses the high-dimensional state space without requiring a priori dynamics statistics.Source: §VI · p. 6
  • The algorithm achieves up to 56% performance improvement over three baselines, reflecting a tradeoff among execution delay, handover delay, and task-dropping cost.Source: §VI · p. 6
Loading 1804.00514v1…