PaperlayerView exact revision on arXiv

Source-linked AI summary

Edge Intelligence for Energy-efficient Computation Offloading and Resource Allocation in 5G Beyond

Yueyue Dai, Ke Zhang, Sabita Maharjan, Yan Zhang

arXiv:2011.08442v2cs.LGeess.SP

TL;DR

The paper tackles energy-efficient offloading and resource allocation in dynamic, heterogeneous end-edge-cloud networks with incomplete network information. It models the joint problem as an MDP and proposes a DRL-based scheme, whose real-world-dataset evaluation significantly outperforms benchmark policies in system energy consumption.

  • Problem

    Dynamic wireless conditions, diverse application requirements, and heterogeneous end-edge-cloud resources make joint computation offloading and resource allocation difficult.

  • Method

    The paper formulates joint offloading and resource allocation as an MDP and uses a DDPG-based DRL algorithm with action refinement to handle discrete offloading and continuous resource allocation.

  • Results

    Real-world-dataset numerical results show that the proposed DRL-based algorithm significantly outperforms benchmark policies in system energy consumption.

  • Takeaways & Limitations

    The proposed approach jointly schedules offloading and heterogeneous computation resources in a multi-user end-edge-cloud network to reduce system energy consumption.

Abstract

from arXiv · show

5G beyond is an end-edge-cloud orchestrated network that can exploit heterogeneous capabilities of the end devices, edge servers, and the cloud and thus has the potential to enable computation-intensive and delay-sensitive applications via computation offloading. However, in multi user wireless networks, diverse application requirements and the possibility of various radio access modes for communication among devices make it challenging to design an optimal computation offloading scheme. In addition, having access to complete network information that includes variables such as wireless channel state, and available bandwidth and computation resources, is a major issue. Deep Reinforcement Learning (DRL) is an emerging technique to address such an issue with limited and less accurate network information. In this paper, we utilize DRL to design an optimal computation offloading and resource allocation strategy for minimizing system energy consumption. We first present a multi-user end-edge-cloud orchestrated network where all devices and base stations have computation capabilities. Then, we formulate the joint computation offloading and resource allocation problem as a Markov Decision Process (MDP) and propose a new DRL algorithm to minimize system energy consumption. Numerical results based on a real-world dataset demonstrate that the proposed DRL-based algorithm significantly outperforms the benchmark policies in terms of system energy consumption. Extensive simulations show that learning rate, discount factor, and number of devices have considerable influence on the performance of the proposed algorithm.

I. INTRODUCTION

The paper addresses energy-efficient computation offloading and resource allocation in dynamic, heterogeneous 5G-beyond networks where complete network information is difficult to obtain. It proposes edge-intelligence and DRL methods for jointly optimizing multi-tier offloading and distributed resources.

  • Motivation: 5G-beyond end-edge-cloud networks support local computing and offloading to edge servers or the cloud, but heterogeneous capabilities complicate optimization.The architecture is intended for computation-intensive and delay-sensitive applications.
  • Motivation: Dynamic wireless conditions, incomplete network information, diverse application requirements, and multiple radio access modes make optimal offloading and resource allocation difficult.Existing optimization methods often require complete and accurate channel, bandwidth, and computation-resource information.
  • Motivation: Edge intelligence can use locally learned information and resource correlations at distributed edge entities without requiring the entire network state from the central cloud.This supports more localized resource scheduling in heterogeneous edge environments.
  • Contributions: The paper formulates joint computation offloading and resource allocation to minimize system energy consumption under stringent delay constraints.The formulation targets a multi-user end-edge-cloud network with heterogeneous computation resources.
  • Contributions: The proposed DRL-based algorithm jointly optimizes offloading decisions and resource allocation, with real-world-dataset results significantly outperforming benchmark policies in energy consumption.The work includes a multi-user architecture in which tasks can be offloaded to edge servers and the macro-cell base station.
  • Contributions: Prior DRL offloading studies generally address local-versus-edge decisions, whereas this work targets three-tier offloading with heterogeneous resource allocation using action refinement.The action space combines discrete offloading decisions with continuous resource-allocation decisions.

III. SYSTEM MODEL

The system model describes a multi-user end-edge-cloud 5G heterogeneous network in which devices can execute tasks locally or offload them to nearby SBSs or the MBS. Communication depends on coverage, bandwidth, transmission power, interference, and channel conditions.

  • Network Architecture: The network contains one MBS, M SBSs with edge servers, and N devices distributed under MBS coverage.The MBS provides cloud resources, while SBSs provide computation resources close to end devices.
  • Task Model: Each device has a computation-intensive, delay-sensitive task represented by input size, required computation resource, and execution deadline.The task is denoted as D_i = (d_i, c_i, τ_i).
  • Execution Options: A device may execute its task locally, offload it to a nearby SBS, or offload it to the MBS, with one execution choice permitted per task.Offloading decisions are represented by binary variables for local, SBS, and MBS execution.
  • Wireless Communication: OFDMA is used, while SBSs reuse the MBS frequency resources and are deployed at hotspots with limited coverage overlap.These assumptions shape the interference and data-rate model.
  • Wireless Communication: Devices communicate with the nearest SBS when within its coverage radius; otherwise, they communicate with the MBS.Wireless rates depend on bandwidth, transmission power, channel gain, path loss, noise, and interference from other SBSs.

B. Computation Model

The computation model compares local execution with offloading to an SBS or the MBS, modeling completion delay and energy consumption under allocated computation resources. Offloading delay includes transmission and computation components, subject to resource limits.

  • Execution Modes: Each task can be executed locally, at an SBS, or at the MBS.These three execution modes form the basis for the delay and energy models.
  • Local Computing: Local execution uses the device’s full computation resource, and its completion delay contains only task computing time.The device computation resource is measured in CPU cycles per second.
  • Local Computing: Local energy consumption depends on the task’s required computation and the effective switched capacitance, with unit computation energy proportional to ς(f_i^l)^2.The effective switched capacitance depends on chip architecture.
  • SBS Offloading: When multiple devices offload to an SBS, each receives part of the SBS computation resource, whose total allocation cannot exceed the SBS capacity.The allocated SBS resource is denoted f_ij^s, and the total SBS resource is F_j^s.
  • SBS Offloading: SBS offloading delay comprises uplink transmission and task computation, while downlink result transmission is ignored because the result is often much smaller than the input.The corresponding completion delay is modeled using d_i/R_ij^s + c_i/f_ij^s.
  • MBS Offloading: MBS offloading follows a process similar to SBS offloading and is constrained by the MBS computation capacity.Its energy model includes the MBS computation resource and energy per computation resource.

IV. PROBLEM FORMULATION

The paper formulates joint offloading and resource allocation as an MDP whose objective is minimizing system energy consumption while satisfying delay, assignment, and computation-capacity constraints. The action controls execution choices and resource allocations, and the reward represents long-term performance with penalties for missed deadlines.

  • Optimization Problem: The optimization objective is to minimize system energy consumption subject to computation-resource constraints.The formulation jointly determines computation offloading and resource allocation.
  • Optimization Problem: Constraint (8a) limits task-processing delay, constraints (8b) and (8e) enforce one execution mode, and constraint (8c) limits allocated base-station resources.Because of the integer constraint, the optimization problem is NP-hard.
  • MDP Formulation: The MDP state includes task data sizes, remaining computation requirements, deadlines, available base-station resources, locations, and wireless data rates.These variables describe the network condition observed at each time slot.
  • MDP Formulation: The action combines local, SBS, and MBS offloading decisions with computation-resource allocations at SBSs and the MBS.The action therefore represents both execution selection and resource assignment.
  • MDP Formulation: After each action, task data, computation requirements, deadlines, and available resources are updated according to local or offloaded execution.The state transition differs depending on whether a task is processed locally, at an SBS, or at the MBS.
  • Reward Design: The long-term objective maximizes cumulative reward, with a penalty when tasks cannot all be completed within their stringent delay constraints.The discount factor ε weights future rewards.

V. DRL-BASED COMPUTATION OFFLOADING AND RESOURCE ALLOCATION SCHEME

The proposed scheme uses DDPG to solve the continuous-action MDP and adds action refinement to jointly determine computation offloading and resource allocation.

  • DDPG-Based Scheme: DDPG is used because the action space contains continuous variables, and action refinement is incorporated for joint offloading and resource allocation.DDPG is an actor-critic-based deep reinforcement learning algorithm.

A. DDPG-based Computation Offloading and Resource Allocation Algorithm

The paper uses DDPG to learn computation offloading and resource allocation policies from network states. Its training loop combines actor–critic networks, target networks, and replay memory over repeated time slots.

  • A. DDPG-based Computation Offloading and Resource Allocation Algorithm: DDPG maps the current state to detailed computation offloading and resource allocation actions through primary actor and critic DNNs.The actor generates actions, while the critic evaluates them; target networks provide training targets.
  • A. DDPG-based Computation Offloading and Resource Allocation Algorithm: The DDPG architecture contains primary and target networks plus replay memory.Replay memory stores state, action, reward, and next-state experience tuples.
  • A. DDPG-based Computation Offloading and Resource Allocation Algorithm: Algorithm 1 initializes actor and critic networks, executes actions in each time slot, observes rewards and next states, and stores the resulting experience.The procedure repeats this interaction across episodes and time slots.
  • A. DDPG-based Computation Offloading and Resource Allocation Algorithm: Each training iteration samples a mini-batch, computes target values, updates the critic by minimizing loss, updates the actor with a sampled policy gradient, and updates the target network.These updates are performed after experiences are stored in replay memory.

1) Primary Actor DNN Training:

The primary actor generates exploratory offloading and resource allocation actions and updates its parameters using sampled policy gradients.

  • 1) Primary Actor DNN Training:: The explored policy maps the current state to a proto-actor action through the primary actor DNN.The policy parameter is θπ, and the generated action represents the explored edge policy.
  • 1) Primary Actor DNN Training:: Ornstein-Uhlenbeck noise provides random exploration for DDPG.The noise is added to the actor-generated action.
  • 1) Primary Actor DNN Training:: The primary actor updates θπ using a sampled policy gradient based on mini-batch experience tuples from replay memory.The actor learning rate is denoted by απ.

2) Primary Critic DNN Training:

The primary critic evaluates actions with an action-value function and trains its parameters using Bellman targets computed from sampled experience.

  • 2) Primary Critic DNN Training:: The primary critic evaluates selected actions using an action-value function.The action-value function is calculated using the Bellman optimality equation.
  • 2) Primary Critic DNN Training:: The critic takes current and next states as inputs to calculate Q(st, at|θQ) for each action.This evaluation supports the critic’s action-value estimation.
  • 2) Primary Critic DNN Training:: The primary critic updates θQ by minimizing a loss function based on target values.The target value yt is obtained through the target network.
  • 2) Primary Critic DNN Training:: Critic updates use mini-batches of experience tuples randomly sampled from replay memory.The critic learning rate is denoted by αQ.

3) Target Network Training:

The target network is maintained as a separately parameterized version of the primary network, while action refinement converts continuous DDPG outputs into integer offloading decisions through weighted matching.

  • 3) Target Network Training:: The target network has the same structure as the primary network but different parameters and supplies target values for critic training.Its parameters are updated separately from the primary network.
  • 3) Target Network Training:: The algorithm updates primary networks from replayed experience and then updates the target network parameters.The procedure initializes both primary and target actor–critic networks before training.
  • 3) Target Network Training:: The algorithm’s complexity is mainly determined by four neural networks and one activation layer.Actor and critic layer widths determine the stated time-complexity calculation.
  • 3) Target Network Training:: Because DDPG outputs continuous values while offloading decisions require integers, the method refines actions through rounding, graph construction, and integer matching.The weighted bipartite graph relates devices to offloading strategies before matching produces binary decisions.
  • 3) Target Network Training:: The bipartite graph uses virtual nodes whose count represents how many devices choose each offloading strategy.Edges connect devices and virtual nodes with weights derived from continuous action values.
  • 3) Target Network Training:: The Hungarian algorithm finds a maximum-weight bipartite matching that yields binary offloading decisions.The action-refinement algorithm has polynomial complexity O(|V||E|).

VI. NUMERICAL RESULTS

The evaluation uses Python and TensorFlow to assess the proposed DDPG-inspired computation offloading and resource allocation algorithm on a real-world dataset.

  • The proposed algorithm is evaluated using Python and TensorFlow.The evaluation targets the DDPG-inspired computation offloading and resource allocation algorithm.
  • The evaluated method is DDPG-inspired.
  • The evaluation is based on a real-world dataset.

A. Simulation Setup

The study evaluates a DDPG-based strategy against local-computing and full-offloading benchmarks across device counts and task types. Results attribute its energy advantage to jointly selecting offloading destinations and computation resources, while performance varies with task characteristics.

  • Simulation Setup: The network includes one MBS, 10 SBSs, and 100 devices, with device transmission power capped at 100 mW.
  • Simulation Setup: The simulation uses 10 MHz MBS bandwidth, 5 MHz SBS bandwidth, and a 1 s latency limit.
  • Simulation Setup: The proposed DDPG-inspired algorithm is evaluated over up to 6000 episodes and 20 steps per episode, using tanh(x)+1 as its activation function.
  • Simulation Setup: The benchmarks are local computing, where every task executes locally, and full offloading, where all tasks are sent to the MBS.
  • Performance Analysis: Across device counts, the proposed algorithm outperforms both benchmarks by jointly optimizing offloading decisions and computation resources.
  • Performance Analysis: For computation-intensive tasks, the proposed algorithm has the lowest energy consumption, whereas local computing is best for large-data, low-computation tasks.

C. Impact of Factors on Performance

The proposed DDPG-based algorithm is evaluated across learning rates, discount factors, device counts, and changing bandwidth, with each factor affecting performance or convergence. The study reports best settings, increased training delay with larger state and action spaces, and adaptation to a changed network environment.

  • Learning rate: The proposed DDPG-based algorithm achieves its best performance when both actor and critic learning rates are 10^-3.Only learning-rate pairs with απ and αQ equal to 10^-3 or 10^-4 converge in the reported comparison.
  • Learning rate: The critic learning rate αQ considerably influences the convergence trend, while mixed learning-rate settings eventually reach the same value.The reported convergence trends differ with αQ, whereas απ = 10^-3, αQ = 10^-4 and απ = 10^-4, αQ = 10^-3 converge to the same value.
  • Discount factor: The best discount factor is ε = 0.6, whose normalized cumulative reward exceeds the results for ε = 0.65 and ε = 0.7.The algorithm converges for all tested discount factors from 0.5 to 0.7.
  • Number of devices: Normalized cumulative reward increases as the number of devices rises from 20 to 60 and 100, while training delay increases dramatically.More devices enlarge the state and action spaces and require more communication and computation resources.
  • Time-varying bandwidth: After bandwidth is increased tenfold, normalized cumulative reward sharply increases and stabilizes at a higher value as the algorithm converges to a new solution.The time-varying-bandwidth evaluation uses a small network with 10 devices and 5 SBSs.
Loading 2011.08442v2…