PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks

Yueyue Dai, Ke Zhang, Sabita Maharjan, Yan Zhang

arXiv:2011.08430v2cs.LGcs.AI

TL;DR

The paper addresses computation offloading in IIoT networks with resource constraints, stochastic task arrivals, and heterogeneous resources. It proposes a Digital Twin Network, transforms the stochastic optimization using Lyapunov optimization, and applies AAC for policy learning. Numerical results report that the proposed algorithm significantly outperforms benchmark policies.

  • Problem

    IIoT offloading must jointly handle resource-constrained devices, stochastic task arrivals, heterogeneous resources, and coupled allocation decisions.

  • Method

    The paper builds a Digital Twin Network, transforms the stochastic problem into a deterministic per-time-slot problem with Lyapunov optimization, and uses AAC for offloading and resource allocation.

  • Results

    The proposed AAC-based algorithm significantly outperforms the benchmark policies in numerical results.

  • Takeaways & Limitations

    Digital Twin Networks provide the paper’s framework for modeling IIoT topology and stochastic task arrivals while supporting joint computation offloading and resource allocation.

Abstract

from arXiv · show

The rapid development of Industrial Internet of Things (IIoT) requires industrial production towards digitalization to improve network efficiency. Digital Twin is a promising technology to empower the digital transformation of IIoT by creating virtual models of physical objects. However, the provision of network efficiency in IIoT is very challenging due to resource-constrained devices, stochastic tasks, and resources heterogeneity. Distributed resources in IIoT networks can be efficiently exploited through computation offloading to reduce energy consumption while enhancing data processing efficiency. In this paper, we first propose a new paradigm Digital Twin Networks (DTN) to build network topology and the stochastic task arrival model in IIoT systems. Then, we formulate the stochastic computation offloading and resource allocation problem to minimize the long-term energy efficiency. As the formulated problem is a stochastic programming problem, we leverage Lyapunov optimization technique to transform the original problem into a deterministic per-time slot problem. Finally, we present Asynchronous Actor-Critic (AAC) algorithm to find the optimal stochastic computation offloading policy. Illustrative results demonstrate that our proposed scheme is able to significantly outperforms the benchmarks.

I. INTRODUCTION

The paper introduces Digital Twin Networks for IIoT computation offloading under stochastic task arrivals and heterogeneous, resource-constrained devices. It combines Lyapunov optimization with AAC-based offloading and resource allocation to address these challenges.

  • I. INTRODUCTION: IIoT’s expanding device and data volume increases the need for industrial network digitalization and efficient processing.The cited projection reaches 41.6 billion connected devices and nearly 80 zettabytes of data by 2025.
  • I. INTRODUCTION: Digital twins mirror IIoT physical entities through virtual models that use sensing data to understand, predict, estimate, and analyze system changes.The paper applies this capability to network topology and system management.
  • I. INTRODUCTION: Computation offloading sends data and tasks from resource-constrained IIoT devices to distributed servers to improve processing efficiency and prolong device battery lifetime.Prior work commonly assumes one deterministic task, whereas IIoT devices continuously generate stochastic arrivals and use heterogeneous resources.
  • I. INTRODUCTION: Existing DRL offloading solutions mainly use deterministic task arrivals, limiting their direct applicability to IIoT networks with stochastic task generation.The paper identifies stochastic arrivals as the key distinction from these earlier approaches.
  • I. INTRODUCTION: The proposed DTN maps IIoT and digital systems by monitoring real-time device and base-station states and modeling network topology, devices, and base stations.Devices collect data and synchronize parameters, SBSs build local virtual models, and the MBS constructs topology and designs policies.
  • I. INTRODUCTION: Lyapunov optimization transforms the stochastic offloading problem into a deterministic per-time-slot problem, while AAC solves joint offloading and resource allocation.The contributions also report that the proposed algorithm significantly outperforms benchmark policies.

B. Network Model

The DTN models IIoT as a time-slotted digital representation with devices, base stations, wireless associations, communication conditions, and stochastic task queues.

  • The virtual DTN representation contains wireless topology, communication models, and stochastic task queueing models.
  • The wireless edge network is represented by a directed graph connecting devices to small-cell or macro base stations.
  • Base-station digital twins use location, bandwidth, and computation-resource attributes to characterize edge servers.
  • Device association depends on proximity and coverage: devices communicate with the nearest base station and otherwise use the MBS.
  • At each slot, devices generate stochastic tasks, execute some locally, offload some to associated base stations, and queue the remainder in local buffers.The arrivals λ_i(t) are independent across slots with mean E[λ_i(t)] = λ, and task queues are dynamically updated.

C. Task Offloading Model

The task offloading model divides computation between local devices and edge servers, while accounting for local execution, wireless offloading, computation, and transmission energy.

  • Local Execution: Devices execute D^l_i(t) locally using their available CPU resources, with task capacity determined by CPU frequency, slot duration, and per-bit cycle requirements.
  • Local Execution: Local computation energy depends on the device’s allocated computation resource and the effective switched capacitance of its chip architecture.
  • Edge Server Execution: Devices offload D^e_ij(t) to associated base stations through wireless communication.
  • Edge Server Execution: Edge-execution energy includes uplink transmission and computation components, while downlink result-transmission energy is ignored because the result size is very small.
  • Total energy combines local execution, edge-server execution, and transmission energy for computation offloading.

III. PROBLEM FORMULATION

This section formulates stochastic computation offloading in DTNs as an optimization problem and prepares it for Lyapunov-based transformation.

  • The paper formulates the DTN stochastic computation offloading problem as an optimization problem before applying Lyapunov optimization.

A. Stochastic Computation Offloading Problem

The problem minimizes long-term energy efficiency under dynamic resource, power, bandwidth, computation, and queue-stability constraints. Its stochastic nature motivates Lyapunov optimization, which converts long-term objectives and constraints into short-term subproblems and queue-stability conditions.

  • The objective η_EE is the ratio of long-term total energy consumption to long-term aggregate accomplished computation tasks.
  • The per-slot operation jointly controls bandwidth allocation, transmission power, edge-task departures, and local and edge computation resources.
  • The formulation constrains bandwidth, transmission power, computation resources, edge processing capacity, and stability of all task queues.
  • Stochastic task arrivals, dynamic channel and buffer states, and coupled rate-resource effects make classic convex optimization methods difficult to apply.
  • Lyapunov optimization transforms long-term objectives into short-term subproblems and long-term constraints into queue-stability constraints with limited prior information requirements.

B. Lyapunov-based Problem Transformation and Digital twin-predicted Perturbation

The framework uses Lyapunov optimization to control task queues and network energy efficiency, with digital twin-predicted perturbations enabling per-slot optimization of the stochastic offloading problem. DRL is then used to obtain near-optimal real-time solutions as network conditions change.

  • Lyapunov-based Problem Transformation and Digital twin-predicted Perturbation: The quadratic Lyapunov function uses queue backlogs and a perturbation vector β to keep the Lyapunov value small.The perturbation parameters are simulated in the digital twin.
  • Lyapunov-based Problem Transformation and Digital twin-predicted Perturbation: Minimizing Lyapunov drift shortens task queues, while drift-plus-penalty optimization targets network stability and energy efficiency.The non-negative weight V balances these objectives in the drift-plus-penalty formulation.
  • Lyapunov-based Problem Transformation and Digital twin-predicted Perturbation: The stochastic offloading algorithm observes current queues and task arrivals, then determines the action a(t) by solving a problem in each time slot.The algorithm updates queue lengths and advances to the next time slot after action execution.
  • Lyapunov-based Problem Transformation and Digital twin-predicted Perturbation: Digital twin predicts the perturbation vector β from local task queues and uses it to optimize the per-time-slot drift bound.The predicted perturbation is incorporated before solving the per-slot problem.
  • Lyapunov-based Problem Transformation and Digital twin-predicted Perturbation: DRL replaces repeated decomposition and alternating optimization because changing wireless channels or task queues require frequent subproblem recalculation.The paper presents DRL as a real-time approach for finding a near-optimal solution to P2.

IV. DRL-EMPOWERED STOCHASTIC COMPUTATION OFFLOADING ALGORITHM FOR DTN

The digital twin mirrors the physical wireless network, predicts relevant environment information, and supplies system states to a DRL agent. The agent selects continuous offloading and resource-allocation actions, updates the state through immediate rewards, and seeks to maximize cumulative reward.

  • Digital Twin-simulated Network Environment: Digital twin mirrors network topology and physical-network parameters, then transmits network state to DRL for energy-efficiency optimization.The state is constructed by the digital twin and outputted to the DRL agent.
  • Digital Twin-simulated Network Environment: Digital twin predicts device and base-station location, energy, and generated task flow to gather network environment information.KNN and position-prediction methods can be used for users’ locations, while predicted energy updates maximal transmission power.
  • State and Action Design: The system state includes wireless data rates, available computation resources, transmission power, and local and edge-server queue lengths.These components define the state space S at each time slot.
  • State and Action Design: The action comprises bandwidth allocation, transmission power, edge-task departures, and local and edge computation-resource allocation.Bandwidth, power, and task-departure variables are explicitly represented as action components.
  • State and Action Design: All action variables are continuous, motivating a policy gradient-based DRL algorithm rather than a value-based method designed for discrete actions.The policy gradient approach is used to explore the policy in the continuous action space.
  • DRL Interaction: After executing an action, digital twin updates the system state and estimates immediate reward, while the agent maximizes cumulative discounted reward.Constraint violations produce a negative-constant penalty.

B. Asynchronous Actor-Critic Algorithm

The paper selects policy gradient-based DRL for continuous computation-offloading actions and deploys an Asynchronous Actor-Critic algorithm in the digital twin. AAC combines multiple asynchronous environment interactions with actor-generated actions and critic-based evaluation.

  • Asynchronous Actor-Critic Algorithm: Value-based methods such as DQN and double DQN are limited for continuous action spaces, whereas policy gradient methods learn stochastic policies for them.Policy gradient updates parameterized policies using gradients of expected reward.
  • Asynchronous Actor-Critic Algorithm: The proposed method deploys AAC in the digital twin to optimize cumulative reward.AAC is selected as the policy-gradient DRL algorithm for the offloading problem.
  • Asynchronous Actor-Critic Algorithm: AAC uses multiple agents that independently interact with replicas of the environment during asynchronous learning.Each agent contains its own environment replica.
  • Asynchronous Actor-Critic Algorithm: The actor generates actions, while the critic evaluates the current policy using rewards obtained from the environment.This actor-critic division supports policy learning through interaction.

1) Actor-Critic based policy gradient training:

The actor-critic training procedure learns an offloading and resource-allocation policy from the current state. The actor updates its policy using the critic’s advantage estimate, while the critic estimates state values and updates its own parameters.

  • Actor-Critic based policy gradient training:: The actor network maps the current state to an offloading and resource-allocation policy through a deep neural network.The policy parameters θπ are trained using the policy gradient method.
  • Actor-Critic based policy gradient training:: The advantage function measures the difference between expected cumulative discounted rewards after choosing an action and following policy π.It indicates whether the action performs better or worse than expected.
  • Actor-Critic based policy gradient training:: The critic estimates each state’s cumulative discounted reward under the current actor policy and updates its parameters using learning rate αv.The actor uses the critic’s output to update policy parameters with learning rate απ.

2) Asynchronous Learning with Experience Replay:

The paper replaces experience replay with asynchronous learning in AAC, using distributed learning agents and a global agent to update shared policy and value parameters.

  • Learning Agent: Each learning agent synchronizes parameters, constructs the current state through the digital twin, performs actions, computes rewards, and sends accumulated rewards globally.The agent repeats this process until the training limit is reached.
  • Global Agent: The global agent receives rewards, accumulates gradients, and asynchronously updates shared policy and value parameters.Training terminates when the shared counter exceeds the maximum training episodes.
  • Asynchronous Learning: AAC reduces correlations between adjacent samples through asynchronous online learning without requiring replay memory.The approach also uses considerably less computation than replay-memory-based learning.
  • Agent Architecture: DTN deploys a global agent and multiple learning agents, with learning agents interacting with their own environments.Learning agents are deployed at SBSs and communicate rewards to the global agent.

V. NUMERICAL RESULTS

Numerical results evaluate system cost under different schemes, network sizes, device loads, DRL algorithms, and learning rates. The proposed joint optimization with AAC outperforms the reported benchmarks, while system cost generally increases with device count.

  • Experimental Setup: The evaluation uses one MBS, three SBSs, and twenty devices under Rayleigh fading channels.Device transmission power is capped at 100 mW, with 10 MHz MBS bandwidth and 5 MHz per-SBS bandwidth.
  • Different Schemes: The proposed scheme outperforms benchmarks by jointly optimizing computation offloading, bandwidth, transmission power, and computation resource allocation.The comparison is based on system cost over training episodes and under varying numbers of devices.
  • Number of SBSs: When N = 40, system cost decreases as the number of SBSs increases; when N = 20, it changes little.Increasing SBSs therefore has greater effect when the device population is large.
  • Number of Devices: System cost increases with the number of devices across the compared schemes.The paper attributes this to more offloading requests consuming additional communication and computation resources.
  • DRL Algorithms: AAC performs considerably better than DQN as the number of devices increases.The paper attributes DQN's weaker performance to action discretization potentially skipping better actions.
  • Learning Rate: When the learning rate is 0.001, the proposed algorithm converges to its lowest system cost.The paper identifies 0.001 as the best learning rate for the proposed algorithm.
Loading 2011.08430v2…