Source-linked AI summary
Delay-aware and Energy-Efficient Computation Offloading in Mobile Edge Computing Using Deep Reinforcement Learning
Laha Ale, Ning Zhang, Xiaojie Fang, Xianfu Chen, Shaohua Wu, Longzhuang Li
TL;DR
IoT devices generate computation-intensive tasks while remote cloud processing can increase congestion and latency, motivating computation offloading in dynamic MEC networks. The paper proposes end-to-end DRL for jointly selecting offloading servers and allocating computational resources, and reports that it outperforms existing DRL models and a greedy algorithm in simulations.
Problem
IoT applications require substantial storage and computing capacity, but resource-constrained devices relying on remote cloud servers may experience traffic congestion and prolonged service latency.
Method
An end-to-end DRL approach jointly selects MEC offloading servers and allocates computational resources to maximize long-term utility based on task and network information.
Results
The proposed DRL method outperforms existing DRL models and the greedy algorithm in rewards, completed tasks, and energy consumption simulations.
Takeaways & Limitations
End-to-end DRL can make joint offloading and resource-allocation decisions without relying on additional optimization functions while targeting timely task completion and energy efficiency.
Abstract
from arXiv · showhide
Internet of Things (IoT) is considered as the enabling platform for a variety of promising applications, such as smart transportation and smart city, where massive devices are interconnected for data collection and processing. These IoT applications pose a high demand on storage and computing capacity, while the IoT devices are usually resource-constrained. As a potential solution, mobile edge computing (MEC) deploys cloud resources in the proximity of IoT devices so that their requests can be better served locally. In this work, we investigate computation offloading in a dynamic MEC system with multiple edge servers, where computational tasks with various requirements are dynamically generated by IoT devices and offloaded to MEC servers in a time-varying operating environment (e.g., channel condition changes over time). The objective of this work is to maximize the completed tasks before their respective deadlines and minimize energy consumption. To this end, we propose an end-to-end Deep Reinforcement Learning (DRL) approach to select the best edge server for offloading and allocate the optimal computational resource such that the expected long-term utility is maximized. The simulation results are provided to demonstrate that the proposed approach outperforms the existing methods.
I. INTRODUCTION
The paper addresses computation offloading in dynamic MEC systems by jointly selecting edge servers and allocating computational resources to complete more tasks before deadlines while reducing energy consumption. It uses end-to-end DRL to make long-term decisions in changing, complex environments without relying on additional optimization functions.
- Motivation: IoT devices are resource-constrained, while remote-cloud processing can cause traffic congestion and prolonged service latency.MEC places computation and storage resources closer to IoT devices to mitigate congestion and reduce latency.
- Research challenge: Classical MEC optimization methods may require reformulation under changing network conditions and can converge only to local optima.The MEC system is difficult to describe mathematically, and optimization methods may require many iterations.
- Research challenge: Standard RL supports model-free long-term control but is unsuitable when the state or action space is huge.DRL addresses this issue by using neural networks to approximate Q values.
- Proposed approach: The proposed end-to-end DRL model jointly maximizes timely task completion and minimizes energy consumption without standard optimization methods.It selects offloading servers and computational frequencies to maximize expected long-term compensation.
- Proposed approach: The model represents time-varying channel conditions, task profiles, and server states while providing a model-free solution for complex MEC environments.The state includes continuous variables such as channel gain, and clip-reward techniques are used to reduce training oscillation.
- Evaluation: Simulation results show that the proposed approach processes more tasks before deadlines while consuming less energy than existing methods.The paper evaluates the approach through extensive simulations.
B. Problem Formulation
The paper formulates dynamic MEC offloading as an MDP that jointly assigns tasks to servers and computational frequencies while maximizing deadline-completed tasks and minimizing long-term energy consumption.
- Objective: The objective is to maximize deadline-completed tasks and minimize energy consumption over the long term.The system is modeled as an MDP with expected long-term rewards.
- Action-space challenge: 27?
- State-transition design: Sequential task processing within each time slot is used to avoid the exponential action-space growth associated with jointly distributing multiple tasks.The formulation assumes channel and other MEC parameters remain unchanged during a time slot and state transitions depend on agent actions.
- Actions: The action space assigns each task to an MEC server and recommends a CPU-frequency percentage from 0% to 100%.Its size is K × fp, where K is the number of MEC servers and fp is the frequency-resolution parameter.
- Reward: The reward combines normalized on-time task completion with energy cost using a tunable weight η.η ∈[0, 1] balances completed tasks and energy consumption, while C rewards longer non-overloaded episodes.
- Policy optimization: The long-term return combines immediate rewards with discounted future rewards under a learned policy.The discount factor satisfies 0 < γ < 1.
III. PROPOSED METHOD
The proposed method uses DRL to dynamically choose MEC offloading servers and computational-frequency allocations, optimizing completed tasks and energy consumption through learned long-term policies.
- Proposed method: The DRL method jointly determines MEC servers and computational frequency allocations to maximize completed tasks and minimize system energy consumption.The model takes observed MEC-network data as input and produces control parameters.
A. Data Prepossessing
The preprocessing pipeline addresses noisy, time-varying, differently scaled, and high-dimensional MEC features before they enter the learning model.
- Motivation: Raw channels, server states, and user tasks are time-varying, differently scaled, and high-dimensional, making preprocessing important for DRL training.Without preprocessing, the agent may converge too slowly or to non-optimal solutions.
- Feature organization: Features are normalized, concatenated, and organized in a tree-like structure containing server states, transfer speeds, and queued tasks.The structure uses a root, MEC-server branch, sub-branches, and leaf-level nodes.
- Feature construction: The normalized sub-features are concatenated into a single feature for the learning model.
B. DRL Model
The DRL model treats the MEC network as an environment and learns policies through interaction, balancing exploitation of learned Q-values with exploration of the unknown system.
- Agent and environment: The MEC network is modeled as the RL environment, while the DRL model acts as an agent learning from interaction and experience.The agent is expected to generate policies maximizing long-term accumulated rewards.
- Learning principle: RL learns from sequential evaluative feedback rather than labeled training data.
- Exploration and exploitation: The agent balances exploitation of learned Q-values with exploration of the unknown MEC environment.
1) Reinforcement learning Framework:
The reinforcement learning framework seeks policies that maximize long-term accumulated rewards by selecting actions using action-value estimates. Classical Q-table methods become impractical for very large or continuous state-action spaces, motivating function approximation.
- Reinforcement learning Framework: The ϵ-greedy strategy balances exploitation and exploration by choosing greedily with probability 1−ϵ and randomly with probability ϵ.The exploration proportion decreases over episodes as the agent acquires knowledge of the MEC environment.
- Reinforcement learning Framework: The agent seeks an optimal policy π∗ by maximizing long-term accumulated rewards through optimal action-values Q∗(s, a).Q(s, a) represents the value of taking action a in state s and then following policy π.
- Reinforcement learning Framework: Bellman updates repeatedly choose the next action a′ with the greatest value to improve Q(s, a) toward Q∗ as τ approaches infinity.The optimal action-value is obtained by considering the maximizing next action over successive state updates.
- Reinforcement learning Framework: Classical Q-table reinforcement learning is unsuitable when the state or action space is huge, while approximate functions can reduce the search-space burden.The paper notes that exhaustive iteration and exploration are impractical in vast or continuous spaces.
2) Deep Reinforcement Learning Model:
The deep reinforcement learning model uses neural-network function approximation to handle complex state spaces and coordinates offloading decisions with computational-frequency recommendations. A fixed target network and replay-based learning support training stability.
- Deep Reinforcement Learning Model: A neural network, or Q-Network, approximates action-values with parameters w for the proposed DRL model.This replaces storing all state-action value pairs in a Q-table.
- Deep Reinforcement Learning Model: The coordinator collects task profiles, supplies system states to the DRL agent, and executes its offloading and server-frequency actions in the MEC environment.The agent stores state, action, reward, and next-state data for subsequent training.
- Deep Reinforcement Learning Model: The primary network learns from replay-buffer data, while a fixed-parameter target network is periodically synchronized to reduce training oscillation and divergence.The target-network weights are copied from the primary network after every N episodes.
- Deep Reinforcement Learning Model: The learning-network parameters are updated by minimizing mean squared error against a target formed from the reward and discounted next-state estimate.The passage describes the fixation term as r + γ max ... for constructing the target.
C. Training Process
The DQN training process initializes memory, networks, exploration, and scores; generates experience through environment interaction; and learns from sampled replay experiences over repeated episodes.
- Training Process: Training begins by initializing replay memory D, learning and target networks, scores, and the exploration parameter ϵ.The learning network starts with random weights, and the target network is initialized from those weights.
- Training Process: Each episode preprocesses the initial raw input x1 into the initial state S before iterating through time steps up to Tmax.The algorithm explicitly initializes x1 and applies the state-preprocessing function φ.
- Training Process: The agent selects an action with an ϵ-greedy policy, observes reward R and the next input, preprocesses the next state, and stores the experience tuple in D.The stored tuple is (S, A, R, S′), after which the next state replaces the current state.
- Training Process: Learning samples random mini-batches from replay memory, computes targets for nonterminal transitions, and updates the network using Adam.The target uses the reward plus the discounted maximum target-network estimate.
- Training Process: Every N steps, the target-network weights are updated, ϵ is decayed toward ϵend, and the current episode score is stored.The decay rule is ϵ ← max(ϵend, ϵ ∗ decay).
1) Initialization:
The initialization stage prepares the DRL training components and begins each episode with raw input preprocessing before time-step interaction.
- Initialization: The algorithm initializes replay memory, learning and target networks, scores, and ϵ before processing each episode’s initial raw input.The initial state is formed as S ← φ(< x1 >).
- Initialization: The learning network starts with random weights and is replicated to the target network before iterative data generation and episode training.The episode ends when its time horizon is reached, as described in the passage.
2) Exploration and Data Acquisition:
The DRL agent gathers experience tuples through ε-greedy interaction with the MEC environment and trains using replayed batches. Experience replay reduces correlation-related instability, while learning and target networks are updated through loss minimization and periodic parameter copying.
- Exploration and Data Acquisition: The agent uses ε-greedy interaction to explore the MEC environment, select greedy or alternative actions, and collect state, action, reward, and next-state tuples.Each interaction generates a tuple <s_τ, a_τ, r_τ, s_τ+1> that is stored for training.
- Replay Experience Buffer: Experience replay stores tuples in a fixed-size queue and samples them uniformly at random instead of training on sequential interactions.The oldest entries are removed when the buffer is full.
- Replay Experience Buffer: Random replay decouples sequential correlations, permits repeated learning from entries, recalls rare occurrences, and reduces oscillation or divergence from outlier samples.Batch sampling also helps mitigate overfitting caused by biased training-sample distributions.
- Network Training: During forward propagation, batches from the experience buffer are fed to learning and target networks, and their reward-prediction errors determine the loss.The local learning network is then updated using backpropagation.
- Network Training: The loss gradient is simplified by removing terms independent of learning-network parameters and substituting the target with the discounted maximum next-state Q value.The parameter update uses Adam, and learning-network parameters are copied to the target network every N episodes.
5) Reward Clipping:
Reward clipping limits extreme reward and loss values to stabilize learning, while the end-to-end DRL model learns offloading decisions and resource allocation over dynamic MEC conditions. Simulations compare convergence, rewards, task completion, and energy consumption across proposed and baseline methods.
- Reward Clipping:: Clipping rewards and loss errors mitigates oscillation and supports smoother convergence under variable MEC features.Values below min are replaced with min, and values above max are replaced with max.
- Learning Curves: Around 500 episodes, the proposed E2E DRL and existing DRL models converge, with E2E DRL achieving higher rewards.Existing DRL models show larger variances because traditional optimization handles part of the decisions.
- Reward Comparison: The proposed DRL algorithm significantly outperforms greedy and other DRL models in reward comparison.DRL models obtain more rewards with more users, whereas the greedy algorithm performs poorly as users increase.
- Task Completion and Energy: The proposed method outperforms existing DRL models and the greedy algorithm in task completion and energy-consumption comparisons.Performance increases while DRL models learn from replay-buffer data, whereas the greedy algorithm remains almost unchanged.
- Energy Consumption: The proposed DRL model learns to reduce energy cost over time while maximizing long-term accumulated rewards.The DRL model using CVX can save energy but does not increase the number of completed tasks because it controls only part of the decision variables.
- Energy–Task Trade-off: Adjusting reward-function weights allows the network operator to balance completed tasks against energy cost.Energy peaks can arise from higher task computational cycles or exploitation costs, while energy consumption scales with the square of CPU frequency.
V. CONCLUSION
The paper presents an end-to-end DRL method for dynamic MEC computation offloading that jointly optimizes edge-server selection and computing-power allocation. It aims to maximize on-time task completion and minimize energy consumption while optimizing long-term rewards.
- The proposed method jointly optimizes edge-server selection and computing-power allocation in a dynamic MEC network.
- It simultaneously maximizes tasks completed on time and minimizes energy consumption.
- The DRL method maximizes long-term accumulated rewards rather than optimizing only a single time step.
- The model makes decisions without relying on additional optimization functions for joint optimization.
- Experience replay and clipping facilitate training by preventing oscillation and divergence.
- Future work will study dynamic offloading with arbitrarily partitioned tasks.