Source-linked AI summary
Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing
Jie Xu, Lixing Chen, Shaolei Ren
TL;DR
Energy-harvesting mobile edge computing must manage costly or infeasible grid alternatives while coping with intermittent renewable supply. The paper formulates joint offloading and server provisioning as an online learning problem, develops a decomposed reinforcement-learning algorithm, and reports improved edge-computing performance with a convergent, monotone policy.
Problem
Grid power for mobile edge computing can be costly or infeasible in some areas, while intermittent and unpredictable renewables complicate reliable service.
Method
The paper uses an MDP and a post-decision-state reinforcement-learning algorithm to learn joint cloud offloading and edge-server provisioning policies online.
Results
The proposed scheme significantly improves edge-computing performance under intermittent and unpredictable renewable energy.
Takeaways & Limitations
Foresightedness and adaptivity are identified as keys to reliable and efficient operation of energy-harvesting MEC systems.
Abstract
from arXiv · showhide
Mobile edge computing (a.k.a. fog computing) has recently emerged to enable in-situ processing of delay-sensitive applications at the edge of mobile networks. Providing grid power supply in support of mobile edge computing, however, is costly and even infeasible (in certain rugged or under-developed areas), thus mandating on-site renewable energy as a major or even sole power supply in increasingly many scenarios. Nonetheless, the high intermittency and unpredictability of renewable energy make it very challenging to deliver a high quality of service to users in energy harvesting mobile edge computing systems. In this paper, we address the challenge of incorporating renewables into mobile edge computing and propose an efficient reinforcement learning-based resource management algorithm, which learns on-the-fly the optimal policy of dynamic workload offloading (to the centralized cloud) and edge server provisioning to minimize the long-term system cost (including both service delay and operational cost). Our online learning algorithm uses a decomposition of the (offline) value iteration and (online) reinforcement learning, thus achieving a significant improvement of learning rate and run-time performance when compared to standard reinforcement learning algorithms such as Q-learning. We prove the convergence of the proposed algorithm and analytically show that the learned policy has a simple monotone structure amenable to practical implementation. Our simulation results validate the efficacy of our algorithm, which significantly improves the edge computing performance compared to fixed or myopic optimization schemes and conventional reinforcement learning algorithms.
I. INTRODUCTION
The paper frames renewable-powered mobile edge computing as a joint delay, energy, offloading, and server-provisioning problem under intermittent supply. It proposes and analyzes an online learning approach that optimizes this system adaptively.
- Motivation: Mobile edge computing processes workloads near users to reduce transmission latency, but edge operation depends on effective power provisioning.Base stations and related edge devices provide cloud-like computing and storage capabilities close to users and data sources.
- Motivation: Grid power can be costly or infeasible in remote and hazardous areas, motivating harvested renewable energy as a major or sole power source.The paper cites unreliable grids, diesel generation, carbon footprint, and electricity prices as supporting motivations.
- Problem: Intermittent renewable supply creates a trade-off between transmission delay and processing delay because edge capacity varies with available power.Batteries buffer energy, but limited computing capacity and time-varying supply can cause substantial processing time at the edge.
- Approach: The paper jointly controls workload offloading to the centralized cloud and dynamic edge-server provisioning to minimize long-term service delay and operational cost.The problem is formulated as a Markov decision process using workload, congestion, battery, and anticipated renewable-arrival information.
- Approach: A post-decision-state learning algorithm decomposes offline value iteration and online reinforcement learning to improve convergence speed and run-time performance over Q-learning.The method exploits the energy-harvesting system’s transition structure and permits batch learning of algorithm components.
- Results: The analysis proves convergence and shows that optimal power demand is non-decreasing in available battery power, enabling practical implementation.Simulations further report significant performance improvements for the energy-harvesting MEC system.
II. RELATED WORK
Prior MEC research studies user-side workload offloading, while this paper addresses edge-side offloading and autoscaling in energy-harvesting MEC. The paper develops a reinforcement-learning approach that exploits partial system information and problem structure.
- MEC supports lower-latency, more energy-efficient, context-aware, private, and secure mobile applications than mobile cloud computing.
- Prior work mainly studies user-side decisions about what, when, and how to offload workloads to edge systems or clouds.
- This paper focuses on edge-side workload offloading and autoscaling, complementing user-side offloading studies.
- Energy-harvesting MEC jointly manages radio and computation resources, unlike prior research that typically considers only one decision.
- The paper formulates energy-harvesting MEC resource management as an MDP and develops a foresighted algorithm for unknown, time-varying environments.
- Factoring known and unknown dynamics lets the algorithm exploit partial information and structure, yielding faster convergence than conventional reinforcement learning such as Q-learning.
- The paper analytically characterizes the learned optimal policy and evaluates its performance through extensive simulations.
III. SYSTEM MODEL
The system is a base station with co-located edge servers sharing a power supply. In each time slot, it observes workload and transmission conditions, chooses local workload processing and active-server capacity, and offloads the remainder to the cloud.
- The modeled edge system consists of a base station and co-located edge servers sharing the cell site’s power supply.
- Operation is divided into equal-length discrete time slots aligned with the timescale for adjusting active-server capacity.
- The system state includes workload arrival, location, wireless transmission conditions, and the number of active servers.
- The total arrival rate is λ(t), the locally processed workload is µ(t) ≤ λ(t), and the remainder ν(t) = λ(t) − µ(t) is sent to the cloud.
- The number of active servers is selected at the beginning of each slot and remains fixed during job execution within that slot.
B. Delay cost model
The model represents service delay and energy demand for local processing and cloud offloading, while renewable power is uncertain but partially anticipated through an observable environment state.
- Local delay reflects processing at capacity-limited edge servers, with negligible transmission delay because the devices and servers are co-located.
- The generic local delay function clo(m(t), µ(t)) can represent average or tail delay and can be instantiated using an M/G/1 queue.
- Cloud-offloading delay is modeled mainly by transmission round-trip time and absorbed cloud service delay, parameterized by network congestion h(t).
- The offloading delay cost is coff(h(t), λ(t), µ(t)) = (λ(t) − µ(t))h(t).
- Total power demand combines basic operation and transmission power with computing power that depends on active servers and locally processed workload.
- Renewable power g(t) is realized after decisions, while observable environment state e(t) provides information about its anticipated budget.
- Given e(t), green power is modeled as conditionally distributed i.i.d. across slots, although e(t) itself may not be i.i.d.
D. Battery model
The battery buffers uncertain renewable supply against edge-system demand, with backup power required when stored energy cannot support basic operation. The resulting online control problem is formulated as an MDP over workload, environment, congestion, and battery states.
- D. Battery model: Battery state b(t) lies between 0 and capacity B and is measured at the beginning of each time slot.
- D. Battery model: When basic operation demand exceeds the battery level, all workload is offloaded, backup power maintains operation, and the battery evolves as b(t + 1) = b(t) + g(t).
- D. Battery model: When the battery supports basic operation, local processing is allowed only if computing demand satisfies dcom(λ(t), m(t), µ(t)) ≤ b(t) − dop(λ(t)).
- D. Battery model: Surplus renewable power recharges the battery up to B, while insufficient renewable power causes battery discharge to cover demand.
- D. Battery model: The model assumes lossless, non-leaky batteries and includes depreciation cost based on discharged power.
- Problem formulation: The online problem minimizes discounted long-term delay, battery depreciation, and backup-power costs using an MDP state s(t) = (λ(t), e(t), h(t), b(t)).
- Problem formulation: Workload, environment, and network states are modeled as finite-state Markov chains with unknown transition distributions.
- Problem formulation: The intermediate action is computing power demand a(t), which determines feasible joint offloading and autoscaling decisions while state transitions depend on a(t), not those individual actions.
V. POST-DECISION STATE BASED ONLINE LEARNING
The PDS-based approach separates known and unknown parts of system dynamics, allowing online learning to exploit partial information and accelerate optimization without prior transition distributions.
- The algorithm learns the optimal offloading and autoscaling policy online using post-decision states.It integrates partially known system dynamics into the learning process.
- Compared with Q-learning, the PDS-based learning algorithm significantly improves convergence speed and run-time performance.The improvement follows from exploiting partial information and the decomposed learning structure.
- PDS places an intermediate state after the computing power demand action but before the green power budget is realized.The post-decision battery state is virtual, while workload, environment, and congestion states remain unchanged by the power-demand action.
- Separating expectation from minimization lets the action optimization proceed without prior knowledge of system dynamics.This contrasts with normal-state Bellman equations, where expectations over unknown arrivals, environments, congestion, and green power precede minimization.
- Independence of workload, environment, congestion, and green-power evolution from battery state enables batch updates of post-decision value functions.The batch scheme updates multiple states sharing post-decision workload, environment, and congestion components.
B. The algorithm
The online algorithm repeatedly chooses power demand, then derives offloading and autoscaling actions, updates the post-decision state, and performs batch estimate updates after green power is realized.
- Batch updates reuse the realized green-power information across states because green power depends only on the current environment state.Post-decision updates cover all states sharing the relevant post-decision workload, environment, and congestion components.
- At each time slot, the algorithm observes the current state and determines power demand from the current cost and post-decision value estimates.The empirically optimal demand is obtained by solving the prescribed optimization using the current estimates.
- The corresponding offloading and autoscaling actions are selected after power demand is determined.They are obtained from the optimal action mappings for the observed state and chosen demand.
- After computing the post-decision state and realizing green power, the algorithm updates one-slot costs and normal and post-decision value estimates.The battery then evolves to the next state according to the realized green power.
VI. ALGORITHM ANALYSIS
The convergence analysis models the learning updates through an associated ordinary differential equation and shows convergence to the optimal post-decision value function and resulting policies.
- The PDS-based online learning algorithm converges to the optimal post-decision value function V ∗(˜s).The theorem requires learning rates satisfying the stated conditions.
- The associated ordinary differential equation is used to establish convergence of the proposed learning algorithm.The proof relies on the convergence equivalence between the updates and the O.D.E.
- The unique equilibrium point corresponds to the optimal post-decision value function for every post-decision state.Its existence is guaranteed through the contraction-based stability argument.
- Because the normal value function is a deterministic function of the post-decision value function, the algorithm learns the optimal power-demand, offloading, and autoscaling policies.The policy result follows from convergence of the post-decision and normal value estimates.
B. Structure of the Optimal Policy
Under convexity assumptions on delay-related costs, the paper derives convex value functions and a monotone optimal power-demand policy that increases with available battery energy.
- One-slot cost: The one-slot cost function is convex in the power-demand action.This property supports the subsequent structural analysis of the optimal policy.
- One-slot cost: The one-slot delay cost is jointly convex in autoscaling and offloading actions under the stated convexity assumptions.The communication-delay and operational-cost components preserve joint convexity.
- Value-function structure: Both optimal value functions are non-increasing and convex in battery state for fixed workload, environment, and congestion states.More battery therefore does not increase the optimal value under the paper’s assumptions.
- Optimal policy structure: The optimal power-demand policy is monotonically non-decreasing in battery state for fixed workload, environment, and congestion states.If b ≤ b′, the policy satisfies π∗(s) ≤ π(s′).
A. Simulation Setup
The simulations evaluate the proposed PDS-based learning algorithm against Q-learning, myopic optimization, and fixed-power schemes under intermittent renewable energy. Results show lower cost, faster convergence, and adaptive battery and computing-power behavior.
- Simulation Setup: The simulation uses 15-minute time slots, workload arrivals from 10 to 100 units/sec, network congestion from 20 to 60 ms/unit, and three renewable-energy states.Green power follows normal distributions with means of 200W, 400W, and 600W for low, medium, and high environmental states.
- Simulation Setup: The system models a 2 kWh battery, 300W base-station static consumption, up to 15 active edge servers, and 150W consumption per server.Each server supports a maximum service rate of 20 units/sec.
- Benchmarks: The proposed PDS-based algorithm is compared with Q-learning, myopic optimization, and fixed-power schemes.Myopic optimization uses current-state information and available battery energy, whereas fixed power applies a constant computation-power level whenever possible.
- Run-time Performance Comparison: The PDS-based learning algorithm incurs significantly lower cost than all benchmark schemes over 10000 time slots.The comparison includes Q-learning, myopic optimization, and fixed-power settings of 0.4kW and 1.0kW.
- Run-time Performance Comparison: Q-learning converges slowly in the large state space and retains a considerable performance gap after 10000 time slots, whereas PDS converges faster.The proposed scheme significantly improves convergence speed and run-time performance.
- Learned Optimal Policy: The learned policy conserves local computing power at low battery levels and increases local processing when workload is high and network congestion worsens.This foresighted behavior saves power for future decisions and reduces long-term system cost.
- Learned Optimal Policy: The PDS policy keeps the battery relatively low while remaining above the insufficient level, achieving the highest harvesting-energy efficiency.Fixed-power schemes may waste green power at high battery levels or fail to adapt to changing system dynamics.
E. Cost Composition
The cost-composition analysis shows that foresighted PDS-based control reduces backup-power costs by managing battery states conservatively. The paper concludes that foresight and adaptivity support efficient operation under intermittent renewable energy, while future work targets larger systems.
- Cost Composition: The PDS-based algorithm reduces backup-power cost by acting conservatively at low battery states and avoiding the insufficient-battery zone.Myopic optimization frequently enters that zone and incurs significant backup-power costs.
- Cost Composition: PDS-based learning and Q-learning reduce total cost by considering future system dynamics.The composition of time-average cost is shown at the end of the simulation.
- Cost Composition: Backup power accounts for 12.6% of total cost under PDS and 16.7% under myopic optimization.These fractions are much lower than those of schemes that do not consider long-term system performance.
- Optimal Offloading Strategy: The optimal offloading strategy processes more workload locally as network congestion increases and more edge servers are active.Local processing saturates after the workload arrival rate reaches a certain level.
- Conclusion: The paper identifies foresightedness and adaptivity as keys to reliable and efficient operation of renewable-powered MEC.The proposed PDS-based reinforcement-learning algorithm learns optimal offloading and autoscaling policies from unknown system parameters.
- Conclusion: Future work includes large-scale renewable-powered edge systems, such as green-power-aware geographical load balancing.This extends the study beyond the evaluated joint offloading and autoscaling setting.