Source-linked AI summary
Dynamic Scheduling for Stochastic Edge-Cloud Computing Environments using A3C learning and Residual Recurrent Neural Networks
Shreshth Tuli, Shashikant Ilager, Kotagiri Ramamohanarao, Rajkumar Buyya
TL;DR
Dynamic Edge-Cloud scheduling must manage constrained, heterogeneous resources and stochastic workloads while meeting QoS and response-time requirements. The paper proposes a decentralized A3C scheduler using R2N2 to exploit temporal patterns, reporting improved efficiency across energy, response time, SLA violations, and cost against existing approaches.
Problem
Efficient scheduling in integrated Edge-Cloud environments is difficult because resources are heterogeneous and constrained, workloads are stochastic, and device or edge-node mobility changes network and response-time characteristics.
Method
The paper combines asynchronous policy-gradient A3C with an R2N2 framework and multiple schedulers to learn adaptive placement policies across distributed Edge-Cloud hosts.
Results
The proposed methodology achieves significant efficiency improvements in energy consumption, response time, Service Level Agreements, and cost compared with existing heuristics and previously proposed reinforcement-learning models.
Takeaways & Limitations
R2N2 improves scheduling adaptability by using temporal patterns and responding to changes in network, workload, and device characteristics.
Takeaways & Limitations
The current model schedules a fixed number of edge nodes and tasks, and real deployment would require profiling and synchronizing resource usage across A3C agents.
Abstract
from arXiv · showhide
The ubiquitous adoption of Internet-of-Things (IoT) based applications has resulted in the emergence of the Fog computing paradigm, which allows seamlessly harnessing both mobile-edge and cloud resources. Efficient scheduling of application tasks in such environments is challenging due to constrained resource capabilities, mobility factors in IoT, resource heterogeneity, network hierarchy, and stochastic behaviors. xisting heuristics and Reinforcement Learning based approaches lack generalizability and quick adaptability, thus failing to tackle this problem optimally. They are also unable to utilize the temporal workload patterns and are suitable only for centralized setups. However, Asynchronous-Advantage-Actor-Critic (A3C) learning is known to quickly adapt to dynamic scenarios with less data and Residual Recurrent Neural Network (R2N2) to quickly update model parameters. Thus, we propose an A3C based real-time scheduler for stochastic Edge-Cloud environments allowing decentralized learning, concurrently across multiple agents. We use the R2N2 architecture to capture a large number of host and task parameters together with temporal patterns to provide efficient scheduling decisions. The proposed model is adaptive and able to tune different hyper-parameters based on the application requirements. We explicate our choice of hyper-parameters through sensitivity analysis. The experiments conducted on real-world data set show a significant improvement in terms of energy consumption, response time, Service-Level-Agreement and running cost by 14.4%, 7.74%, 31.9%, and 4.64%, respectively when compared to the state-of-the-art algorithms.
1 INTRODUCTION
IoT growth and latency-sensitive applications motivate Edge-Cloud computing, but heterogeneous, mobile, resource-constrained, and stochastic environments make task scheduling difficult. The paper proposes decentralized A3C scheduling with an R2N2 framework to exploit temporal patterns and improve adaptation.
- Motivation: Edge-Cloud computing addresses latency-sensitive IoT applications, while edge resources remain limited by cost and feasibility.Traditional cloud-centric deployments can suffer increased network latency for time-critical applications.
- Problem: Heterogeneous capacities, speeds, response times, energy use, mobility, and changing bandwidth complicate continual scheduling optimization.These differences occur both between cloud and edge layers and within each layer.
- Research gap: Heuristic policies fail to account for dynamic workloads and continuous system changes, while prior value-based RL methods are poorly suited to highly stochastic Edge-Cloud deployments.Existing policy-gradient work also commonly optimizes only a single QoS parameter and lacks asynchronous updates or temporal-pattern exploitation.
- Approach: The paper proposes A3C policy-gradient scheduling with multiple asynchronous agents, enabling decentralized learning and rapid adaptation across edge or cloud nodes.Agents train in parallel, periodically update shared global parameters, then continue independent exploration and training.
- Evaluation: The proposed model shows improved energy, response time, SLA violation, and cost efficiency against heuristics and previously proposed RL models.Experiments use an extended iFogSim Toolkit with elements of CloudSim 5.0.
- Approach: R2N2 captures temporal workload, network, and node patterns for scheduling in a hybrid Edge-Cloud setup.The contribution includes an architectural system model, a generic asynchronous learning model, and a residual recurrent framework.
2 SYSTEM MODEL AND PROBLEM FORMULATION
The system model represents a heterogeneous edge-cloud infrastructure whose resource-management system schedules and migrates tasks under changing workloads. The problem formulation maps interval states to allocation and migration actions to minimize scheduling loss.
- 2.1 System Model: Edge-cloud infrastructure combines distributed, heterogeneous resources from nearby edge nodes to multi-hop remote cloud nodes.Edge devices offer lower response times but constrained computation, whereas cloud nodes provide greater computational capacity and concurrent processing.
- 2.1 System Model: The Resource Management System receives tasks with QoS and SLA requirements and uses CPU, RAM, bandwidth, disk, and deadline information for scheduling decisions.It includes scheduling, migration, and resource-monitoring services, and periodically evaluates whether active tasks should move to new hosts.
- 2.1 System Model: Multiple schedulers handle separate task and node partitions while interacting with a deep reinforcement learning module that recommends task placements.The schedulers may run on one node or across edge-cloud nodes, enabling asynchronous parameter updates among multiple actors.
- 2.2 Workload Model: Execution is divided into equal-duration scheduling intervals because task computation and bandwidth requirements change with user demand and IoT-device mobility.The ith interval, SIi, starts at ti and ends at ti+1; active tasks are tracked across successive intervals.
- 2.3 Problem Formulation: Scheduler performance is quantified by Lossi for each interval, with lower Loss indicating better scheduler performance.The optimization target is defined over the dynamic edge-cloud environment and stochastic workload model.
- 2.3 Problem Formulation: At the beginning of SIi, the system state contains host parameters, remaining active tasks, and new tasks, which determine allocation and migration actions.The model selects hosts for migratable active tasks and new tasks, while non-migratable tasks are excluded from migration decisions.
- 2.3 Problem Formulation: The scheduler is formalized as Model: Statei → Actioni, subject to applying its action to every migratable or newly arriving task.The interval loss depends on task allocation actions, so the optimization seeks an effective mapping from observed state to host decisions.
- 2.3 Problem Formulation: A symbol table and Venn diagram accompany the mathematical formulation to clarify notation and relationships among task sets.These supporting elements are presented with the problem formulation rather than introducing additional scheduling results.
3 REINFORCEMENT LEARNING MODEL
The model represents host and task conditions as a state, converts ranked host preferences into feasible assignments, and optimizes a weighted loss incorporating performance, cost, SLA, and constraint penalties.
- Input and output specification: The scheduler state combines host utilization and capacity characteristics with new-task requirements and previous host assignments for active tasks.
- Input and output specification: The model outputs host assignments for new tasks and migration decisions for eligible active tasks, subject to feasibility and capacity constraints.
- Input and output specification: The unconstrained policy output is a ranked list of host preferences for each task, which the constraint module converts into executable assignments.
- Loss function: The penalty combines migration failures with higher-ranked unsuitable host preferences, guiding learning toward constraint-compliant decisions.
- Loss function: The loss is a convex combination of normalized energy, response time, migration time, cost, and SLA-violation metrics, weighted by application-specific hyperparameters.
4 STOCHASTIC DYNAMIC SCHEDULING USING
The scheduling procedure uses an R2N2 policy model within a repeated state–action–execution loop, updating parameters asynchronously from accumulated experience to adapt to changing environments and requirements.
- Scheduling procedure: At each interval, task requests and host characteristics are processed by the learning model, whose output is converted into scheduling and migration actions.
- R2N2 architecture: The R2N2 network uses recurrent layers with skip connections to model temporal relationships between task and host characteristics while improving gradient propagation.
- R2N2 architecture: Its actor head produces a 100 × 100 task–host probability map, supporting at most 100 tasks and 100 hosts in the current configuration.
- Learning and updates: The critic head estimates cumulative loss, while back-propagation uses reward, predicted loss, and one-step look-ahead error to update the network.
- Scheduling procedure: Algorithm 1 repeatedly preprocesses state, obtains a probability map, applies constraint satisfaction, executes allocations, and updates the model over N intervals.
- Learning and updates: Experience is accumulated over 12 scheduling intervals before parameter updates, balancing slow updates from large batches against variance from small batches.
- Complexity: Ignoring GPU-based forward and backward passes, the total preprocessing and constraint-processing complexity over N intervals is O(abN).
5 PERFORMANCE EVALUATION
The evaluation section introduces comparisons of the proposed deep-learning scheduler with baseline heuristics and related reinforcement-learning methods across selected metrics.
- The experiments compare the proposed framework with several baseline algorithms using defined evaluation metrics and detailed result analysis.
5.1 Experimental Set Up
The experiments simulate stochastic scheduling in a heterogeneous edge–cloud environment using five-minute intervals and real Bitbrain workload traces, with separate training and testing partitions.
- Simulation environment: The simulation extends iFogSim and CloudSim, using five-minute scheduling intervals for comparison with baseline algorithms.
- Dataset and workload: Dynamic cloudlet workloads are generated from real Bitbrain traces containing resource-consumption data from over 1000 virtual machines.
- Dataset and workload: The traces provide five-minute CPU, RAM, network, and disk workload characteristics, whose volatility is illustrated for a random node.
- Dataset and workload: The dataset is partitioned into 75% training workloads and 25% testing workloads for evaluation, sensitivity analysis, and comparison with related work.
- Task generation: The workload generator samples the number and duration of new tasks, while the simulation limits the system to 100 hosts and 100 active tasks across 10 actor-agents.
- Heterogeneous infrastructure: The infrastructure models edge nodes with 1 ms response time and constrained resources alongside cloud nodes with 10 ms response time and greater capacity.
- Heterogeneous infrastructure: Testing runs for one day, corresponding to 288 scheduling intervals, using heterogeneous host configurations and a cost model.
5.2 Evaluation Metrics
The evaluation considers energy, responsiveness, SLA adherence, cost, and task-completion and migration measures for Edge-Cloud scheduling.
- Total energy consumption measures integrated host power over the complete simulation duration.
- Average response time measures the average time required to respond to tasks.
- SLA violations quantify failures to meet service-level agreements for tasks.
- Total cost measures integrated host cost over the complete simulation duration.
- Additional metrics include completion time, completed tasks, expected-time completion fraction, task migrations, and migration time.Completion time combines scheduling, execution, and host response time.
5.3 Baseline Algorithms
The study compares A3C-R2N2 with heuristic scheduling policies and standard deep reinforcement-learning baselines implemented for the same problem.
- The heuristic baselines combine sub-heuristics for overload detection and task or virtual-machine selection, using Best Fit Decreasing for target-host identification.
- LR-MMT combines Local Regression overload detection with Minimum Migration Time task selection.
- MAD-MC combines Median Absolute Deviation overload detection with Maximum Correlation Policy task selection.
- DDQN is implemented as an optimized Double Deep Q-Learning baseline.
- DRL (REINFORCE) uses policy-gradient learning with a fully connected neural network, without asynchronous updates or a recurrent network.
- The comparison RL models use the same state representation for fairness, with DQN updated through the Bellman equation.
5.4 Analysis of Results
Sensitivity experiments show that individually optimized networks specialize in different metrics, while combined optimization yields a better overall model; A3C-R2N2 also scales with parallel agents and outperforms baselines across key measures.
- Sensitivity Analysis: Individual loss functions produce specialized networks such as EMN, CMN, RTMN, SLAVMN, and migration-minimizing networks.The experiments vary hyper-parameters and evaluate the resulting metric-specific scheduling behavior.
- Sensitivity Analysis: Seven migrations occur for EMN versus zero for the other networks among 30 initial tasks.
- Sensitivity Analysis: Combined optimization produces lower loss and a better network than optimizing a single evaluation metric.Single-variable optimization may reach a local optimum.
- Overhead Analysis: Increasing recurrent layers decreases loss, while scheduling overhead depends on the simulation system.
- Scalability Analysis: With 100 actor agents, A3C-R2N2 achieves speedup 34.3 and efficiency 0.37, while efficiency declines as agents increase because of communication delays and GPU-less agents.
- Evaluation with Baseline Algorithms: A3C-R2N2 has slightly higher scheduling overhead because the R2N2 network updates hourly, but the reported 0.002% difference from RL is described as negligible.
- Evaluation with Baseline Algorithms: 14.4% lower energy consumption and 7.74% lower average response time are reported against the best corresponding baselines.
- Evaluation with Baseline Algorithms: 31.9% fewer SLA violations and 4.64% lower total cost are reported compared with REINFORCE.
5.5 Summary of insights
The results attribute A3C-R2N2's performance to adaptation to changing workloads, networks, and device characteristics, while recurrent processing captures temporal scheduling patterns.
- A3C-R2N2 adapts scheduling policies to changing network, workload, and device characteristics through asynchronous policy gradients.
- Compared with REINFORCE-Dense, the recurrent strategy uses temporal workload patterns to reduce sub-optimal allocation decisions and migration time.
- The strategy distinguishes edge and cloud placement as task resource requirements change, assigning intensive tasks to cloud when edge execution may slow the node.
6 RELATED WORK
Prior Edge-Cloud scheduling research has used heuristics and learning-based optimization, but dynamic workloads, changing environments, and limited adaptability remain central challenges. Existing DRL studies also narrow their metrics or omit asynchronous and recurrent architectures.
- Heuristics generally target specific application parameters and fail to respond to dynamic environmental changes.
- Predictive optimization studies commonly use machine learning or deep reinforcement learning, often prioritizing energy efficiency.
- Value-based reinforcement learning methods face scalability problems when representing edge-cloud states through tables or neural-network Q functions.
- Prior REINFORCE-based edge-only schedulers consider response time but do not exploit asynchronous or recurrent networks for adaptability and robustness.
- Relevant-work comparisons define dynamic scheduling as continuously optimizing active and newly arriving tasks under changing arrival rates and resource consumption.
7 CONCLUSIONS AND FUTURE WORK
The paper presents an A3C-R2N2 scheduler for real-time task allocation across integrated edge-cloud environments with dynamic workloads. It reports low scheduling overhead, while identifying real-world deployment, scalability, privacy, and security as future directions.
- The A3C-R2N2 scheduler uses asynchronous updates and recurrent modeling to incorporate host, task, and temporal workload information.
- 0.002% scheduling overhead is reported relative to the existing baseline for dynamic task scheduling in stochastic environments.
- The current model schedules a fixed number of edge nodes and tasks, so scalability testing and scalable reinforcement-learning alternatives remain future work.
- Real-world implementation would require profiling and synchronizing resource usage across A3C agents, alongside future investigation of data privacy and security.