Source-linked AI summary
Deep Reinforcement Learning-based Methods for Resource Scheduling in Cloud Computing: A Review and Future Directions
Guangyao Zhou, Wenhong Tian, Rajkumar Buyya, Ruini Xue, Liang Song
TL;DR
Cloud scheduling must manage complex, large-scale systems while addressing resource use, cost, service quality, and environmental effects. This paper surveys DRL-based scheduling, analyzes RL and DRL frameworks, and reviews applications, advantages, challenges, and future directions. The reviewed studies report adaptability across continuous or high-dimensional scheduling settings and better performance than several commonly compared algorithms.
Problem
Cloud scheduling is difficult because realistic environments have implicit objectives, high-dimensional or continuous states, large-scale requests, and complex physical dynamics.
Method
The paper formulates Cloud scheduling, reviews classic and DRL-based algorithms, analyzes RL and DRL through their mapper structures, and organizes reviewed literature structurally.
Results
The reviewed literature reports that DRL adapts to continuous or high-dimensional states, varied scenarios and objectives, and outperforms several commonly compared algorithms.
Takeaways & Limitations
DRL-based scheduling is presented as effective for complex Cloud scenarios, including dynamic resource scheduling with dependent tasks and heterogeneous servers.
Takeaways & Limitations
Classic algorithms remain applicable mainly when scheduling elements such as time, cost, energy, and load are given or easy to calculate, while realistic DRL modeling still requires theoretical and object-level research.
Abstract
from arXiv · showhide
As the quantity and complexity of information processed by software systems increase, large-scale software systems have an increasing requirement for high-performance distributed computing systems. With the acceleration of the Internet in Web 2.0, Cloud computing as a paradigm to provide dynamic, uncertain and elastic services has shown superiorities to meet the computing needs dynamically. Without an appropriate scheduling approach, extensive Cloud computing may cause high energy consumptions and high cost, in addition that high energy consumption will cause massive carbon dioxide emissions. Moreover, inappropriate scheduling will reduce the service life of physical devices as well as increase response time to users' request. Hence, efficient scheduling of resource or optimal allocation of request, that usually a NP-hard problem, is one of the prominent issues in emerging trends of Cloud computing. Focusing on improving quality of service (QoS), reducing cost and abating contamination, researchers have conducted extensive work on resource scheduling problems of Cloud computing over years. Nevertheless, growing complexity of Cloud computing, that the super-massive distributed system, is limiting the application of scheduling approaches. Machine learning, a utility method to tackle problems in complex scenes, is used to resolve the resource scheduling of Cloud computing as an innovative idea in recent years. Deep reinforcement learning (DRL), a combination of deep learning (DL) and reinforcement learning (RL), is one branch of the machine learning and has a considerable prospect in resource scheduling of Cloud computing. This paper surveys the methods of resource scheduling with focus on DRL-based scheduling approaches in Cloud computing, also reviews the application of DRL as well as discusses challenges and future directions of DRL in scheduling of Cloud computing.
1 INTRODUCTION
Cloud computing provides elastic distributed services but its scale, uncertainty, and resource demands make efficient scheduling difficult. The paper surveys scheduling methods with a focus on DRL, analyzes RL/DRL frameworks, and identifies challenges and future directions.
- Cloud computing and scheduling challenges: Cloud computing delivers dynamic services through large-scale distributed infrastructure using resources including CPU, RAM, GPU, storage, bandwidth, time, and space.Its physical resources consume electrical energy over the platform’s service life and deployment space.
- Cloud computing and scheduling challenges: Huge device scale, complex scenarios, unpredictable requests, component randomness, and uncertain temperatures challenge effective Cloud resource scheduling.These characteristics complicate scheduling across diverse operating conditions.
- Cloud computing and scheduling challenges: Cloud scheduling uses a scheduler to collect user tasks, generate scheduling schemes, and allocate tasks to server nodes that provide services.The process connects client requests, the Cloud center, scheduling algorithms, and server execution.
- Prior approaches: Classical scheduling approaches include dynamic programming, randomization, heuristic, meta-heuristic, hybrid, and machine-learning methods, alongside extensive prior surveys.Existing reviews cover workflow scheduling, virtual-machine placement, evolutionary methods, and machine learning in Cloud and edge-Cloud systems.
- Paper scope and contributions: The paper comprehensively reviews Cloud scheduling algorithms, analyzes RL and DRL frameworks by model structure, surveys DRL-based scheduling research, and discusses challenges and future directions.The paper organizes these analyses across scheduling formulation, RL/DRL structure, reviewed applications, and future research.
2 SCHEDULING AND ALGORITHMS IN CLOUD
Cloud scheduling formulates task-to-resource allocation and execution timing as an optimization problem across heterogeneous, dynamic systems. The review contrasts classic algorithms with DRL, emphasizing the former’s modeling limits and DRL’s adaptability to implicit objectives and changing environments.
- Scheduling Objectives: Scheduling objectives include minimizing energy consumption, makespan, delay, and response time while improving load balancing, reliability, utilization, and provider profit.The problem is usually NP-hard in distributed systems and may involve multiple objective functions over the scheduling scheme and system parameters.
- Mathematical Formulation: Cloud scheduling assigns indivisible tasks to server nodes and determines their execution start times across multiple resource dimensions.A scheduling scheme combines allocation matrix X with start-time matrix S; resources include CPU, GPU, RAM, bandwidth, disk storage, time, and space.
- Dynamic Scheduling: Dynamic scheduling must evaluate how a decision at time t affects subsequent server states, making prediction important for assessing scheme quality.The scheduling scheme must satisfy current requirements while accounting for later system changes.
- Classic Algorithms: Heuristic methods mainly target single objectives, whereas meta-heuristics address more complex single- and multi-objective problems at the cost of computational complexity and randomness.Meta-heuristic objectives include cost, energy consumption, makespan, running time, and resource utilization, often under simplified assumptions.
- Classic Algorithms: Classic algorithms may fail to evaluate implicit objectives in realistic stochastic settings, preventing guaranteed optimality or even feasible solutions.They may also need to resolve each new optimization problem from scratch because they lack memorability.
- Machine Learning: DRL combines deep neural networks with reinforcement learning to model complex systems, adapt to objectives and environments, process multidimensional states, and retain experience.The review surveys DRL-based Cloud scheduling research and identifies challenges and future directions for more realistic scenarios.
3 ANALYSIS OF RL AND DRL FRAMEWORKS IN SCHEDULING
The paper traces RL frameworks from agent–environment interaction toward DRL architectures that use neural networks for complex Cloud scheduling. This evolution addresses time-varying states, extrinsic stimuli, high-dimensional inputs, and implicit scheduling mappings.
- Section Scope: The paper surveys DRL-based Cloud scheduling research, analyzes RL and DRL framework evolution, and identifies challenges and future directions for realistic scenarios.Its review is motivated by DRL’s reported performance and potential in Cloud scheduling.
- RL Framework Evolution: RL learns action strategies through agent–environment interaction, using state, action, environment, feedback, and strategy updates.The agent selects actions from its strategy and updates that strategy using timely and long-term environmental feedback.
- RL Framework Evolution: Classical RL frameworks evolve from basic action selection and strategy updates toward structures incorporating changing agent-states and extrinsic stimuli.The basic action-selection framework is insufficient for time-related scheduling, while later structures model real-time state changes and external interference.
- Motivation for DRL: As state and action spaces grow in size and dimension, conventional RL representations face exponential computational and storage costs, especially for non-discrete scheduling states.The paper identifies Q-tables, advantage functions, policy gradients, and related mappings as difficult to apply in these settings.
- Motivation for DRL: DRL integrates DNNs into decision-making to handle more complex scenarios than conventional RL frameworks.The DNN serves as a practical information-perception and decision component when increasing input complexity makes non-neural RL frameworks unsuitable.
- Generalized DRL Framework: The generalized RL framework represents scheduling as interacting mappers for time variation, stimulus evolution, decision, environment dynamics, and feedback.In Cloud computing, large-scale requests and physical conditions act as time-varying stimuli, while several mapper relationships may require implicit-function modeling.
4 OVERVIEW OF DRL-BASED SCHEDULING IN CLOUD
The review organizes DRL-based Cloud scheduling research by decision mappers, objectives, scenarios, experimental data, and baselines. It finds broad applicability across dynamic scheduling settings and reports that DRL methods outperform many commonly compared algorithms.
- Framework analysis: DNNs commonly serve as DRL decision-makers, while DDQN adds a target network as a feedback mapper.The review frames scheduling around producing schemes and evaluating their quality.
- Framework analysis: Reviewed methods combine Q-learning, DQN, DDQN, actor–critic, recurrent, convolutional, and other network structures for Cloud scheduling.Examples include Q-tables, CNNs, LSTMs, seq2seq networks, actor–critic networks, and federated learning.
- Applications and objectives: DRL-based methods address diverse optimization objectives and are mainly applied to dynamic or online scheduling.The reviewed scenarios include heterogeneous resources, independent or dependent tasks, and non-preemptive tasks.
- Applications and objectives: DRL methods support scheduling across heterogeneous resources, independent or dependent tasks, workflows, and homogeneous or heterogeneous servers.The review identifies dynamic multi-resource scheduling in Cloud or Edge-Cloud environments as the scenario closest to realistic operation.
- Experimental comparisons: Experiments in the reviewed literature report better performance than commonly compared algorithms, including Randomization, FCFS, Round-robin, Greedy, Q-learning, MDP, FIFO, HEFT, and conventional DQN.The paper presents these methods as baselines for evaluating future DRL-based scheduling algorithms.
- Experimental comparisons: DDQN is the most commonly used reviewed model and uses action-value and target-Q networks with the same structure.The action-value network generates current-state action values, while the target network contributes to training loss.
5 CHALLENGES AND FUTURE DIRECTIONS FOR DRL-BASED SCHEDULING
The paper identifies computational, predictive, theoretical, and explainability challenges that limit DRL-based scheduling in realistic large-scale Cloud systems. It proposes combining DRL with other methods, improving training and mappers, using DRL as an algorithm selector, and strengthening baselines and benchmarks.
- Challenges: DRL training and computation consume substantial resources, making deployment in large-scale or multi-cluster systems costly and time-consuming.The paper states that DRL may require a period of training before practical use and is not cost-effective for single, computable objectives.
- Challenges: DRL scheduling results remain unpredictable, and extremely poor schemes can create a nonzero risk of system collapse.The paper also notes that worst-case performance is difficult to evaluate.
- Challenges: Performance depends on DNN state perception, limited training coverage, and retraining for new scenarios.Real scheduling involves dynamic tasks without preemptive or prior knowledge, while training data may not cover real-system scenarios.
- Challenges: Gradient descent and Bellman-based Q-learning can converge locally, while unavailable reliable public datasets widen the gap from theoretically optimal solutions.The paper also identifies unexplainable training processes and difficulties deriving theory for high-dimensional continuous state spaces.
- Future directions: Existing DRL work mainly assumes homogeneous Markov processes, motivating theoretical research on nonhomogeneous and hidden Markov settings.The paper connects this direction to mathematical theory and modeling of realistic physical processes.
- Future directions: Future work should combine DRL with other policies, accelerate convergence, reduce computational complexity, and improve mapper architectures.Suggested techniques include meta-heuristics, imitation learning, queue models, CNNs, LSTMs, and Transformers.
- Future directions: DRL can assist analyzable schedulers, select among scheduling algorithms, and treat algorithms themselves as selectable resources for changing scenarios.The paper presents algorithm-selection frameworks because a single algorithm is not recommended for all scenarios.
- Future directions: Future baselines and benchmarks should be reproducible, stable, multi-scenario, dynamically scaled, fairly compared, and tested across devices and extreme cases.The paper lists properties intended to reduce parameter-induced bias and improve experimental verification.
6 CONCLUSIONS
The paper formulates Cloud scheduling, reviews classic and DRL-based approaches, and synthesizes their applications, challenges, and future directions. It concludes that DRL offers adaptability and portability, while algorithm selection and improved theory remain open research directions.
- Contributions and conclusions: The review identifies solution production and evaluation as two key factors in Cloud scheduling and uses them to motivate DRL-based methods.It argues that classic algorithms struggle when optimization objectives are implicit or difficult to evaluate.
- Contributions and conclusions: DRL is reported to adapt across scenarios and optimization objectives because DNNs can represent higher-dimensional or continuous agent state spaces.The paper emphasizes cases where objectives are implicit or hard to calculate.
- Challenges and future work: The review highlights complexity, unexplainability, local convergence, and unpredictable results as the main challenges of DRL-based Cloud scheduling.It also identifies combining policies, improving DRL structures, and developing algorithm selectors as future directions.
- Challenges and future work: Combining DRL with other algorithm types remains a research direction, including treating algorithms as resources for scheduling selection.The paper presents this as a way to address more difficult scheduling problems.