PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning

Ning Liu, Zhe Li, Zhiyuan Xu, Jielong Xu, Sheng Lin, Qinru Qiu, Jian Tang, Yanzhi Wang

arXiv:1703.04221v2cs.DCcs.AI

TL;DR

Cloud resource allocation and power management require adaptive decisions despite high-dimensional state and action spaces, while limiting energy use and performance degradation. The paper proposes a hierarchical framework combining DRL-based global VM allocation with distributed local power management, and experiments report substantial energy savings with comparable latency.

  • Problem

    Complete cloud resource allocation and power management must jointly manage high-dimensional decisions while reducing power consumption and maintaining acceptable performance degradation.

  • Method

    A hierarchical framework uses DRL for global VM resource allocation, an autoencoder and weight sharing for state handling, and distributed local power management with LSTM workload prediction and model-free RL.

  • Results

    53.97% power and energy consumptions were saved in a 30-server cluster with 95,000 job requests while achieving similar average latency.

  • Takeaways & Limitations

    The framework provides a power-latency trade-off that outperforms the reported baselines in the evaluated server-cluster settings.

Abstract

from arXiv · show

Automatic decision-making approaches, such as reinforcement learning (RL), have been applied to (partially) solve the resource allocation problem adaptively in the cloud computing system. However, a complete cloud resource allocation framework exhibits high dimensions in state and action spaces, which prohibit the usefulness of traditional RL techniques. In addition, high power consumption has become one of the critical concerns in design and control of cloud computing systems, which degrades system reliability and increases cooling cost. An effective dynamic power management (DPM) policy should minimize power consumption while maintaining performance degradation within an acceptable level. Thus, a joint virtual machine (VM) resource allocation and power management framework is critical to the overall cloud computing system. Moreover, novel solution framework is necessary to address the even higher dimensions in state and action spaces. In this paper, we propose a novel hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. The proposed hierarchical framework comprises a global tier for VM resource allocation to the servers and a local tier for distributed power management of local servers. The emerging deep reinforcement learning (DRL) technique, which can deal with complicated control problems with large state space, is adopted to solve the global tier problem. Furthermore, an autoencoder and a novel weight sharing structure are adopted to handle the high-dimensional state space and accelerate the convergence speed. On the other hand, the local tier of distributed server power managements comprises an LSTM based workload predictor and a model-free RL based power manager, operating in a distributed manner.

I. INTRODUCTION

Cloud resource allocation and power management must handle large state/action spaces while balancing latency, energy use, and reliability. The paper proposes a hierarchical framework that separates global VM allocation from distributed local server power management.

  • Cloud workloads vary over time, motivating online adaptive VM allocation and management.
  • High-dimensional state and action spaces limit traditional reinforcement learning for complete cloud resource allocation.
  • Power management must reduce consumption while keeping performance degradation acceptable, because high consumption harms reliability and increases cooling cost.
  • The proposed hierarchy uses DRL for global VM allocation and distributed local power management to reduce state/action dimensions and online complexity.The global tier uses event-driven decisions at VM-request arrivals, while the local tier includes workload prediction and power management.
  • 53.97% power and energy consumptions were saved in a 30-server cluster with 95,000 job requests.The framework achieved similar average latency and reported the best trade-off between latency and power/energy consumption in that case.

II. BACKGROUND OF THE AGENT-ENVIRONMENT INTERACTION SYSTEM AND CONTINUOUS-TIME Q-LEARNING

The agent-environment model represents sequential decisions through states, actions, transitions, and rewards. Continuous-time Q-learning for SMDP extends this framework with event-driven updates and discounted rewards over state sojourns.

  • An agent observes the environment state, selects an action, receives a reward, and then continues from the updated state.The reward feedback mechanism supports learning an optimal behavior and policy over time.
  • Q-learning selects actions by maximizing Q(s, a), the expected accumulated discounted reward from a state-action pair.
  • For continuous-time systems, Q(s, a) integrates the reward rate over future time with discount rate β.Discrete-time systems can define the value function similarly.
  • SMDP Q-learning operates online in continuous time and updates values when events occur, reducing overhead from periodic discrete-time updates.
  • The SMDP update uses reward, sojourn time τ_k, learning rate α, and discount rate β at decision epoch t_k.

III. SYSTEM MODEL AND PROBLEM STATEMENT

The system model combines VM allocation with server-level power management in a hierarchical framework. It models server execution, sleep/active transitions, workload prediction, and adaptive distributed power control.

  • System model: The framework considers M physical servers offering D resource types, with servers operating in active or sleep mode.
  • System model: The job broker assigns arriving jobs to servers, which queue jobs and allocate resources using first-come-first-serve processing.
  • Problem statement: Overloaded servers delay jobs, increasing latency beyond job duration and motivating dynamic scheduling and resource allocation.
  • Power management: Power management must balance reduced consumption against acceptable performance degradation while accounting for active, sleep, and transition states.
  • Power management: Effective local power management can simultaneously reduce power or energy consumption and average job latency.
  • Power management: The local tier uses workload prediction and model-free reinforcement learning to determine server power-management actions online and in a distributed manner.

IV. OVERVIEW OF DEEP REINFORCEMENT LEARNING

The paper presents deep reinforcement learning as a scalable extension of RL for high-dimensional cloud control. Its workflow combines offline value-function approximation with online deep Q-learning and action selection.

  • Offline phase: The DRL framework first collects state-transition and Q-value data, stores experience, and pre-trains a deep neural network.
  • Online phase: During online control, the agent estimates Q(sk, a) for available actions and uses an ϵ-greedy policy to select actions.
  • Design constraint: DRL supports large or continuous state spaces, but its decision process still requires a relatively low-dimensional enumerable action space.
  • Online phase: Observed transitions and rewards are stored in experience memory, and the network parameters are updated using new Q-value estimates.
  • Motivation: Traditional reinforcement learning becomes impractical when the cloud allocation problem has too many state-action pairs.

V. THE GLOBAL TIER OF THE HIERARCHICAL FRAMEWORK – DRL-BASED CLOUD RESOURCE ALLOCATION

The global tier applies DRL to VM resource allocation under high-dimensional state spaces. Event-driven decisions reduce each action to selecting the target server for an arriving request.

  • Global-tier design: The global tier uses DRL to handle the high-dimensional state space of VM resource allocation.
  • Global-tier design: Each decision epoch coincides with a new VM or job arrival, making the action the target server for allocation.
  • Global-tier design: The event-driven formulation keeps available actions enumerable and uses continuous-time Q-learning for semi-Markov decision processes.

A. DRL-based Global Tier of Resource Allocation

The global tier uses event-driven DRL to allocate each incoming VM to a server while optimizing power, latency, and reliability. Autoencoders and shared-weight Sub-Q networks reduce high-dimensional representations and support scalable Q-value estimation.

  • Global-tier formulation: The DRL agent controls the job broker, while the server cluster acts as the environment and decisions occur when VM requests arrive.The action is the index of the server selected for VM allocation.
  • Global-tier formulation: The state combines server-cluster and job information, including server-group states, resource utilizations, job requirements, and estimated duration.The resulting state space is high-dimensional.
  • Online operation: The event-driven continuous-time framework reduces the action space to the number of servers, and online inference evaluates candidate actions at each VM arrival.The DNN is updated using SMDP Q-learning during online operation.
  • Global-tier formulation: The global reward penalizes total power, pending VMs, and reliability-objective values, linking allocation decisions to energy, latency, and reliability.Pending VMs are proportional to average job latency under Little’s Theorem.
  • DNN construction: An autoencoder extracts lower-dimensional server-group representations, while Sub-Q networks estimate Q-values for allocating VMs to target server groups.The design uses representation learning to address the large state and action spaces.
  • DNN construction: Weight sharing across all autoencoders and Sub-Q networks lets training samples be reused and reduces parameters and training time.The shared structure is intended to improve scalability.

B. Convergence and Computational Complexity Analysis of the Global Tier

The global-tier convergence claim depends on stationary, memoryless SMDP dynamics and sufficiently accurate DNN Q-value estimates. Its online computation scales with the number of server actions at each VM arrival.

  • Convergence: Convergence to the optimal policy requires a stationary, memoryless SMDP environment and a sufficiently accurate DNN.The claim also relies on sufficiently small learning rates through the underlying Q-learning result.
  • Computational complexity: Online computational complexity is proportional to the number of actions, which equals the number of servers at each VM arrival.The paper characterizes this complexity as insignificant for cloud computing systems.

VI. THE LOCAL TIER OF THE HIERARCHICAL FRAMEWORK – RL-BASED POWER MANAGEMENT FOR SERVERS

The local tier manages server power by turning local servers on or off while targeting lower power consumption and average job latency. It combines LSTM workload prediction with model-free continuous-time Q-learning for distributed control.

  • Local power management: The local power-management tier controls turning local servers ON or OFF to reduce power consumption and average job latency.Power management is performed at individual local servers.
  • Local power management: The tier combines an LSTM workload predictor with a model-free, continuous-time Q-learning power manager for SMDPs.The components operate as part of distributed local-server power management.

A. Workload Predictor Using the Long Short-Term Memory (LSTM) Network

The local power manager uses predicted future workload characteristics and current server conditions to make adaptive timeout decisions. An LSTM predicts inter-arrival times, while the RL manager balances power consumption against queued jobs.

  • Workload prediction: The LSTM workload predictor supplies partial observations of future workload characteristics, specifically estimated job inter-arrival times, to define RL states.These workload characteristics result from global-tier VM allocation.
  • Workload prediction: The LSTM predicts the next job inter-arrival time from past time-series observations and is designed to capture long-term dependencies.The paper motivates LSTM over linear prediction because a very long inter-arrival time can disrupt subsequent predictions.
  • Distributed power management: The local power manager uses continuous-time Q-learning for SMDPs and operates in a distributed manner to reduce decision frequency and overhead.Its decisions are based on workload-prediction results.
  • Distributed power management: The manager observes machine power state and estimated next-job inter-arrival time, with decision epochs triggered by idle, arriving-job, or sleeping-server conditions.At idle with no queued job, timeout values form the action set.
  • Distributed power management: The reward rate is the negative of a weighted combination of server power consumption and buffered jobs.The queue term represents latency because average buffered jobs are proportional to average job latency under Little’s Law.
  • Distributed power management: At each decision epoch, the manager selects a timeout through epsilon-greedy Q-learning, then updates the value function after observing the next state and reward rate.The chosen timeout determines whether the server remains active or enters sleep until another job arrives.

VII. EXPERIMENTAL RESULTS

The experiments compare the proposed framework with DRL-based resource allocation and round-robin VM allocation using power consumption and job latency, then examine their trade-off.

  • Experiments compare the proposed framework against DRL-based resource allocation and round-robin VM allocation using power consumption and job latency.

A. Simulation Setups

The evaluation uses homogeneous server clusters, real Google workload traces, offline global-tier training, and five one-week simulations compared with two baselines.

  • Simulation Assumptions: The experiments assume homogeneous clusters with 30 or 40 machines, while noting applicability to larger server counts.Each server is modeled with 145W peak and 87W idle power, with 30-second on and off transitions.
  • Workload Traces: Real Google cluster-usage traces provide job arrival times, durations, and normalized CPU, memory, and disk requests.The traces cover a month-long period in May 2011, with job durations from 1 minute to 2 hours.
  • Training Configuration: The global tier is trained offline with Google traces, five M-machine profiles, epsilon-greedy transitions, and an autoencoder-based DNN.
  • Evaluation Procedure: Five one-week job traces are evaluated against DRL-based resource allocation and round-robin policies using accumulated power consumption and job latency.
  • Results: For 30 machines and 95,000 jobs, the hierarchical framework saves 53.97% power and energy versus round robin and 16.12% versus DRL-based allocation.It also reduces latency by 16.67% compared with DRL-based resource allocation.
  • Results: For 40 machines and 95,000 jobs, the hierarchical framework uses 59.99% less power and energy than round robin and 17.89% less than DRL-based allocation.The passage also reports a 13.32% latency reduction compared with DRL-based allocation, but the comparison is truncated.
  • Results: The 30- and 40-machine comparisons examine latency and energy trends for the hierarchical and DRL-based frameworks.

C. Trade-off of Power/Energy Consumption and Average Latency

The trade-off analysis compares average latency and energy usage across timeout baselines and finds that the proposed framework offers the best supported trade-off.

  • The proposed framework achieves the smallest area against the power and latency axes, denoting the best trade-off among the fixed-timeout baselines.
  • 14.37% maximum average latency saving at equal energy and 16.13% maximum power/energy saving at equal average latency are reported versus the 60-second baseline.
  • 16.16% maximum average latency saving at equal energy and 16.20% maximum average power/energy saving at equal latency are reported versus the 90-second baseline.
  • The conclusion states that the proposed framework achieves the best trade-off between latency and power or energy usage.
Loading 1703.04221v2…