PaperlayerView exact revision on arXiv

Source-linked AI summary

Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning

Yuanlong Li, Yonggang Wen, Kyle Guan, Dacheng Tao

arXiv:1709.05077v4cs.AIeess.SY

TL;DR

Data-center cooling control is difficult because conventional approaches depend on challenging approximate system models. The paper uses offline deep reinforcement learning to learn cooling policies from monitoring data and reports savings in simulation and trace-based evaluation.

  • Problem

    Conventional cooling-control approaches often rely on difficult-to-design approximated mechanical, electrical, and thermal system models.

  • Method

    The paper proposes an end-to-end offline, off-policy cooling control algorithm adapted from actor-critic and DDPG methods, trained on pre-collected data traces.

  • Results

    The method saves about 11% cooling energy in EnergyPlus and around 15% cooling cost on an NSCC trace while maintaining predefined temperature thresholds.

  • Takeaways & Limitations

    The results demonstrate the feasibility and effectiveness of applying an end-to-end neural control algorithm to data-center cooling optimization.

Abstract

from arXiv · show

Cooling system plays a critical role in a modern data center (DC). Developing an optimal control policy for DC cooling system is a challenging task. The prevailing approaches often rely on approximating system models that are built upon the knowledge of mechanical cooling, electrical and thermal management, which is difficult to design and may lead to sub-optimal or unstable performances. In this paper, we propose utilizing the large amount of monitoring data in DC to optimize the control policy. To do so, we cast the cooling control policy design into an energy cost minimization problem with temperature constraints, and tap it into the emerging deep reinforcement learning (DRL) framework. Specifically, we propose an end-to-end cooling control algorithm (CCA) that is based on the actor-critic framework and an off-policy offline version of the deep deterministic policy gradient (DDPG) algorithm. In the proposed CCA, an evaluation network is trained to predict an energy cost counter penalized by the cooling status of the DC room, and a policy network is trained to predict optimized control settings when gave the current load and weather information. The proposed algorithm is evaluated on the EnergyPlus simulation platform and on a real data trace collected from the National Super Computing Centre (NSCC) of Singapore. Our results show that the proposed CCA can achieve about 11% cooling cost saving on the simulation platform compared with a manually configured baseline control algorithm. In the trace-based study, we propose a de-underestimation validation mechanism as we cannot directly test the algorithm on a real DC. Even though with DUE the results are conservative, we can still achieve about 15% cooling energy saving on the NSCC data trace if we set the inlet temperature threshold at 26.6 degree Celsius.

I. INTRODUCTION

The paper frames data-center cooling as an energy-optimization problem whose complexity motivates data-driven, end-to-end deep reinforcement learning. It proposes an offline cooling-control algorithm and evaluates it through simulation and trace-based validation.

  • Motivation: Cooling is a significant source of data-center energy consumption, while selecting optimal set-points across complex cooling components is challenging.The cooling system includes components such as cooling towers, chillers, and ventilation systems.
  • Motivation: Existing cooling optimization commonly uses a two-stage process that builds an approximate system model before solving for control settings.The paper questions whether an end-to-end approach can match or exceed two-stage control performance.
  • Approach: The proposed cooling control algorithm adapts actor-critic and DDPG methods into an off-policy, offline approach trainable from pre-collected traces.A recurrent version is also tested for potentially improved performance with noisy data.
  • Evaluation: The study evaluates five-set-point cooling control in EnergyPlus and on a real trace from Singapore’s National Super Computing Centre.The simulation maintains zone temperature within a predefined range while comparing against standard control algorithms.
  • Results: 11% cooling cost saving is achieved in EnergyPlus compared with the baseline approach.The result is reported for the proposed control algorithm in the simulation study.
  • Results: The DUE validation method produces conservative, lower-risk energy-saving calculations when direct testing on a real data center is unavailable.The method addresses predicted-temperature underestimation and reflects the importance of operational risk management.

II. RELATED WORKS

Prior cooling-control research largely relies on staged optimization and idealized models, while this paper positions DRL as a data-driven alternative evaluated on complex simulation and real traces.

  • A. Recent Progress on DC Cooling Optimization: Many existing cooling studies use two-stage optimization, first constructing thermal-dynamics or airflow models and then optimizing cooling efficiency.Computational fluid dynamics is also used to analyze airflow efficiency.
  • A. Recent Progress on DC Cooling Optimization: Related work also optimizes ice storage or IT-side workload dispatch to reduce cooling-related energy costs.The paper notes that its approach could be combined with IT-side optimization.
  • A. Recent Progress on DC Cooling Optimization: The authors characterize most existing studies as relying on ideal models in simplified situations.They therefore evaluate their DRL solution on both a complicated simulation system and a real data trace.
  • B. Reinforcement Learning: Reinforcement learning learns improved actions from experience interacting with an environment, providing a foundation for neural control agents.The related work surveys adaptive dynamic programming and actor-critic reinforcement-learning applications.
  • B. Reinforcement Learning: Deep reinforcement learning extends reinforcement learning with deep neural-network approximations and has been adapted for continuous control variables.The paper situates its cooling-control method within this development.

III. THE COOLING OPTIMIZATION PROBLEM FORMULATION

The simulation formulation models two independently cooled server zones and represents cooling optimization through state, action, and reward spaces. Five learned outlet-temperature set-points control DX and chiller systems.

  • III. THE COOLING OPTIMIZATION PROBLEM FORMULATION: The EnergyPlus-based simulation captures major cooling dynamics while simplifying the overall system for cooling-control studies.The model is described as adequate for studying the optimization problem despite its simplification.
  • A. Simulation System Model: The simulated data center contains two server zones, z1 and z2, with independently operated cooling facilities.DX cooling serves z1, while chiller-plant cooling serves z2.
  • A. Simulation System Model: Figure 2 identifies the main DX and chiller components and five outlet temperatures used as control set-points.The set-points are Tdec, Tiec, Tcw, Tdx, and Tch.
  • 1) Data Center Model – State Space and Reward:: The zones differ in size and load density, with IT equipment as the dominant heat source and workload varying through a load factor.Each zone is simplified as a single point-heat source rather than modeled with finer-grained CFD analysis.
  • 1) Data Center Model – State Space and Reward:: The reinforcement-learning state uses workload level and ambient temperature because both affect cooling load.The reward uses PUE and each zone’s IT-equipment outlet temperature.
  • 1) Data Center Model – State Space and Reward:: The cooling systems use different arrangements: DX cools airflow through coils, whereas the chiller system uses cool water in its cooling process.Both systems receive cool water from a cooling tower.
  • 1) Data Center Model – State Space and Reward:: The DefaultE+ controller uses five fixed-design set-points derived from system-dynamics knowledge, while the proposed method learns the same variables from pre-collected traces.The learning algorithm does not use the variables’ physical meanings or relationships.

B. Problem Statement

The problem is to choose five cooling set-points from ambient temperature and load information, minimizing PUE while penalizing overheating. The paper then applies an offline, end-to-end DRL controller to this formulation.

  • B. Problem Statement: Given ambient air temperature Tamb and load factor Hite, the controller determines five set-points to minimize the cooling objective.The five control variables are the cooling-system set-points defined in the simulation model.
  • B. Problem Statement: The objective balances minimizing PUE against penalties for overheating in both server zones.The overheating penalty uses λ·ln(1 + exp(Tzi − φ)) for each zone.
  • B. Problem Statement: The cooling control algorithm is an end-to-end method adapted from DDPG that combines DQN, DPG, and actor-critic techniques.The algorithm description covers its complete flow and neural-network design.

A. Overview of Q-learning and Policy Gradient

The paper frames cooling control as learning an optimal policy from offline state, action, and reward traces, using an actor-critic approach adapted to continuous control. Its offline design avoids risky online learning and trains Q and μ networks to evaluate and optimize control actions.

  • Cooling control is formulated as learning an optimal policy from discrete sequences of states, actions, and rewards.
  • DDPG combines value-based and policy-based learning for continuous state and action variables, avoiding costly discretization.
  • Offline pre-training is used because erroneous online control settings could risk data-center cooling operation.
  • The proposed algorithm adapts canonical DDPG into an off-policy cooling controller trained from pre-collected traces.
  • The critic Q network estimates action cost, while the policy μ network selects control actions from the current state and recent history.
  • The Q architecture outputs predicted energy and temperature data in its second-last layer for computing the loss in (1).

D. Neural Network Design

CCA trains paired Q and μ neural networks from normalized offline traces, using validation and periodic reinitialization to select stable parameter settings. The Q network predicts energy and temperature components before computing the penalized objective.

  • Q uses three hidden layers and predicts negative reward through intermediate energy and temperature outputs used in the objective.
  • Q and μ networks are initialized separately and trained across multiple epochs using randomly batched data.
  • Offline traces are organized into Q inputs, μ inputs, and energy-temperature readings, then divided into training and validation sets.
  • Validation errors determine the best saved Q and μ parameters, which are returned after training.
  • All data are normalized to (-1, 1) for tanh activations and de-normalized when real energy and temperature values are required.

V. SIMULATION BASED NUMERICAL EVALUATION AND ANALYSIS

The simulation evaluation compares CCA against EnergyPlus’s default controller and a two-stage optimization algorithm. These baselines differ in whether they use fixed temperature targets or an explicit optimization stage.

  • EnergyPlus simulation evaluates CCA against the default EnergyPlus controller and a general two-stage cooling optimization algorithm.
  • DefaultE+ computes five control set-points from a target zone temperature using an underlying system model.
  • The two-stage baseline first models cooling efficiency and then solves for control settings using optimization.

A. Simulation Configurations

The simulation uses EnergyPlus because real-data-center testing is risky and slow, while the platform supports recognized cooling models and user-defined controls. A one-year Singapore workload trace supplies training data, and algorithms are evaluated on simulated test states.

  • A. Simulation Configurations: EnergyPlus is used because direct real-data-center testing risks erroneous settings and requires long running times.
  • A. Simulation Configurations: EnergyPlus is presented as a recognized cooling-energy simulation platform with flexibility for user-defined algorithms, actions, and schedules.
  • A. Simulation Configurations: The simulation uses Singapore weather, an EnergyPlus data-center model, and a Wikimedia CPU-loading trace over one year.
  • A. Simulation Configurations: A randomized, smoothed one-year control trace trains CCA, with the final 45% reserved for testing.
  • B. Algorithm Configurations: CCA uses five set-points and tuned neural-network settings, including λ = 0.01, while the two-stage baseline uses differential evolution for test-state optimization.
  • A. Simulation Configurations: Each algorithm’s generated control settings are applied in EnergyPlus at test states, and resulting state changes and rewards are recorded.

C. Comparing CCA to Baseline Algorithms

CCA achieved lower PUE than DefaultE+ and TS while keeping outlet temperatures under or near the threshold. TS showed unstable performance, whereas CCA directly produced offline-tunable control settings.

  • Simulation comparison: 11% cooling power saving: CCA with τ = 1 reduced PUE from 1.37 to 1.33 while maintaining both zones under or near threshold 29.The result indicates that the actor network attained optimal or close-to-optimal control settings.
  • Simulation comparison: TS with the general optimization algorithm showed unstable performance and would require specialized optimization to improve.CCA instead directly outputs settings through a pretrained policy network that can be tuned and tested offline.
  • Simulation comparison: CCA’s PUE curve was lower than DefaultE+ and TS while its temperature curves remained higher but satisfying in both zones.The comparison used PUE and maximum outlet temperatures during the test period.
  • Simulation comparison: The recurrent CCA with τ = 3 produced unstable results in the noise-free simulated case.Recurrent decision making is expected to be more useful when data are noisy.
  • Simulation comparison: The test distributions report PUE and maximum outlet temperatures Tz1 and Tz2 across 10 independent tests using medians and means.Orange lines denote medians and green triangles denote means.

D. Neural Network Design Study

The study compares alternative neural-network architectures and examines hyper-parameter choices for CCA. Architectures perform similarly, while penalty tuning controls the energy–temperature trade-off and depends on the problem.

  • Neural network design: LstmNet processes recent state and action histories with an LSTM layer before feeding them into the Q network.It was evaluated with τ = 3, whereas the other designs were tested with τ = 1.
  • Neural network design: TargetNet, ReluNet, and LstmNet achieved results very similar to the original CCA design.TargetNet was nearly identical; ReluNet produced slightly higher temperatures with lower PUE.
  • Hyper-parameter setting: Adadelta is used as the training optimizer, avoiding the need to set the learning rate manually.Smaller initialization ranges tend to produce more stable training.
  • Hyper-parameter setting: The CCA penalty factor λ must be manually tuned to obtain a satisfactory energy–temperature trade-off.The study varies λ from 0.0 to 0.04, and an appropriate setting differs across problems.

VI. TESTS ON REAL DATA TRACE FROM NSCC

The NSCC trace study applies CCA to airflow control using real measurements and validates it conservatively because direct real-DC testing is unavailable. CCA achieved substantial estimated energy savings under specified temperature thresholds.

  • NSCC trace setup: The NSCC experiments optimize cooling energy while satisfying rack intake-temperature requirements using collected operational measurements.Additional warm-water and rear-door cooling measurements improved Q-network approximation accuracy.
  • NSCC trace setup: The NSCC system contains 26 racks cooled by three PCUs, and CCA optimizes total PCU supply airflow while maintaining average rack intake temperature.The cold-air path runs from the PCUs through the raised floor and cold aisle, across the racks, and back to the PCUs.
  • NSCC trace setup: The Q network’s normalized MAE was below 0.1 degrees, with τ = 4 giving the best temperature estimation on noisy real data.The trace was sampled every three minutes from March 1–15, 2017, using the first 85% for training and the last 15% for testing.
  • Validation: DUE changes validation to consider underestimation error, reducing underestimation cases and producing more conservative, lower-risk savings calculations.Without direct real-DC testing, this validation strategy addresses over-optimistic energy-saving estimates caused by temperature underestimation.
  • Energy-saving results: 15% energy saving was achieved with DUE when the temperature threshold exceeded 26.6 degrees.At a maximum temperature target of 26.4, savings were about 10%; below 26.2, more cooling power was required than the actual setting.

VII. CONCLUSION

The paper proposes and verifies an end-to-end deep reinforcement learning approach, CCA, for optimizing data-center cooling control from observed historical data. CCA maintains temperature constraints while reducing cooling energy in both simulated and trace-based evaluations.

  • CCA directly learns a cooling policy from observed historical data and outputs optimized control settings for a given system state.
  • 11% cooling energy savings were achieved in EnergyPlus versus a baseline with manually designed control settings while maintaining the predefined temperature threshold.
  • Around 15% cooling cost reduction was predicted on a real-data trace while maintaining rack intake temperature below a predefined threshold.
Loading 1709.05077v4…