PaperlayerView exact revision on arXiv

Source-linked AI summary

Carbon-Aware Computing for Datacenters

Ana Radovanovic, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, Saurav Talukdar, Eric Mullen, Kendal Smith, MariEllen Cottman, Walfredo Cirne

arXiv:2106.11750v1cs.DCeess.SY

TL;DR

Growing datacenter electricity use matters because grid carbon intensity varies by time and location, while much compute can tolerate delay. The paper presents Google’s Carbon-Intelligent Compute Management system, which forecasts demand and carbon intensity and uses risk-aware optimization to create hourly VCC limits for flexible workloads. Operational measurements show greener-time shifting and a 1-2% power-consumption drop during the highest-carbon-intensity periods.

  • Problem

    Growing datacenter electricity use requires methods that reduce environmental and economic impacts while accounting for substantial time- and location-dependent variation in electricity carbon intensity.

  • Method

    The system combines next-day carbon-intensity forecasts, cluster-level demand forecasts, power models, and risk-aware optimization to generate hourly VCC limits for flexible workloads while preserving daily computation.

  • Results

    1-2% power-consumption drop at times with the highest carbon intensity was measured across Google datacenter clusters.

  • Takeaways & Limitations

    VCCs shift flexible workload execution toward greener hours while reducing electricity-based carbon footprint and supporting resource and power efficiency.

Abstract

from arXiv · show

The amount of CO$_2$ emitted per kilowatt-hour on an electricity grid varies by time of day and substantially varies by location due to the types of generation. Networked collections of warehouse scale computers, sometimes called Hyperscale Computing, emit more carbon than needed if operated without regard to these variations in carbon intensity. This paper introduces Google's system for Carbon-Intelligent Compute Management, which actively minimizes electricity-based carbon footprint and power infrastructure costs by delaying temporally flexible workloads. The core component of the system is a suite of analytical pipelines used to gather the next day's carbon intensity forecasts, train day-ahead demand prediction models, and use risk-aware optimization to generate the next day's carbon-aware Virtual Capacity Curves (VCCs) for all datacenter clusters across Google's fleet. VCCs impose hourly limits on resources available to temporally flexible workloads while preserving overall daily capacity, enabling all such workloads to complete within a day. Data from operation shows that VCCs effectively limit hourly capacity when the grid's energy supply mix is carbon intensive and delay the execution of temporally flexible workloads to "greener" times.

I. INTRODUCTION

Google’s Carbon-Intelligent Computing System shifts temporally flexible datacenter workloads toward lower-carbon periods using day-ahead forecasts, risk-aware optimization, and hourly Virtual Capacity Curves. The global system preserves daily flexible workload capacity while targeting reduced carbon emissions, operating costs, and peak resource usage.

  • Motivation: Datacenter electricity demand is growing rapidly, increasing the need for methods that limit environmental, economic, and performance impacts.Datacenter workloads and compute instances increased more than sixfold between 2010 and 2018, while datacenters accounted for approximately 1% of global electricity usage.
  • Motivation: CICS exploits temporal workload flexibility because electricity-related carbon intensity varies substantially across time and location.Flexible workloads can tolerate delays when their work is completed within 24 hours; examples include data compaction, machine learning, simulation, and data-processing pipelines.
  • System approach: The system generates cluster-level VCCs that impose hourly resource limits while preserving the total daily computation of flexible workloads.VCCs are computed from demand forecasts, uncertainty, carbon-intensity forecasts, business and environmental targets, performance expectations, and datacenter energy-provider limits.
  • System approach: At hours expected to have high local-grid carbon intensity, smaller VCC values reduce compute and power usage by delaying flexible tasks to later hours.The VCCs are pushed to datacenter clusters before the next day and directly affect real-time admission of flexible workloads.
  • Design contributions: CICS combines global carbon-aware time shifting with risk-aware optimization, generic uncertainty modeling, extensibility, reliability controls, modularity, and scheduler independence.The optimization incorporates fleetwide cluster-level forecasts, power models, workload and infrastructure expectations, and datacenter power-contract information.

A. Related research

Prior work includes theoretical, prototype, simulation, and real-time approaches to carbon- and cost-aware datacenter management, but Google’s system emphasizes day-ahead, cluster-specific forecasting and power modeling at fleet scale. Its distinctive design incorporates operational performance risk and separates capacity constraints from real-time scheduling.

  • Prior approaches: Earlier research studied flexible datacenter consumption through theoretical models, small prototypes, simulations, and optimization of carbon, cost, or grid-stability objectives.Much of this work focused on self-managed datacenters and stylized models of flexible demand.
  • Prior approaches: Carbon- and cost-aware management has used workload shifting across locations, VM migration or placement, real-time request rebalancing, and delayed execution.Several approaches achieve benefits by powering down redundant machines or using real-time marginal carbon intensity.
  • Prior approaches: Temporal shifting studies generally use optimization frameworks with stylized job-level demand models and deadline constraints to trade environmental, cost, and performance objectives.These studies have mostly been theoretical, with numerical demonstrations rather than the operational fleet-scale system described here.
  • Distinctive approach: Google’s approach differs by using day-ahead capacity allocation while leaving real-time workload management to constraints supplied to the scheduler.This separates the capacity-allocation mechanism from time-based job-scheduling decisions.
  • Distinctive approach: Cluster-specific resource-demand forecasts and power models capture diversity in workloads and hardware configurations at Google scale instead of relying on stylized uncertainty models.The methodology also explicitly incorporates application and infrastructure performance risk into forecasting, power translation, and optimization.

B. Google’s real-time resource management and its reliability principles

Google’s cluster operating system manages jobs, resources, and service tiers, while CICS shapes only delay-tolerant workloads through day-ahead VCC limits. The mechanism shifts flexible usage away from high-carbon periods and incorporates safeguards intended to preserve inflexible workload performance and reliability.

  • Real-time resource management: Borg manages cluster jobs by allocating tasks and requested CPU, RAM, and disk resources across feasible machines.Jobs include indefinitely running servers and batch processes, and may contain many tasks for reliability or scale.
  • Workload flexibility: Google divides workloads into temporally inflexible higher-tier jobs and flexible lower-tier batch jobs that tolerate delays.Flexible workload may be delayed as long as its daily computation is preserved; scheduling uses estimated upper bounds across resource dimensions.
  • Load-shaping mechanism: A VCC artificially limits hourly cluster compute and power usage, and Borg uses those values to determine real-time CPU availability for incoming flexible jobs.Reducing capacity during high-carbon or high-cost periods delays flexible work rather than changing the treatment of inflexible workloads.
  • Load-shaping mechanism: In the illustrated cluster, flexible usage moves from midday, when carbon intensity is highest, toward evenings and early mornings, while daily peak CPU and power consumption decrease.The passage describes the mechanism shown in Figure 3 rather than a fleetwide measured percentage.
  • Reliability principles: The system relies on advance VCC knowledge, safety checks, gradual rollout, monitoring, feedback, and rollback-oriented reliability practices.Its design principles also require fairness for delayed flexible jobs and restrict impact to flexible workload.

III. LOAD SHAPING ANALYTICS

CICS combines forecasting, power modeling, carbon-intensity retrieval, risk-aware optimization, and SLO monitoring to compute next-day cluster-level VCCs. These curves shape flexible compute usage while accounting for uncertainty, infrastructure constraints, and workload requirements.

  • III. LOAD SHAPING ANALYTICS: CICS uses analytical pipelines to forecast next-day demand, model CPU-to-power relationships, retrieve carbon-intensity forecasts, optimize VCCs, and detect SLO violations.The pipelines also trigger feedback when flexible-demand violations persist.
  • III. LOAD SHAPING ANALYTICS: Power models map CPU usage to power consumption for each power domain and are retrained and evaluated daily across Google’s fleet.The models support carbon-aware load shaping across heterogeneous power architectures and machine platforms.
  • III. LOAD SHAPING ANALYTICS: Load forecasting predicts cluster-level flexible and inflexible demand, preserving flexible load’s total daily compute while modeling reservations and forecasting error.Forecast uncertainty is embedded in risk-aware optimization to preserve infrastructure and application SLOs.
  • III. LOAD SHAPING ANALYTICS: The daily optimization co-optimizes expected fleetwide carbon footprint and appropriately scaled power peaks subject to SLO, contractual, and resource-capacity constraints.Its output is a cluster-level VCC specifying optimal capacity for each hour of the next day.
  • III. LOAD SHAPING ANALYTICS: Daily pipelines generate next-day predictions, after which a central optimizer computes 24-hour VCCs for all clusters and distributes them fleetwide.All usage data and VCC horizons are tracked in Pacific Standard Time.

A. Power modeling

The power-modeling approach represents cluster power sensitivity using power-domain CPU usage and locally approximates dynamic power with piecewise linear models. Evaluation reports low daily prediction error for most power domains, while scheduler behavior motivates aggregating power-domain usage.

  • A. Power modeling: A piecewise linear model accurately captures the relationship between CPU usage and dynamic power consumption for a given power domain.The model locally approximates how changes in power-domain CPU usage affect power.
  • A. Power modeling: MAPE is less than 5% for more than 95% of power domains under a rigorous evaluation using 5-minute usage data.The reported evaluation concerns the proposed power-consumption model.
  • A. Power modeling: CPU usage fractions across power domains vary insignificantly over time within a cluster because the scheduler assigns tasks to randomly selected feasible machines.Fleetwide variation is estimated at 1%, while values close to 25% occur only for new clusters with very little workload.
  • A. Power modeling: Power-domain power consumption is modeled from CPU usage, with cluster-level sensitivity defined from the power-domain contributions.The notation distinguishes power consumption, model slopes, CPU-usage changes, and stochastic usage processes over time.

B. Day-ahead forecasting

The system forecasts next-day cluster-level compute demand using weekly averages updated with recent deviations, then evaluates prediction errors across Google's fleet. Most clusters achieve low errors for inflexible usage, reservations, and reservation-to-usage ratios, while flexible demand is more variable.

  • Forecast targets: Next-day forecasts cover hourly inflexible CPU usage, daily flexible compute usage, daily total reservations, and hourly reservations-to-usage ratios.These load components and their relationships are illustrated in Figure 6.
  • Forecasting method: Forecasts use a two-step approach that predicts weekly averages and augments them using the previous day’s deviations.EWMA models forecast weekly means and time factors, while a linear model captures intra-week deviations before final predictions are passed to the optimizer.
  • Forecasting method: A linear model predicts reservation-to-usage ratios above 1 from log CPU usage, enabling conversion of optimal usage profiles into VCCs.The ratio generally decreases as cluster CPU usage increases.
  • Forecast evaluation: Median APEs for inflexible usage, total reservations, and reservation-to-usage ratios are below 10% for more than 90% of clusters.This fleetwide evaluation uses daily model training and three months of day-ahead prediction errors.
  • Forecast evaluation: Daily flexible compute forecasts have larger errors because flexible demand is more variable, with rare errors above 50% during small-load periods or transient infrastructure upgrades.Such cases can make shaping inactive or ineffective for affected clusters and days.

2) Service Level Objective awareness

The framework preserves each cluster’s daily flexible-workload compute requirement while accounting for prediction uncertainty. It uses risk-aware capacity inflation and feedback mechanisms to manage violations caused by forecast errors or changing demand.

  • SLO definition: The primary workload SLO requires preserving cluster-level daily compute usage when flexible workloads are shifted in time.The target limits violations to roughly one day per month, equivalent to an upper violation probability of about 0.03 over 100 days.
  • Violation conditions: SLO violations arise when reserved CPUs exceed actual needs or when flexible daily usage is constrained by unexpected flexible or inflexible demand growth.Both over-reservation and demand growth can undermine the daily flexible-workload requirement.
  • Risk handling: The framework computes the 97th percentile of total daily capacity requirements from the previous 90 days of relative day-ahead prediction errors.This quantile supports risk-aware capacity planning for all-load reservations.
  • Feedback control: If demand approaches the VCC limit for two consecutive days, feedback is triggered and shaping may be paused for about a week while forecasts adapt.This response addresses unpredicted demand growth.
  • Risk handling: The system inflates forecasted flexible usage with α(c)(d) so the optimal plan satisfies CPU-capacity constraints under uncertainty.The inflation assigns extra capacity to the daily flexible-usage requirement using predicted reservation-to-usage ratios.

3) Carbon intensity forecasting

The carbon-intensity pipeline supplies near-term forecasts for grid electricity carbon intensity at each datacenter location. These forecasts provide the temporal and spatial carbon signal used in next-day capacity planning.

  • Carbon-intensity forecasts: The optimizer retrieves 48-hour average carbon-intensity forecasts from Tomorrow, which estimates grid-consumption intensity using demand, generation, and imports.The optimizer uses each location’s forecast for every hour of the next day.

C. Optimization framework

The optimization framework computes hourly cluster capacity limits by combining demand forecasts, carbon-intensity forecasts, costs, and operational constraints. It reshapes flexible usage while preserving daily requirements and respecting infrastructure and application SLOs.

  • Optimization process: The system computes next-day hourly reservation capacities for every cluster using risk-, cost-, and carbon-aware optimization.Capacity values are computed once daily and gradually propagated before the next day.
  • Optimization objective: The objective minimizes a weighted sum of expected carbon footprint and fleetwide daily power peaks.The weights represent carbon costs in $ / kg CO2e and infrastructure costs in $ / MW / day.
  • Optimization objective: Hourly flexible-usage deviations and cluster peak-power bounds are jointly optimized to reduce carbon footprint and infrastructure peaks.The resulting usage shape is translated into the corresponding VCC.
  • Constraints: The optimization preserves daily flexible usage and honors application, infrastructure, and contractual power-demand constraints.Constraints include power capping, campus-level energy contracts, and cluster machine capacity.
  • Constraints: Risk-aware constraints use historical prediction-error quantiles to bound cluster usage and reduce the probability of exceeding power thresholds.The framework uses a specified γ parameter and the corresponding historical (1−γ)th quantile.
  • Constraints: Additional constraints, such as bounds on intraday usage drops or hour-to-hour VCC changes, can be added as hard or soft penalties.The framework supports hinge or quadratic penalty formulations for soft constraints.

D. Discussion

The discussion frames carbon-aware load shaping as a risk-aware, scalable optimization problem that balances carbon reduction, power efficiency, reliability, and operational constraints. Its day-ahead design relies on predictable aggregate demand rather than uncertain job-level behavior and remains decoupled from real-time scheduling.

  • Carbon and efficiency trade-offs: Using only carbon footprint can increase required machine capacity, whereas combining carbon and peak-power costs reduces expected carbon footprint and future infrastructure demand.The optimization explicitly represents the trade-off between carbon reduction and efficiency.
  • Open questions: The effectiveness of day-ahead average carbon-intensity shaping depends on the chosen grid-level signal, which requires further research and data-driven investigation.Average carbon intensities are described as fairly predictable, but the most effective signal remains unresolved.
  • Operational design: Day-ahead VCC computation is decoupled from lightweight real-time scheduling operations.The separation allows real-time scheduling to remain operationally lightweight.
  • Forecasting and risk: VCCs use predictable aggregate flexible-load demand, carbon forecasts, and capacity limits instead of uncertain flexible-job arrivals and resource usage.Cluster-level daily usage is more predictable within the day-ahead horizon than individual job behavior.
  • Forecasting and risk: Risk awareness explicitly conserves each cluster’s daily flexible usage while tracking violations and supporting contractual power limits.The formulation can incorporate additional hardware and workload requirements.
  • Scalability: Aggregate cluster-level modeling makes centralized fleetwide optimization more scalable than diverse job-level models while preserving reliability criteria.The approach also supports future spatial shifting and portfolio-level optimization extensions.

IV. DEMONSTRATION AND IMPACT

Operational evaluations show that carbon-aware shaping reduces electricity use during high-carbon hours, but its impact varies with forecast uncertainty, flexible-load availability, and operating regime. More aggressive shaping can reduce daily flexible usage and violate conservation, while carbon-impact measurement itself remains an open methodological issue.

  • Mechanism: The mechanism constrains load to shift flexible compute away from high-carbon hours while aiming to complete flexible jobs within 24 hours.VCCs use carbon-intensity forecasts and demand uncertainty to shape the next day’s profile.
  • Cluster demonstrations: Electricity usage decreased precisely when the grid was most carbon intense in selected cluster examples.The figures compare VCC-constrained reservations with normalized cluster power and carbon intensity.
  • Cluster demonstrations: 18% higher average VCC than average load demand in cluster X reflects forecast uncertainty and still enabled roughly 50% flexible-load reduction and an 8% power decrease during peak-carbon hours.Cluster X sustained the reduction for 6 hours, making shaping more effective than in cluster Y.
  • Cluster demonstrations: 33% higher average VCC than average load demand in cluster Y produced nearly 50% flexible-load reduction but only 3 hours of sustained shaping and roughly 8% lower peak-carbon-hour power.Higher forecast uncertainty reduced the duration and carbon impact relative to cluster X.
  • Cluster demonstrations: Small flexible load and substantial inflexible-load uncertainty prevented meaningful flexible curtailment or significant peak-carbon-hour power change in cluster Z on the selected day.The cluster’s load composition limited the optimization’s ability to shift flexible demand.
  • Operating boundaries: About 10% of clusters on a given day receive a VCC equal to total machine capacity when shaping is blocked by high occupancy or insufficient forecasting and power-model data.In these cases, no shaping constraint is imposed.
  • Aggregate impact: 1-2% average cluster-power reduction occurred during the highest-carbon hours when CICS was active in the controlled experiment.More aggressive regimes slightly decreased total daily flexible compute usage and daily energy consumption.
  • Operating boundaries: More aggressive regimes can violate daily flexible-compute conservation, and spontaneous spatial load shifting may increase or decrease carbon emissions.Future models are intended to characterize spatially flexible demand explicitly.

V. CONCLUDING REMARKS

The paper presents carbon-aware computing as a way to use datacenter load flexibility amid growing electricity demand and emissions. Google’s system applies forecast-driven VCCs globally and demonstrates a 1-2% power reduction during the highest-carbon periods, while future extensions target spatial shifting and grid integration.

  • Motivation: Datacenter electricity demand is growing, and flexible computing loads may support emissions reduction and more robust, resilient, and cost-efficient grid decarbonization.The paper places datacenter load flexibility within the broader growth of electricity demand and climate impact.
  • System contribution: Google’s Carbon-Intelligent Computing System shifts datacenter computing in time using automated adjustments based on current and forecasted grid conditions.Its core mechanism uses cluster-level VCCs to shape resource and power usage over the following day.
  • Observed impact: 1-2% power consumption drop at the highest-carbon times was demonstrated using actual measurements from Google datacenter clusters.The result is reported as an operational measurement across the system’s datacenter clusters.
  • Future scope: Future enhancements are expected to increase benefits through spatial workload shifting and could support datacenter microgrid energy management and grid-level demand response.These extensions build on the system’s power models and VCC formulation.
  • Scope: The framework is aligned with Google’s compute-management systems and workload properties, while other providers’ carbon-aware approaches will necessarily vary.The authors present the results as motivation for diverse approaches to cluster and hyperscale computing management.
Loading 2106.11750v1…