Source-linked AI summary
Measurement of Generative AI Workload Power Profiles for Whole-Facility Data Center Infrastructure Planning
Roberto Vercellino, Jared Willard, Gustavo Campos, Weslley da Silva Pereira, Olivia Hull, Matthew Selensky, Juliane Mueller
TL;DR
The paper addresses limited, heterogeneous, and often proprietary power data that hinder whole-facility AI data-center planning. It measures standardized GenAI workloads at high temporal resolution and scales those profiles through a bottom-up facility model. The resulting profiles capture workload- and user-driven temporal variation, while showing distinct relationships among utilization, facility consumption, and operating conditions.
Problem
Existing AI data-center power evidence is fragmented and often proprietary, limiting whole-facility energy estimation and infrastructure planning.
Method
The study combines 0.1-second NVIDIA H100 workload measurements with standardized benchmarks and a bottom-up, discrete-event whole-facility energy model.
Results
Training and inference jobs showed pronounced power transients; training power scaled linearly with node count under tested configurations, while online inference power saturated at modest request rates.
Takeaways & Limitations
The open dataset and scaling methodology provide inputs for facility-level demand profiles used in infrastructure planning and load forecasting.
Takeaways & Limitations
Measurements covered one hardware setup and limited workload configurations, while simulations omitted auxiliary loads and relied partly on externally adapted utilization and arrival-rate distributions.
Abstract
from arXiv · showhide
The rapid growth of generative artificial intelligence (AI) has introduced unprecedented computational demands, driving significant increases in the energy footprint of data centers. However, existing power consumption data is largely proprietary and reported at varying resolutions, creating challenges for estimating whole-facility energy use and planning infrastructure. In this work, we present a methodology that bridges this gap by linking high-resolution workload power measurements to whole-facility energy demand. Using NLR's high-performance computing data center equipped with NVIDIA H100 GPUs, we measure power consumption of AI workloads at 0.1-second resolution for AI training, fine-tuning and inference jobs. Workloads are characterized using MLCommons benchmarks for model training and fine-tuning, and vLLM benchmarks for inference, enabling reproducible and standardized workload profiling. The dataset of power consumption profiles is made publicly available. These power profiles are then scaled to the whole-facility-level using a bottom-up, event-driven, data center energy model. The resulting whole-facility energy profiles capture realistic temporal fluctuations driven by AI workloads and user-behavior, and can be used to inform infrastructure planning for grid connection, on-site energy generation, and distributed microgrids.
1. Introduction
AI data-center growth creates planning challenges, but available workload and facility power evidence remains fragmented, proprietary, or insufficiently reproducible. This paper addresses the gap by combining standardized high-resolution measurements with whole-facility modeling.
- 176 TWh in 2023 represented 4.4% of U.S. electricity use, following growth from approximately 76 TWh in 2018.
- AI data centers create planning, operational, and grid-stability challenges across infrastructure, siting, capacity, and real-time load management.
- Existing studies vary in rigor and transparency, often rely on proprietary data, and do not consistently provide facility-level, geographically resolved power profiles.
- Prior measurements characterize GPU, server, or rack power and reveal workload phases, but provide limited integration with reproducible whole-facility planning models.
- The paper measures GenAI training, fine-tuning, and inference power at 0.1 s resolution using standardized MLCommons and vLLM benchmarks.
2.1. Overview
The data-center hierarchy connects compute components and racks to rooms and centralized facility infrastructure. This organization provides the physical basis for scaling node-level measurements to facility-level demand.
- A GPU node contains CPUs, NVIDIA H100 GPUs, system memory, and associated hardware within the data-center hierarchy.
- Chassis house nodes and provide mechanical support, local power distribution, and cooling interfaces before nodes are organized into racks.
- Racks are grouped into temperature-controlled rooms that also contain power, distribution, cooling, and other server equipment.
- Facility-level aggregation combines IT loads across rooms with centralized electrical and thermal systems, including transformers, UPS equipment, generators, chillers, and pumps.
2.2. Measuring Power Profiles of Individual AI Jobs
The study combines fine-grained software-based CPU/GPU power monitoring with standardized training, fine-tuning, and inference benchmarks. These workload configurations expose how model, scale, batching, and request behavior shape measured power profiles.
- WattAMeter records CPU and GPU power as time series across multi-node HPC jobs and integrates with schedulers such as Slurm.
- WattAMeter uses NVML and RAPL readers whose accuracy depends on hardware and measurement method, with NVML documentation reporting ±5% power-draw error.
- MLCommons benchmarks cover AI training and fine-tuning across model types including large language, natural-language-processing, and computer-vision workloads.
- Distributed fine-tuning scaled from 2 to 16 nodes with data parallelism and ZeRO-3, while effective global batch size increased linearly with GPU count.
- The study fixed per-device batch size at 1 to match MLPerf defaults because higher batch sizes caused training-loss instability without broader hyperparameter tuning.
- Inference profiling with vLLM evaluates throughput, latency, memory efficiency, and multi-GPU scaling across token and batching conditions.
2.3. Whole-Facility Simulation - DIPLOEE
DIPLOEE scales workload-level power profiles into whole-facility simulations using workload, architecture, utilization, and user-behavior inputs. It uses job-centered and inference-specific simulation structures to represent scheduling, request handling, node occupancy, and power consumption.
- Model structure: DIPLOEE combines server-level workload power samples with facility utilization distributions to simulate whole-facility data-center operation.The model structure differs for training and fine-tuning jobs versus inference workloads.
- Model inputs: Each workload is defined by type, node count, duration, and a time-resolved power profile, while probability distributions determine which workloads run and when.Facility architecture inputs include node composition, processor and GPU TDP, and total node count.
- Model inputs: DIPLOEE defines utilization as the time-averaged fraction of occupied nodes, independent of CPU or GPU efficiency and power relative to TDP.The utilization measure is based on N(t) divided by Ntotal over the simulation interval.
- Job-centered simulation: Job generation uses bisection to adjust the average daily job count until simulated jobs match the user-defined target utilization.Jobs are distributed across days, hours, workload types, and node counts using user-selected probabilities.
- Job-centered simulation: The job-centered simulator uses discrete-event scheduling to model workload starts, execution, and power consumption on capacity-limited data-center nodes.The generated job list is passed to a SimPy-based simulation that iterates between scheduling and execution.
- Inference simulation: Inference simulation replaces discrete job scheduling with model instances serving user prompts, assigning measured load samples to nodes after matching utilization and request rates.The model assumes minimum latency has priority and distributes requests among inference types and model instances.
- Inference simulation: Inference capacity is limited by per-instance request-rate thresholds and user-defined node allocations; excess requests are tracked as incomplete when capacity is exceeded.The framework assumes request rates are uniformly distributed across model instances and can allocate nodes according to expected request fractions.
3. Results
The workload profiles show model-specific, time-varying power behavior whose aggregate metrics often scale with nodes but can conceal important transient dynamics. Training energy depends on the interaction between power, runtime, evaluation, and batching, while inference power and efficiency depend on prompt load, output length, and request rate.
- Llama-2 fine-tuning: Llama-2 fine-tuning began with an approximately 3-minute ramp-up, followed by high-frequency power variation and intermittent low-activity periods caused by synchronized computation, communication, and data transfers.These transient dynamics became more pronounced as node count increased, reflecting more data-parallel ranks, communication, and synchronization.
- Llama-2 fine-tuning: Average power and power variability for Llama-2 fine-tuning increased linearly with node count, while runtime and total energy also depended on scaling efficiency, epochs, and evaluations.Total energy was calculated by multiplying average power by runtime, equivalent to integrating the power profile.
- Llama-2 fine-tuning: Increasing node count increased Llama-2 training energy under these conditions because average power rose proportionally while the number of epochs also increased.The result highlights a trade-off between training energy and time to solution when runtime changes substantially.
- Stable Diffusion training: Stable Diffusion training produced longer runtimes and larger power ranges than Llama-2, including approximately 12–48 kW at 16 nodes during evaluation.The fluctuations were associated with iterative diffusion sampling, repeated denoising, and intermittent synchronization.
- Stable Diffusion training: Stable Diffusion average power and variability increased linearly with node count, but total energy tended to decrease as faster execution offset the energy from additional nodes.The authors state this energy trend is not generalizable because it depended on the benchmark’s evaluation details.
- Llama-3 inference: For offline Llama-3 inference, execution time increased with output-token length, power rose with input prompts before saturating around 325 prompts, and energy showed a more linear trend with workload size.Average power peaked between 300 and 500 prompts before decreasing slightly, while output-token length affected variability, runtime, and energy proportionally.
- Online inference: Online inference showed early request-processing saturation: tests at 10 and 20 prompts per second differed more than tests at 100 and 1,000 prompts per second.Higher request rates increased throughput but eventually leveled off, while latency and output-token length created performance trade-offs.
- Online inference: Processing a fixed prompt set on fewer model instances can improve energy efficiency by raising per-instance request rates, but it increases latency and degrades user experience.Average power rapidly approached approximately 2.9 kW below 100 prompts per second, while execution time and energy decreased at higher request rates.
4. Whole-Facility Simulation Results
The study simulates year-long operation of representative colocation and inference facilities by scaling measured AI workload profiles into whole-facility demand. Results show utilization- and behavior-driven temporal variability, saturation at high utilization, and distinct service constraints for inference.
- Case studies: Two case studies model a 10 MW colocation facility and a 1 MW online inference facility.The colocation case mixes training and fine-tuning jobs, while the inference case serves Llama-3 70B models.
- Simulation setup: Each facility is simulated for one year at one-minute resolution across 20%, 40%, 60%, and 80% average target utilization.The simulations use four NVIDIA H100 GPUs and two AMD EPYC CPUs per node, with 3.520 kW rated node power and 420 W idle power.
- Colocation data center: Colocation load increased after 8 AM and peaked after 4 PM, with diurnal variability most visible at 40% and 60% utilization.At 20% utilization, idle server consumption dominated; at 80%, sustained utilization and queuing produced load saturation.
- Inference data center: At 60% and 80% inference utilization, approximately 3% and 15% of requests were not processed because of node limits and latency constraints.At 20% utilization, the facility received about 300 million prompts daily; at 80%, it received 1.42 billion.
- Load variability and service trade-offs: Inference PAR was 9% lower on average than colocation, but higher utilization reduced variability through saturation while risking degraded real-time service.The 40% inference scenario had a higher PAR than 20%, whereas PAR decreased at higher utilization as resources saturated.
- Inference data center: In the inference facility, requests ramped after 9 AM and declined after 10 PM, while higher utilization reduced the visible diurnal pattern through server saturation.The simulated profiles also aligned with typical business-time behavior across the week.
5. Conclusion
The paper connects high-resolution measurements of representative AI workloads to whole-facility energy modeling through a bottom-up, discrete-event framework. The results reveal workload- and utilization-dependent power behavior, while the authors identify hardware, workload, and omitted-facility-load limitations.
- Framework: The framework links node-level power measurements with whole-facility demand profiles using reproducible benchmarks and discrete-event simulation.The simulation includes job submission, scheduling, and execution profiles for infrastructure planning and operational studies.
- Workload measurements: Training power scaled linearly with node count under tested configurations, while runtime was strongly influenced by increased global batch size at higher node counts.Both training and inference jobs exhibited pronounced power transients that varied with model type and node count.
- Workload measurements: Online inference server power appeared to saturate at relatively modest request rates, motivating further study of customer-experience and energy-consumption trade-offs.The conclusion describes the resulting performance impact on power as diminishing beyond certain operating points.
- Facility-level findings: Whole-facility consumption followed diurnal, weekly, and seasonal patterns, with stronger variability at 20%, 40%, and 60% utilization and saturation at 80%.At high utilization, profiles were more grid-stable but could worsen customer experience, especially for inference facilities.
- Facility-level findings: Even with all servers utilized, facility power reached only 73% and 80% of rated design in the modeled cases, indicating possible auxiliary-infrastructure cost optimization.The conclusion presents this as an opportunity rather than a demonstrated optimization outcome.
- Limitations: The study is limited by one hardware setup, a limited workload and algorithm set, omitted auxiliary loads, and utilization distributions adapted from external datasets.The authors note that different accelerators, models, interconnects, and operational environments may have different power-performance characteristics.
- Practical contribution: The open-source node-level dataset and scaling methodology are intended to support data-driven AI infrastructure planning and load forecasting.The stated application scope includes decision support for data center infrastructure planning.
Appendix A. GPU and CPU Stress and Idle Tests
Appendix A measures GPU and CPU power under stress and idle conditions and reports an HPL-NVIDIA reference run on H100 nodes. These tests establish hardware power bounds and a compute-bound baseline for interpreting workload measurements.
- GPU stress tests: 668.2 W was the mean stressed GPU power after a 150-second warm-up, with ±1.4 W standard deviation.The figure reports the median and a 99% confidence bound for gpu-burn measurements.
- CPU stress tests: 338.6 W was the mean stressed CPU power after an 11-second warm-up, with ±1.0 W standard deviation.The test used a dense matrix multiplication kernel and reports a 99% confidence bound.
- Idle tests: Each idle GPU consumed 72.5 W on average with ±0.1 W standard deviation, while GPUs within a node operated at slightly different average power levels.The idle profiles also showed in-phase oscillations among GPU power curves.
- HPL-NVIDIA benchmark: The HPL-NVIDIA test produced two matrix-solve regions near 695 W on one four-GPU H100 node.The solves achieved 168.6 TFLOPs and 169.5 TFLOPs, equivalent to 42.15 and 42.38 TFLOPS per GPU.