Source-linked AI summary
Characterizing Job Power Elasticity for Power-Flexible AI Training
Philip Colangelo, Charles Dawson, Shayan Sengupta, Ayse Coskun, Varun Sivaram
TL;DR
LLM training is a growing electricity demand, while power availability limits new AI infrastructure and jobs can respond differently to the same GPU power cap. The paper measures this power elasticity, introduces PFI, estimates it from telemetry, and shows how it can support power-aware workload orchestration.
Problem
Power availability constrains AI infrastructure, but jobs with similar baseline throughput and power draw can respond differently to GPU power caps, limiting heuristic power allocation.
Method
The paper characterizes LLM training power elasticity through controlled GPU power modulation, defines PFI to quantify performance cost, and develops telemetry-driven PFI estimators for orchestration.
Results
Training-job power elasticity can be quantified, estimated at runtime from telemetry, and exploited for power-aware workload orchestration.
Takeaways & Limitations
PFI provides a control-oriented basis for protecting relatively inflexible jobs while assigning deeper power reductions to relatively flexible jobs under a global power budget.
Takeaways & Limitations
The study is scoped to NVIDIA H200 GPUs, specific training workloads and 30-minute runs; out-of-sample curtailment validation, closed-loop scheduler validation, and broader accelerator coverage remain future work.
Abstract
from arXiv · showhide
Large language model (LLM) training is among the fastest-growing sources of electricity demand in modern data centers, and power availability is a primary bottleneck to continued AI infrastructure growth. Making the power consumption of these workloads flexible could unlock additional power for AI growth, limit increases in electricity prices, and improve the utilization of existing grid infrastructure. However, to realize this flexibility, we must first understand how the performance of training workloads changes when GPU power is reduced. This paper presents the first systematic characterization of \emph{job power elasticity} (the sensitivity of throughput to power reductions) in LLM training. To quantify elasticity, we introduce the \emph{Power Flexibility Index (PFI)}, a normalized metric that quantifies the performance cost of power reductions and provides a control primitive for SLA-aware power flexibility. We collect data from 131 LLM training runs on H200 (plus 24 H200 validation runs and 34 matched H100 runs), including both dense and mixture-of-experts models, pretraining and fine-tuning tasks, and up to 32 GPUs. We find that LLM training jobs exhibit substantial but variable power elasticity, and we identify telemetry signals that predict PFI at runtime. Finally, we demonstrate that PFI-aware power allocation maximizes total tokens/second throughput under power constraints. Under a 30\% power reduction, PFI-aware power allocation recovers ~1.5k tokens/s per job, 63\% of the performance gap between an equal-weight allocation and an oracle with perfect information. Our results establish power elasticity as a measurable property of training jobs and provide a foundation for power-aware, grid-responsive AI infrastructure.
1 Introduction
LLM training is a growing source of data-center electricity demand, while power constraints make SLA-aware flexibility important. The paper characterizes throughput sensitivity to power reductions, introduces PFI, predicts it from telemetry, and applies it to power allocation.
- Motivation: Power availability constrains AI infrastructure growth, motivating training workloads that reduce power while maintaining acceptable SLA-defined performance.The relevant control target is throughput sensitivity to sustained power reductions during peak grid-load periods.
- Research gap: Existing efficiency metrics provide limited insight into how job throughput responds to sustained reductions in available power.Prior metrics focus on nominal energy efficiency, MFU, tokens per joule, or individual-job energy–time tradeoffs.
- Approach: The paper presents a systematic characterization of LLM power elasticity across architectures, tasks, and distributed GPU scales, using 131 controlled training runs.The study computes PFI from GPU power-cap sweeps to compare jobs’ throughput costs under power reduction.
- Approach: PFI is a normalized job-level metric that quantifies throughput cost under power reduction and enables cluster managers to rank jobs by relative flexibility.This ranking supports assigning deeper reductions to flexible jobs while protecting less flexible jobs.
- Operationalization: Telemetry-driven PFI prediction and PFI-aware allocation improve aggregate throughput under cluster-wide GPU power constraints.The approach estimates flexibility online from GPU monitoring signals and uses it to allocate power across realistic job mixes.
2 Power Elasticity and the Power Flexibility Index
The paper defines job power elasticity as throughput sensitivity to sustained power reductions and introduces PFI to quantify power-flexibility trade-offs. It motivates runtime prediction and power-aware orchestration because jobs with similar baseline behavior can respond differently to power caps.
- Motivation and Research Questions: The paper addresses two gaps: no accepted quantitative definition supports systematic comparison, and job characteristics determining or predicting elasticity remain limited.The motivating questions span architecture, training task, scale, mechanisms, runtime prediction, and real-time orchestration.
- Power Flexibility Index: PFI is a normalized ratio of relative power decrease to relative throughput decrease, averaged across capped measurements against an uncapped reference.The uncapped reference uses the throughput and power measured at thermal design power; the uncapped point is excluded because throughput decrease is zero.
- Power Flexibility Index: High-PFI jobs preserve throughput under power reduction, whereas low-PFI jobs lose throughput relatively quickly and are less power-flexible.The paper labels PFI > 1 inelastic, PFI = 1 linear, and PFI < 1 elastic.
- Power Flexibility Index: PFI is comparable across models on the same accelerator but should not be compared directly across accelerator generations because power-cap mechanisms differ.The metric normalizes against platform-specific reference points, making hardware dependence an explicit scope condition.
- Motivation and Research Questions: Because similar jobs can respond differently to identical caps, power managers can preserve throughput by ranking jobs according to their ability to absorb reductions.The paper frames PFI as a control-oriented signal for allocating reductions while minimizing disruption under SLA constraints.
3 Experimental Setup
The experiments sweep per-GPU power caps across LLM training configurations spanning model architectures, tasks, and GPU counts. Measurements combine controlled cap changes with GPU telemetry and step-level throughput logging during representative runs.
- Experiment Design: 131 training runs cover four open-weight models, dense and MoE architectures, pretraining and rank-32 LoRA fine-tuning, and 8, 16, or 32 GPUs.The study also includes 24 runs in four held-out validation sweeps.
- Experiment Design: Power caps range from 200 W to 700 W per GPU, with 100 W increments for 8-GPU jobs and four selected cap values for 16- and 32-GPU jobs.The 16- and 32-GPU sweeps use 200, 500, 600, and 700 W.
- Measurement Procedure: Figure 2 shows aggregate GPU power and training throughput for a representative QWEN3-32B pretraining run on 32×H200 at 700 W/GPU.
- Measurement Procedure: The setup measures aggregate GPU power, utilization, memory, training throughput, and MFU, while aggregating reported metrics over the steady training phase.DCGM samples telemetry every 5 seconds, and the analysis uses the mean of the top 10% of samples to exclude evaluations and checkpoints.
4 Results
LLM training jobs show substantial variation in power elasticity, with PFI differing across jobs, architectures, and training tasks. Memory- and compute-related telemetry is strongly associated with these differences, although the analysis is limited by dataset size.
- Summary: The study identifies PFI variation, architecture- and task-dependent distributions, and strong memory- and compute-related telemetry associations as its three principal observations.Complete data tables are provided in Appendix C.
- PFI variation: PFI ranges from 1.10 to 2.67 across jobs, showing that training workloads differ substantially in throughput preserved under GPU power capping.Some jobs degrade nearly linearly with power, while others retain high throughput under aggressive reductions; no job had PFI < 1.
- Architecture and training task: Dense jobs have the lowest median PFI at 1.27, while MoE fine-tuning has the highest at 2.10 and MoE pretraining falls between at 1.43.Dunn’s test supports MoE fine-tuning as a distinct cluster, but does not distinguish MoE pretraining as a separate third cluster.
- Telemetry associations: Memory-related telemetry positively correlates with PFI, whereas tensor pipe activity correlates negatively, linking flexibility regimes to memory and compute intensity.The reported signals include mean DRAM activity, DRAM copy product, and tensor pipe activity.
- Scope: The analysis limits its feature set to signals with strong mechanistic links to power flexibility because the dataset is small.This constraint affects the scope of the telemetry analysis rather than the reported PFI measurements.
5 Discussion
The discussion connects PFI differences to how H200 power caps reduce compute frequency while preserving memory bandwidth, and uses telemetry to estimate PFI for production settings. The estimator remains constrained by the small number of independent PFI observations.
- Mechanism: H200 power capping preferentially reduces SM clock frequency while leaving memory frequency essentially unchanged, so memory-bound jobs lose less throughput than compute-bound jobs.MoE models show higher memory-related telemetry, providing a measurable proxy for these flexibility regimes.
- Scope: The mechanism’s cross-architecture generality remains outside the study’s full scope, with only limited H100 validation reported.The H100 validation shows a similar pattern of throttling compute speed more than memory bandwidth.
- Production use: PFI requires measurements across multiple power caps, making direct production calculation difficult and motivating telemetry-based estimation for jobs whose architecture or task is unknown to the power manager.Runtime telemetry can connect observable GPU behavior to the relevant PFI regime.
- PFI prediction: A linear model using DRAM copy product alone performs best across leave-one-out and leave-one-cluster-out evaluation, with validation R2 values consistent with LOCO results.The feature is motivated by the memory-bound mechanism and may proxy both job type and flexibility regime.
- Limitations: The study cannot justify higher-order models because only 25 PFI samples were derived from 131 training runs.Future work may expand the dataset and evaluate nonlinear relationships or additional DCGM metrics.
6 Exploiting Power Flexibility Index
PFI-aware allocation uses telemetry-estimated job flexibility to assign deeper reductions to flexible jobs and protect inflexible ones under a global power budget. In simulations, it matched or exceeded equal-weight allocation and recovered substantial throughput toward an oracle.
- PFI-aware orchestration protects relatively inflexible jobs while assigning deeper reductions to relatively flexible jobs to maximize tokens/sec under a global power budget.
- The evaluation compares oracle, equal-weight, MoE fine-tuning-weighted, PFI-aware, and uncalibrated-DCGM allocation strategies.
- Simulations use 500 synthetic workloads with 100 jobs each, including random sampling and a production pretraining/fine-tuning mix.
- ∼1.5k tokens/s per job recovered under a 30% production-mix power reduction, closing 63% of the gap between equal-weight allocation and an oracle.PFI-aware allocation outperformed equal weighting particularly for production workloads containing many small fine-tuning jobs and fewer large pretraining jobs.
- PFI-aware allocation yields equal or higher throughput than equal-weight allocation across all tested power-reduction levels and workload-mix scenarios.
7 Related Work
Prior work studies power capping, ML efficiency, energy-aware optimization, carbon-aware orchestration, and demand response, but does not provide this paper’s job-level LLM power-flexibility characterization. The paper positions PFI and telemetry-based prediction as tools for throughput-preserving power-aware operations.
- Existing hardware and ML systems work establishes power capping and energy optimization, but does not characterize power-performance tradeoffs across modern LLM tasks, architectures, and cluster scales.
- Carbon-aware computing and demand-response systems shift workloads or shape power profiles, while this work quantifies throughput costs for specific LLM training jobs.
- PFI provides a job-level scalar for comparing power flexibility and supports scheduler decisions that preserve cluster performance and maintain SLAs.
- Telemetry-based PFI prediction makes the approach practically implementable and improves simulated cluster performance.
8 Limitations
The study is limited in hardware, workload, power-measurement, sample-size, and validation scope. Broader architecture coverage, out-of-sample scheduler validation, and additional workload settings remain future work.
- The study is scoped to NVIDIA H200 GPUs, a3-ultragpu-8g nodes, nvidia-smi power control, 30-minute runs, and pretraining or rank-32 LoRA SFT workloads.The 32-GPU experiments include only Qwen3-32B pretraining because rank-32 LoRA SFT is rarely deployed at that scale and capacity was constrained.
- PFI uses aggregate GPU power only, excluding host, networking, power-supply, cooling, and other facility-level energy streams.The paper identifies broadening the power signal as a direct extension when those measurements become available.
- The 25-sample PFI dataset limits cross-architecture generalization, while the curtailment simulation reuses measurement curves and lacks out-of-sample workload validation.
- Closed-loop scheduler validation and extensions to other accelerators, heterogeneous clusters, RLHF, and inference remain future work.
9 Conclusion
The paper argues that LLM training power elasticity can be quantified, estimated at runtime, and used for power-aware orchestration. It frames PFI as a practical basis for grid-responsive infrastructure while comparing alternative definitions and their tradeoffs.
- 9 Conclusion: LLM training-job power elasticity can be quantified, estimated at runtime, and exploited for intelligent power-aware workload orchestration.
- 9 Conclusion: The conclusion connects power-flexible AI infrastructure with reducing consumption when needed while maximizing throughput and minimizing disruption to customer workloads.
- A Alternative definitions of the power flexibility index (PFI): The ratio PFI definition interprets flexibility as fractional power reduction relative to fractional throughput reduction and preserves job ordering across sweep grids.
- A Alternative definitions of the power flexibility index (PFI): The area-between-curves definition is robust to sample-point choice but less intuitive than the ratio definition.
- A Alternative definitions of the power flexibility index (PFI): The constant-elasticity model maps PFI from 0 for throughput proportional to power to 1 for throughput independent of power, using one degree of freedom per job.
- A Alternative definitions of the power flexibility index (PFI): The three PFI definitions are highly correlated, but the paper adopts the ratio definition for interpretability and its dynamic range of approximately [1, 2.5].
B.7 Compute resources
The paper-included runs used approximately 1,000 H200 GPU-hours, while the broader project used approximately 1,725 H200 GPU-hours across 410 telemetry-producing submissions.
- Compute resources: Approximately 1,000 H200 GPU-hours supported the 131 headline and 24 validation runs included in the paper.The compute accounting uses actual DCGM-bracketed durations rather than the 30-minute training cap.
- Compute resources: Across all profiling regions, 410 H200 submissions produced DCGM telemetry and consumed approximately 1,725 GPU-hours.The 155 reported runs represented roughly one third of submissions and 62% of H200 GPU-hours.
- Compute resources: Table 6 reports PFI for headline H200 sweep groups, with confidence intervals estimated from Monte Carlo resampling of measurement noise.Each group uses one fixed job configuration evaluated at multiple power caps.
C Data tables
The data tables and supplementary figures document PFI measurements, matched H100 comparisons, sweep-grid sensitivity, and task/architecture analyses, with limitations on cross-grid magnitude comparisons and one incomplete sweep.
- Data tables and figures: Tables 6 and 7 provide raw PFI values and confidence intervals, while Figures 11–13 provide corresponding raw and normalized power-throughput curves.The validation set contains four held-out H200 sweep groups.
- Data limitations: One sweep group contains four rather than six runs because the 500 W and 200 W runs failed with transient errors and were not rerun.The affected measurements therefore have an incomplete power-cap sweep.
- Cross-platform validation: The matched H100 validation comprises 34 runs across six sweep groups covering dense and MoE pretraining and fine-tuning cells.The main characterization remains confined to H200 GPUs.
- Cross-platform validation: PFI rank ordering is largely preserved across matched H100 and H200 sweeps, with Spearman ρ = 0.943 and median cross-platform difference of 3.5%.The H200 DCGM-to-PFI model fit achieves R2 = 0.56 on H100 data, but the sample is too small for definitive platform conclusions.
- Power-cap grid sensitivity: PFI remains useful as an ordinal signal across different sweep resolutions, although absolute values are comparable only on the same power-cap grid.Restricting 8-GPU groups to the coarse grid increases PFI by a median 0.154, or 10.7%, while preserving ordering with Spearman ρ = 0.940.
- Task and architecture analysis: The task/architecture analysis rejects equal PFI across dense, MoE pretraining, and MoE fine-tuning groups, with Kruskal–Wallis H = 12.37, df = 2, p = 0.0021.Dunn’s post-hoc comparisons use Holm–Bonferroni correction across all three pairs.
D.5 Sensitivity analysis of fit PFI model
The sensitivity analysis shows that the observed aggregate association between the DRAM copy product and PFI is driven by contrasts among architecture-task clusters rather than within-cluster variation.
- Sensitivity analysis: Table 12 tests predictor sensitivity by removing high-leverage sweep groups and recomputing the Spearman correlation at each step.The removed groups are ordered by leverage computed from the predictor alone.
- Sensitivity analysis: The correlation between DRAM copy product and PFI reflects contrasts between architecture-task clusters rather than variation within those clusters.The architecture-stratified analysis finds no significant dense-subset correlation and no significant MoE-subset correlation.
- Sensitivity analysis: Figure 14 extends the metric analysis to all measured DCGM metrics using fitted lines, Spearman correlations, and FDR-corrected significance annotations.The figure uses Benjamini–Hochberg correction across all metrics shown.