Source-linked AI summary
Power Stabilization for AI Training Datacenters
Esha Choukse, Brijesh Warrier, Scot Heath, Luz Belmont, April Zhao, Hassan Ali Khan, Brian Harry, Matthew Kappel, Russell J. Hewett, Kushal Datta, Yu Pei, Caroline Lichtenberger, John Siegler, David Lukofsky, Zaid Kahn, Gurpreet Sahota, Andy Sullivan, Charles Frederick, Hien Thai, Rebecca Naughton, Daniel Jurnove, Justin Harp, Reid Carper, Nithish Mahalingam, Srini Varkala, Alok Gautam Kumbhare, Satyajit Desai, Venkatesh Ramamurthy, Praneeth Gottumukkala, Girish Bhatia, Kelsey Wildstone, Laurentiu Olariu, Ileana Incorvaia, Alex Wetmore, Prabhat Ram, Melur Raghuraman, Mohammed Ayna, Mike Kendrick, Ricardo Bianchini, Aaron Hurst, Reza Zamani, Xin Li, Michael Petrov, Gene Oden, Rory Carmichael, Tom Li, Apoorv Gupta, Pratikkumar Patel, Nilesh Dattani, Lawrence Marwong, Rob Nertney, Hirofumi Kobayashi, Jeff Liott, Miro Enev, Divya Ramakrishnan, Ian Buck, Jonah Alben
TL;DR
Large synchronized AI training jobs create power swings that can reach grid-relevant amplitudes and frequencies. This paper characterizes those swings with production telemetry and evaluates software, GPU, and infrastructure interventions, finding practical relief but also energy and reliability trade-offs.
Problem
Large synchronized AI workloads create substantial power swings because compute phases approach GPU TDP while communication phases approach idle power.
Method
The paper combines production measurements, real hardware, and Microsoft’s StratoSim simulator to assess software smoothing, GPU controls, energy storage, and telemetry backstops.
Results
10.5% total energy overhead resulted when software power smoothing used a floor set to 90% of TDP on the Figure 1 waveform.
Takeaways & Limitations
Cross-stack co-design offers practical and immediate relief for current deployments while supporting scalable, power-aware AI systems.
Takeaways & Limitations
Software smoothing can require precise telemetry, fallback logic, and calibration, while MPS ties primary and secondary workload failures at GPU scale.
Abstract
from arXiv · showhide
Large Artificial Intelligence (AI) training workloads spanning several tens of thousands of GPUs present unique power management challenges. These arise due to the high variability in power consumption during the training. Given the synchronous nature of these jobs, during every iteration there is a computation-heavy phase, where each GPU works on the local data, and a communication-heavy phase where all the GPUs synchronize on the data. Because compute-heavy phases require much more power than communication phases, large power swings occur. The amplitude of these power swings is ever increasing with the increase in the size of training jobs. An even bigger challenge arises from the frequency spectrum of these power swings which, if harmonized with critical frequencies of utilities, can cause physical damage to the power grid infrastructure. Therefore, to continue scaling AI training workloads safely, we need to stabilize the power of such workloads. This paper introduces the challenge with production data and explores innovative solutions across the stack: software, GPU hardware, and datacenter infrastructure. We present the pros and cons of each of these approaches and finally present a multi-pronged approach to solving the challenge. The proposed solutions are rigorously tested using a combination of real hardware and Microsoft's in-house cloud power simulator, providing critical insights into the efficacy of these interventions under real-world conditions.
NVIDIA
Frontier AI training clusters scale to more than 100,000 synchronized GPUs, whose compute and communication phases create power swings visible from nodes to the grid. The paper proposes cross-stack mitigation grounded in production telemetry.
- A single training job can span more than 100,000 GPUs operating in lockstep under the bulk synchronous paradigm.
- Compute and communication phases differ sharply in power use, with computation near GPU TDP and communication near idle power.
- Synchronous GPU activity produces power swings visible at rack, datacenter, and grid levels, potentially reaching tens or hundreds of megawatts.
- The paper evaluates software workload injection, GPU firmware controls, and rack-level energy storage as three mitigation classes.
- Cross-industry co-design across software, hardware, and infrastructure is advocated to keep AI systems scalable and power-aware.
B. Compute and communication phases
Distributed training alternates local computation with synchronized communication, producing large GPU power variations that aggregate across servers and datacenter infrastructure. These swings motivate systemic stabilization across software, hardware, and infrastructure layers.
- Each iteration distributes data across GPUs, which independently compute forward-pass predictions and local backward-pass gradients.
- All-reduce aggregates gradients across GPUs so every replica receives an identical update before synchronous parameter updates.
- Checkpointing periodically saves model and sometimes optimizer states, enabling recovery without losing significant computational effort.
- Most data-parallel workloads retain end-of-iteration synchronization, while checkpointing, recovery, and load balancing add further activity patterns.
- GPU power drops sharply during communication and checkpointing because compute resources become idle or underutilized.
- GPUs contribute more than 50% of provisioned server power, allowing synchronized utilization changes to create datacenter-scale swings of tens of megawatts.
D. Challenges to generation systems
Large synchronous AI loads can excite resonances in turbine-generators, transmission networks, and other grid components. Effective mitigation therefore requires both time-domain limits and frequency-domain specifications tailored to utility conditions.
- Cyclical load fluctuations near turbine-generator torsional frequencies can induce high-amplitude shaft oscillations and risk mechanical fatigue or failure.
- Sub-synchronous AI workload frequencies can excite transmission-network resonances, causing SSR, inter-area oscillations, voltage flicker, or frequency-modulation effects.
- Growing GPU counts increase aggregate swing amplitude at critical frequencies, motivating prevention of resonance excitation.
- Utilities may constrain ramp-up rate, ramp-down rate, and dynamic power range to limit how quickly loads change.
- Frequency-domain specifications can define a critical range such as 0.1–20 Hz and cap spectral magnitude at 20% of total harmonic energy within that range.
- AI workload FFT energy is concentrated between 0.2–3 Hz, near known resonant modes, so specifications must cover a dynamic frequency range.
B. Understanding the frequency-domain spec
Power-system components contain resonant frequencies from approximately 0.16 Hz to above 60 Hz, creating multiple frequency bands relevant to large synchronous AI loads.
- Resonant frequencies in the generation and delivery network range from approximately 0.16 Hz to greater than 60 Hz.
- <1Hz: Below 1 Hz, long transmission lines connecting independently strong grid portions can produce resonance.
- 1Hz to 2.5Hz: From 1 Hz to 2.5 Hz, oscillations can occur between closely coupled sources, including units within or near plants.
- 7 Hz to >100 Hz: From 7 Hz to above 100 Hz, shaft torsional critical frequencies arise as turbine-generator masses oscillate against one another.
C. Additional requirements
Successful mitigation must satisfy multiple operational requirements beyond reducing synchronous power swings, including adaptability, low performance impact, low energy waste, and EDP control.
- Mitigation must adapt to differing utility specifications and critical-frequency requirements across regions.
- Solutions should introduce negligible degradation to training throughput or convergence times.
- Solutions should reduce power variability without materially increasing net energy consumption.
- EDP peaks can occur over 50 ms while GPUs maintain TDP at a 1-second granularity, requiring control when peaks reach utility infrastructure.
IV. MITIGATION STRATEGIES
The paper evaluates software-based smoothing that injects secondary workloads, emphasizing utility compliance, telemetry, performance, reliability, and energy trade-offs.
- Utility specifications define acceptable magnitudes at critical frequencies, so mitigations must meet the strictest applicable requirements.
- Software-only mitigation: Software smoothing injects power-hungry secondary workloads when GPU activity or power falls, reducing swings across compute and communication phases.
- Software-only mitigation: Artificial secondary workloads avoid state-save delays but waste energy when they perform no useful work.
- Software-only mitigation: Reliable 100 ms counters are too slow for detecting 20 Hz swings, so the software design requires faster telemetry sources.
- Software-only mitigation: Firefly combines NVIDIA MPS, GPU block-activity monitoring, and scaled matrix multiplications, increasing utilization to 100% of TDP.
- Software-only mitigation: MPS reduced primary-workload overhead below 5%, but continuous 1 ms power monitoring consumed substantial CPU cores and host-device bandwidth.
- Software-only mitigation: MPS couples primary and secondary workloads within one GPU context, allowing faults in one workload to propagate to the other.
- Software-only mitigation: Software smoothing is flexible and quick to deploy without hardware changes, but requires calibration, fallback logic, and trade-offs between performance and wasted energy.
B. GPU power smoothing
GPU-level smoothing uses programmable power profiles to constrain ramps and maintain a minimum power floor, reducing response latency while introducing energy and dynamic-range trade-offs.
- GPU power smoothing: GB200 GPU power smoothing lets developers or cloud providers program per-GPU profiles with ramp rates, a minimum power floor, and stop delay.
- GPU power smoothing: Ramp-up and ramp-down rates can directly encode utility time-domain requirements in watts per second.
- GPU power smoothing: The minimum power floor, combined with GPU TDP, constrains short-timescale changes to satisfy a utility dynamic-power-range specification.
- GPU power smoothing: A higher MPF reduces performance impact under the dynamic-range specification but increases energy burn.
- GPU power smoothing: 65% of TDP was used as the power floor in the GB200 square-wave microbenchmark, which shows ramp-up, steady, stop-delay, and ramp-down phases.
- GPU power smoothing: 10.5% total energy overhead resulted when StratoSim applied a 90%-of-TDP floor to the training waveform.
- GPU power smoothing: GB200’s maximum 90% MPF and minimum 1.1× TDP EDP leave at least 20% of TDP as GPU-level dynamic range.
- GPU power smoothing: A 10 MW dynamic-range requirement for a 100 MW job cannot be met with current GPU power smoothing.
C. Energy-storage solution
The energy-storage approach absorbs power during low-power communication phases and releases it during high-power computation, potentially reducing peak demand without wasting energy. Rack-level placement is preferred, but broad frequency coverage and rare ramp events can require costly capacitance.
- Requirements: The solution must directly measure load, provide sufficient capacitance, support sudden power changes, and switch rapidly between charging and discharging.
- Energy-storage operation: Energy storage can charge during low-power communication phases and discharge during high-power computation phases to smooth the training waveform.The simulated example shows battery charge alongside the final waveform.
- Placement: The preferred placement is at the rack level because higher-level placement exposes more power infrastructure to perturbations without useful demand multiplexing for synchronous jobs.The paper identifies rack-level storage as the best option.
- Challenges: Meeting both high- and low-frequency demands, especially rare ramp-up and ramp-down events, may require expensive, space-intensive, high-capacitance storage.Higher frequencies are easier to filter, while ramp events can impose very large capacitance requirements.
D. Putting a solution together.
The paper combines GPU-level smoothing with rack-level energy storage because software and hardware controls address operating constraints while storage reduces wasted energy and peak demand. The combined design improves coverage but requires coordination and additional infrastructure.
- Trade-offs: Software and hardware solutions are readily available with current hardware and have similar energy overheads, while hardware is more reliable at scale without software resource overheads.
- Trade-offs: Stricter dynamic-range specifications may require combining hardware with a Firefly-like software solution because hardware alone is limited by the 90% MPF limit.
- Trade-offs: Energy storage adds hardware cost and embodied carbon, and storage alone requires high capacitance for ramp-up and ramp-down specifications.
- Combined approach: The proposed architecture combines GPU-level software or hardware smoothing with rack-level energy storage.The GPU-level mechanism covers ramp specifications and corner cases when storage capacity is exhausted.
- Future scale: Future deployments may supplement rack-proximate mitigation with long-duration battery energy storage systems at larger scale.
E. Fast telemetry-based backstop
A fast telemetry-based backstop monitors datacenter power waveforms for critical sub-synchronous frequencies that proactive smoothing may occasionally miss. Fine-grained spectral detection enables escalating responses from throttling to power shedding or coordinated disconnects.
- Monitoring: The backstop continuously monitors datacenter power waveforms for early signs of instability or resonance beyond primary mitigation.
- Detection and response: Fine-grained telemetry and real-time spectral analysis, including FFT-bin monitoring, detect problematic frequencies and trigger tiered responses.
- Detection and response: Responses can escalate from soft throttling or load shaping to circuit-level power shedding or coordinated disconnects.Execution is coordinated with site-specific infrastructure logic.
V. CALL TO ACTION
The paper calls for coordinated action across AI system design, utilities, and industry to reduce power swings, share grid requirements, and establish interoperable mitigation standards. It presents cross-stack stabilization as practical for current deployments but requiring sustained collaboration.
- AI Framework and System Designers: AI framework and system designers should explore less synchronous, power-aware training methods such as asynchronous training, staggered scheduling, and overlapped computation and communication.
- Utility Providers and Grid Operators: Utilities and grid operators should openly share resonance and ramp specifications and establish standardized communication with datacenter operators.The stated goal is safer grid operation and avoidance of unplanned outages or equipment degradation.
- Industry Collaboration: Industry should support open forums such as OCP to develop interoperable standards for telemetry, load signaling, and sub-synchronous oscillation mitigation.The paper states that no single customer, vendor, or hyperscaler can solve the problem alone.
- Conclusion: The cross-stack combination of software smoothing, GPU-level controls, and rack-level storage offers practical and immediate relief for current deployments.The approach is backed by real-world measurements and simulation.
- Conclusion: Future progress requires sustained collaboration across research, industry, and utilities through shared standards, validation frameworks, and architectural best practices.