Source-linked AI summary
One Simple Trick for Improving the Performance of Energy-Limited Local Inference and Training
Erik Schultheis, Maximilian Kleinegger, Dan Alistarh
TL;DR
Power and thermal limits can throttle alternating compute- and memory-bound workloads, especially on constrained GPUs. The paper studies chunking as a way to smooth power draw, reporting larger gains on the DGX Spark than on the L40S while acknowledging limited device and workload coverage.
Problem
Power and thermal throttling in alternating compute- and memory-bound LLM workloads remains less studied than runtime efficiency, despite its effects on performance and energy consumption.
Method
The paper evaluates chunking, which interleaves compute and memory operations at finer granularity, across GPUs and frameworks using workload profiling and a minimal reproduction.
Results
On the DGX Spark, chunking reduces training wall-clock time by 20% and total energy by 14%, while the less constrained L40S shows a 1-2% effect.
Takeaways & Limitations
Power-consumption patterns, not only total arithmetic or energy, matter for performance on power-constrained GPUs, making chunk size a potential power-shape optimization parameter.
Takeaways & Limitations
The study covers one edge device and one datacenter GPU, uses a small Qwen2 training variant, and does not provide automatic chunk-size selection.
Abstract
from arXiv · showhide
Energy supply and heat dissipation are two of the main challenges with modern GPU deployments. While typically discussed in the context of new datacenter constructions, the same constraints also apply to small form-factor consumer devices, such as the DGX spark. In workloads characterized by alternating compute-intensive tasks such as matmuls with memory-bound operations such as norms or cross-entropy, the compute-intensive parts might hit power and/or thermal limits and start throttling. In this short paper, we show that chunking the workload into smaller parts that alternate compute and memory in higher frequencies, these power and temperature spikes can be smoothed out, preventing throttling and resulting in considerably faster wall-clock time and reduced total energy consumption. We present several scenarios in which this effect can be exploited on a DGX Spark with up to 2% performance and energy improvements, and demonstrate that the same phenomenon also happens on less constrained systems, such as a multi-GPU server, albeit at significantly reduced effect size of 1-2%.
1. Introduction
The paper identifies power-driven throttling as an underexplored source of inefficient LLM inference and training, then shows that chunking can smooth power spikes and improve runtime and energy efficiency.
- 1. Introduction: The authors characterize periodic throttling during vLLM inference and local training on a DGX Spark, observed as fan-speed variation and performance fluctuations.The behavior was associated with the LM-head layer in large-vocabulary models.
- 1. Introduction: Power spikes arise during dense LM-head computation when the GPU reaches its power limit and DVFS reduces clock speeds.Large vocabularies make the LM-head computationally dominant; the forward and backward passes require approximately 2 · batch · seq len · dim · vocab and 4 · batch · seq len · dim · vocab operations, respectively.
- 1. Introduction: On the DGX Spark, chunking interleaves compute and memory operations, reducing temperature spikes by 10°C, improving performance by up to 20%, and lowering total energy consumption by 14%.It splits entire LM-head batches into smaller sequential segments.
- 1. Introduction: Chunking also produces measurable 1-2% improvements on less constrained L40S GPUs.The experiments span multiple systems and frameworks and include a minimal reproduction script.
- 1. Introduction: The paper identifies and profiles a power-throttling phenomenon that significantly affects performance on edge devices during LLM inference and training.The authors analyze its root causes and evaluate chunking as a mitigation.
- 1. Introduction: Chunking serves as a power-smoothing strategy that improves runtime and energy efficiency by up to 20% on power-constrained devices.The approach extends chunking beyond its traditional role as a memory optimization technique.
2. Related Work
Prior efficiency work emphasizes runtime, memory, computation, and total energy, while this paper studies power-spike patterns and thermal throttling as a distinct issue.
- 2. Related Work: Existing GPU power-management research studies DVFS and energy optimization, but power measurements can be imprecise and power patterns remain difficult to characterize.The paper positions its analysis alongside work on GPU power and thermal management.
- 2. Related Work: Large-vocabulary LM-head projections and cross-entropy can dominate memory footprint and computation time when vocabulary size greatly exceeds hidden dimension.Related mitigations include fused or reformulated loss-computation techniques.
- 2. Related Work: Prior chunking and micro-batching methods primarily target memory consumption, latency, or effective batch size rather than power smoothing.The paper states that prior work had not characterized chunking’s impact on thermal throttling behavior.
- 2. Related Work: Quantization, pruning, and efficient architectures reduce computation and may indirectly reduce energy, but do not address the power-spike patterns studied here.The paper distinguishes these approaches from smoothing instantaneous power consumption.
- 2. Related Work: This work bridges runtime and energy-efficiency research by showing that power-consumption patterns can affect performance on edge devices.It presents chunking as a technique for smoothing power consumption and preventing thermal throttling beyond its memory benefits.
3. Experiments
Experiments show that chunking alternating compute- and memory-bound workloads can smooth power and temperature behavior, improving efficiency and runtime on constrained GPUs. Benefits are strongest for DGX Spark training, remain measurable in synthetic workloads and L40S tests, and are smaller and sequence-length-dependent for vLLM inference.
- 3.1. Training of small LLMs: 20% faster training and 14% lower energy on DGX Spark result from applying LM-head chunking to Qwen2-0.5B training.The configuration changes from batch microbatch size 64 to 16, reducing runtime from 20 minutes 45 seconds to 16 minutes 30 seconds and energy from 57 Wh to 49 Wh.
- 3.2. Minimal reproduction: Chunking stabilized power and temperature by replacing periodic bursts with more uniform warp activity and more stable GPC and SYS clocks.The single-chunk sequence had roughly 10°C lower overall temperature but nearly twice the confidence interval, whereas 16 chunks produced more stable telemetry.
- 3.2. Minimal reproduction: 2.5% lower iteration time and 1.3% lower energy were achieved with four chunks in the synthetic matmul-softmax-matmul workload.Duration decreased from 796 ms to 776 ms, while energy decreased from 2747 J to 2712 J.
- 3.3. Performance Impact during Inference: vLLM inference improvements reached at most 1.6% for sequence lengths near 32k, but chunking overhead can instead reduce performance.No effect appeared at smaller sequence lengths, so inference requires careful evaluation of sequence length and chunk size.
4. Discussion
Chunking smooths instantaneous power draw, helping power-constrained GPUs sustain boost clocks and avoid throttling. Its benefits are larger on edge devices, while deployment frameworks and performance models should account for pacing, despite important scope and measurement limitations.
- Why does chunking help?: Chunking helps by smoothing instantaneous power rather than changing total arithmetic, allowing devices to sustain boost clocks and avoid reactive frequency dips.Additional kernel-launch overhead is offset by longer operation at sustained boost clocks.
- Why edge devices benefit more: 15-22% on the kernel benchmark and up to 20% wall-clock and 14% energy improvements on DGX Spark exceed the 1-2% measured on L40S.The difference is attributed to tighter power and thermal envelopes on compact systems, which trigger DVFS throttling more frequently.
- Practical implications: Frameworks for edge deployment should expose chunk size as a power-shape control, not only a memory parameter.The paper also warns that FLOPS and memory bandwidth alone can misestimate runtime when identical computations are paced differently.
- Limitations: The study covers one edge device, one datacenter GPU, and a small Qwen2 training experiment, limiting conclusions about other platforms and larger models.NVML measurements are cross-checked but cannot fully separate thermal from electrical throttling, and optimal chunk size lacks automatic selection.
- Future work: Online adaptive chunking and partitioning fused kernels are proposed as future directions, with possible application beyond LM-heads to other alternating compute- and memory-bound workloads.Examples include MoE routing, large QK projections, and convolutional bottlenecks.
5. Conclusion
The paper shows that periodic throttling on power- and thermally-constrained GPUs wastes performance and energy. Chunking the LM-head smooths power draw, producing large DGX Spark gains and smaller but measurable L40S improvements, with a reproduction script supporting further study.
- 5. Conclusion: Chunking the LM-head reduces DGX Spark training wall-clock by 20% and total energy by 14%, while producing modest but measurable gains on L40S.A minimal Python reproduction on a synthetic workload indicates that improved power-draw stability, rather than changed arithmetic intensity, underlies the effect.