PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference

Petar Basta

arXiv:2610.17531v1cs.LGcs.AIcs.CY

TL;DR

Evidence on quantization’s energy effects has been limited, despite the importance of inference energy. This paper benchmarks weight-only quantization and finds that fused low-bit methods can substantially reduce energy while preserving strong accuracy.

  • Problem

    Prior quantization evaluations often focused on accuracy and memory, with limited measurement of energy across workloads.

    Source: Introduction · p. 11

  • Method

    The study systematically benchmarks energy use across quantization methods and varying input and output stages, alongside runtime, memory, and accuracy against an FP32 baseline.

    Source: Introduction · p. 11, §5.1 · p. 37

  • Results

    Fused low-bit methods provided the best energy-accuracy trade-offs, with up to 4× energy savings for single-token outputs and EETQ sustaining nearly 2× savings for long sequences.

    Source: Conclusion · p. 65

  • Takeaways & Limitations

    On the evaluated benchmarks, latency is a reliable proxy for total energy consumption, and larger batches can moderately amplify efficiency gains.

    Source: Conclusion · p. 65

    The relative energy contributions of reduced memory traffic and lower-precision computation remain unresolved.

    Source: §6.1.4 · p. 68

Abstract

from arXiv · show

While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce. We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime. We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs. We identify a novel trend showing that quantization techniques tend to exhibit peak energy efficiency relative to full-precision baselines when inputs are sufficiently long and outputs are short. Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest. Notably, fused-kernel implementations such as EETQ int8 and Bitsandbytes int8 offer the highest energy savings, up to 4x compared to FP32 on short text generation, with negligible accuracy loss. Finally, we observe that energy usage closely tracks runtime on our evaluated benchmarks, indicating that, in practice, latency optimization can serve as a reliable proxy for reducing the environmental footprint of LLM services. We conclude with suggested directions for strategically selecting low-carbon quantization strategies based on specific inference requirements.

Author’s Declaration

The thesis introduces a systematic study of weight-only quantization and LLM inference energy, alongside its author’s declaration, dedication, and chapter roadmap.

Author’s Declaration

  • The author declares sole authorship, confirms this is the accepted thesis copy, and notes it may be made publicly available electronically.Source: Author’s Declaration · p. 2

Dedication

Introduction

  • Prior work often emphasized accuracy and memory, motivating a systematic benchmark linking quantization choices to energy across input and output stages.The introduction frames quantization as reducing memory footprint and traffic, and asks how energy per token, output length, and kernel choices affect efficiency.Source: Introduction · pp. 11–12

1.1 Thesis Contributions

  • The thesis contributes a holistic empirical comparison of weight-only post-training quantization, measuring energy, runtime, and accuracy across workloads, models, and GPUs.Source: §1.1 · p. 12

1.2 Thesis Organization

  • The chapters progress from foundations and related work through quantization techniques and empirical evaluation, then conclude with findings and future directions.Source: §1.2 · p. 13

Background

This background explains LLM inference and Transformer components, then introduces quantization schemes and their trade-offs in efficiency and accuracy.

2.1 LLMs for Inference

  • Inference has a parallel prefill phase over prompt tokens and an autoregressive decode phase that generates one token at a time.Source: §2.1 · p. 14

2.2 Transformer Architecture

  • Transformers combine attention and feed-forward networks; scaled dot-product and multi-head attention model token relationships and attend to multiple features in parallel.Source: §2.2 · p. 15, §2.2.1 · p. 15

2.2.2 Encoder vs. Decoder

  • Encoder-only models contextualize the input, while decoder-only models use masked self-attention to generate tokens autoregressively after prompt prefill.Source: §2.2.2 · p. 16

2.3 Quantization Background

  • Quantization maps floating-point weights to lower-precision integers using a scale and optional zero-point, with range endpoints mapped to integer endpoints.Source: §2.3 · p. 16

2.3.1 Symmetric vs. Asymmetric Quantization

  • Symmetric quantization centers the range around zero and uses no zero-point; asymmetric quantization allows an off-center range and uses an offset.The text notes symmetric quantization can simplify hardware implementation, while asymmetric quantization can better represent data not centered around zero.Source: §2.3.1 · pp. 16–17

2.3.2 Quantization Granularity

  • Quantization granularity determines how broadly scale and zero-point parameters are shared, trading memory and compute efficiency against accuracy.Source: §2.3.2 · p. 17

Per-Tensor Quantization

Per-Channel Quantization

Related Work

Prior work examines quantization accuracy, memory, or datacenter energy, but does not jointly measure quantized inference accuracy, latency, and energy across hardware. This chapter also surveys seven PTQ techniques and compares their symmetry and granularity support.

Related Work

  • The study addresses a gap in Li et al.’s accuracy-focused evaluation by jointly benchmarking quantized LLM accuracy, latency, and energy across GPUs and configurations.Li et al. did not measure real-world inference efficiency, GPU latency, or energy, or examine variation across hardware and sequence lengths.Source: Related Work · p. 19
  • Jin et al. report a speed–memory tradeoff but no detailed hardware-level inference-time or power measurements across GPUs or sequence lengths.This study extends that work by measuring quantized inference accuracy and energy/latency across models and GPUs.Source: Related Work · pp. 19–20
  • Hu et al. analyze GPU power at the datacenter level, whereas this benchmark measures how quantization affects model-level inference performance and energy.Their study reports that GPUs draw roughly two-thirds of server power but does not examine quantization or per-query inference.Source: Related Work · p. 20
  • Alizadeh et al. [2] study energy and accuracy across coding tasks but treat quantization uniformly through GGUF, unlike this study’s comparison of ten PTQ methods.The present work also analyzes how algorithmic choices and input/output token ratios relate to energy efficiency.Source: Related Work · p. 20

Quantization Techniques

4.1 Quanto

Quanto, HQQ, and TorchAO offer distinct low-bit PTQ approaches, differing in calibration needs, quantization granularity, inference kernels, and integration choices.

4.1.1 Quantization Approach

  • Quanto uses linear weight quantization from int2 to int8 or float8, with per-channel or per-group scales and symmetric zero-points by default.Its initial PTQ procedure inserts quantize/dequantize hooks and determines scales through calibration or known distributions.Source: §4.1.1 · p. 22

4.1.2 Quantized Inference Mechanism

  • Quanto replaces float layers with quantized modules and uses CUDA integer matrix multiplication, propagating quantized outputs between compatible layers to limit conversion boundaries.It does not tune quantization parameters against sample outputs, which may reduce accuracy at very low bit-widths unless group-wise schemes are used.Source: §4.1.2 · p. 23, §4.1.3 · p. 23

4.2.1 Quantization Approach

  • HQQ is calibration-free: it optimizes weight reconstruction with a heavy-tailed loss and alternating residual-thresholding and parameter updates.Closed-form updates allow convergence in a few iterations; grouping axis and group size trade accuracy, speed, and storage.Source: §4.2 · p. 23, §4.2.1 · p. 24

4.2.2 Quantized Inference Mechanism

  • HQQ produces linearly dequantized weights compatible with optimized kernels, while less common configurations may fall back to slower PyTorch dequantization.Axis=1 with fused CUDA/Triton kernels is recommended for speed; HQQ can also quantize models from weights without calibration data.Source: §4.2.2 · pp. 24–25, §4.2.3 · p. 25

4.3.1 Quantization Approach

  • TorchAO’s weight-only int4 mode quantizes linear weights per row while typically keeping activations in FP16 or FP32.Each output row receives its own scale and zero-point, usually with a symmetric zero-point of zero.Source: §4.3 · p. 25, §4.3.1 · p. 25

4.3.2 Quantized Inference Mechanism

  • TorchAO packs int4 weights and uses kernels to unpack and multiply them with higher-precision activations, delaying dequantization where possible.It also supports optional 2:4 structured sparsity alongside int4 on supported hardware.Source: §4.3.2 · p. 26

4.3.3 Distinctive Features and Tradeoffs

  • TorchAO is a modular framework focused on performance and integration, rather than a single quantization algorithm, and can incorporate other techniques such as HQQ.It emphasizes torch.compile compatibility and supports additional options including FP8 dynamic quantization.Source: §4.3.3 · pp. 26–27

4.4 Bitsandbytes

Bitsandbytes compresses weights with NF4 or int8 schemes, while AutoAWQ uses calibration-derived activation statistics to protect salient channels before low-bit quantization.

4.4.1 Quantization Approach

  • Bitsandbytes NF4 maps normalized weight blocks to 16 nonuniform levels chosen from normal-distribution quantiles, concentrating levels near zero.Each block stores 4-bit codes and a scale based on its maximum absolute weight.Source: §4.4.1 · p. 27
  • Bitsandbytes int8 keeps outlier feature columns in 16-bit while quantizing the remaining weights, and its methods generally do not require calibration data.NF4 is recommended over its alternative FP4 scheme for better accuracy.Source: §4.4.1 · pp. 27–28

4.4.2 Quantized Inference Mechanism

  • Bitsandbytes’ 4-bit CUDA kernels unpack and partially dequantize weights for higher-precision computation rather than using pure integer arithmetic.Double quantization reduces scale-storage overhead by quantizing scales to 8 bits, with a minor runtime overhead.Source: §4.4.2 · p. 28
  • Bitsandbytes int8 kernels separately multiply quantized weights and full-precision outlier columns, while activations remain full precision.Its 4-bit path does not fully leverage int4 Tensor Cores and uses half-precision operations, unlike EETQ’s true int8-core approach.Source: §4.4.2 · p. 28, §4.4.3 · p. 29

4.5.1 Quantization Approach

  • AWQ uses calibration inputs to identify activation-salient channels whose weights are especially sensitive to quantization error.It scales salient weights and inversely scales corresponding activations to preserve the computation.Source: §4.5 · p. 29, §4.5.1 · pp. 29–30
  • AWQ folds offline scale factors into model weights or normalization, then quantizes all weights uniformly, typically to 4 bits.The method reports that roughly 1% of channels typically need significant scaling.Source: §4.5.1 · p. 30

4.5.2 Quantized Inference Mechanism

  • AutoAWQ uses fused CUDA 4-bit matrix-multiplication kernels optimized separately for prefill and single-token decoding.Its kernels support batched GEMM for initial token processing and vector-matrix multiplication for decode.Source: §4.5.2 · pp. 30–31

4.5.3 Distinctive Features and Tradeoffs

  • AWQ uses lightweight activation-statistics calibration rather than iterative reconstruction, and its uniform int4 weights permit one kernel without mixed-precision branching.The cited AWQ work reports that protecting as few as 1% of channels reduced worst-case quantization error.Source: §4.5.3 · p. 31

4.6 GPTQ (Generalized Post-Training Quantization)

GPTQ minimizes layer output error through calibrated, sequential weight quantization, whereas EETQ uses simpler weight-only int8 quantization with GPU-optimized kernels.

4.6.1 Quantization Approach

  • GPTQ minimizes layer output error by sequentially quantizing weight groups and compensating for each group’s error using an approximate Hessian.It uses calibration inputs and commonly applies asymmetric 4-bit quantization with configurable group sizes.Source: §4.6 · p. 31, §4.6.1 · p. 32

4.6.2 Quantized Inference Mechanism

  • GPTQ inference applies quantized weights to full-precision inputs, accumulates in FP32, then applies scales and any zero-point offset.Its grouped weights are stored with accompanying scale and zero-point values.Source: §4.6.2 · pp. 32–33

4.6.3 Distinctive Features and Tradeoffs

  • GPTQ’s error minimization requires calibration activations and can take hours on very large models; its layer-wise procedure is not globally optimal.The method’s calibration also precomputes matrices such as XXT, which can be heavy for large layers.Source: §4.6.3 · p. 33

4.7.1 Quantization Approach

  • EETQ quantizes linear and embedding weights per channel to int8 at model loading, without calibration, while leaving activations unchanged.Its conservative 8-bit design is described as incurring very small relative error.Source: §4.7 · p. 34, §4.7.1 · p. 34

4.7.2 Quantized Inference Mechanism

  • EETQ pairs per-row int8 weights with custom fused GPU kernels that use NVIDIA Tensor Cores for int8 matrix multiplication.The inference engine may dynamically scale input vectors for int8 arithmetic.Source: §4.7.2 · p. 34

4.7.3 Distinctive Features and Tradeoffs

  • EETQ prioritizes speed and efficiency, while its weight-only design permits full-precision LoRA layers to be added during inference.The text contrasts its GPU specialization and single int8 setting with frameworks offering more configuration options.Source: §4.7.3 · p. 35

Evaluation

The evaluation compares quantization’s energy efficiency, runtime, memory use, and accuracy across models, tasks, sequence lengths, batch sizes, and GPUs. Energy benefits vary with workload and implementation, with the strongest gains concentrated in particular quantizers and prefill-heavy settings.

5.1 Methodology

  • The evaluation covers five open-source models, four task benchmarks, ten quantization strategies, and four NVIDIA GPUs, using FP32 as the unquantized baseline.It measures energy, runtime, memory, and accuracy; experiments use PyTorch and Hugging Face Transformers while avoiding specialized inference engines to isolate quantization-related gains.Source: §5.1 · pp. 36–37

5.2 Energy Efficiency & Speedup

  • EETQ int8 and Bitsandbytes int8 lead energy efficiency across the tested A6000 models, while GPTQ int4 also performs well on longer-output tasks.On H100, the reported trends generalize for supported quantizers; EETQ is not supported on that GPU.Source: §5.2 · p. 39, Figure 5.1 · p. 41, Figure 5.2 · p. 43
  • Short-output MNLI and MMLU workloads show substantial energy reductions, whereas decode-heavy HumanEval and GSM8K workloads show minimal savings.Large prefill matmuls benefit more from low-bit quantization than smaller decode matmuls, where kernel-launch and memory-management overheads weigh more heavily.Source: §5.2 · pp. 39–40
  • Quantization’s energy advantage grows when inputs are long and outputs are short, with gains often reaching 4–8× over FP32 for 1,000- or 5,000-token inputs.For Llama 2 7B, gains begin around 100 input tokens on A6000 but significant improvements emerge only around 5,000 tokens on A100.Source: §5.2 · pp. 40, 47
  • Speedup rankings broadly align with energy-efficiency rankings, but some quantizers reduce energy more than they reduce runtime, and speedups are slightly diminished on H100.EETQ int8 and Bitsandbytes int8 lead speedups on A6000; several 4-bit and general-toolkit methods have moderate speedups or underperform FP32.Source: §5.2 · pp. 40, 47, Figure 5.5 · p. 49, Figure 5.6 · p. 51
  • Energy outcomes differ even among methods at the same bitwidth: GPTQ int4 performs strongly, while HQQ and Quanto variants can consume more energy than FP32.The authors attribute these differences to backend kernels, memory access, and compute-path efficiency; 4-bit methods without native hardware acceleration can incur overhead or underuse tensor cores.Source: §5.2 · pp. 39, 52
  • As batch size increases, energy efficiency improves for almost every quantization strategy, but usually only by a small-to-medium margin.For long-context inference, batching to 4–8 captures most remaining benefit; larger batches can yield little or negative marginal improvement after memory saturation.Source: §5.2 · pp. 52–53
  • Batching affects decode-heavy workloads more than prefill-heavy ones: absolute energy falls steeply with batch size for short-input, long-output tasks but stays relatively flat for long-input, short-output tasks.In the prefill-dominated 1,000-token-input, 1-token-output case, Bitsandbytes int8 rises from about 3.6× efficiency at batch 1 to near 6.2× at batch 8.Source: §5.2 · pp. 52–53

5.3 Accuracy Degradation

Accuracy outcomes vary by bit width, quantization strategy, task, and model size. Int8 methods usually preserve performance more consistently, while 4-bit methods require more careful selection.

5.3 Accuracy Degradation

  • Table 5.1 marks drops greater than 5% in red and drops from 2.5% to 5% in yellow relative to vanilla FP32.The table presents performance across the evaluated benchmarks.Source: §5.3 · p. 53, Table 5.1 · p. 56
  • Int8 strategies preserve accuracy within 2.5% of the full-precision baseline in the majority of evaluated cases.The section characterizes Bitsandbytes int8, EETQ int8, HQQ int8, and Quanto int8 as consistently preserving performance across diverse tasks.Source: §5.3 · p. 53
  • AutoAWQ int4 is relatively robust, while several other 4-bit strategies show substantial task-dependent accuracy drops.The comparison identifies Bitsandbytes nf4, GPTQ int4, HQQ int4, Quanto int4, and TorchAO int4 among methods with substantial drops across the majority of tasks.Source: §5.3 · p. 56
  • Larger models tend to tolerate low-bit quantization better than smaller models, particularly at 4-bit precision.The passage attributes this pattern to larger models' parameter redundancy absorbing quantization noise.Source: §5.3 · p. 56
  • The section recommends selecting 4-bit methods according to model size, task type, and acceptable quality thresholds.It describes 8-bit quantization as a reliable compression option with minimal accuracy sacrifice.Source: §5.3 · p. 56

5.4 Runtime as a Proxy for Energy Efficiency

Across the evaluated models and benchmarks, runtime and energy usage are strongly correlated. The paper therefore presents latency reduction as a practical proxy for reducing inference energy.

5.4 Runtime as a Proxy for Energy Efficiency

  • Runtime and energy usage correlate at 0.989–1.000 across models and benchmarks, averaging 0.9966 with a standard deviation of 0.0033.The passage reports that the strong correlation is consistent across three models and four benchmarks.Source: §5.4 · p. 57
  • The paper concludes that reducing inference latency is a reliable proxy for lowering energy use and environmental footprint in practice.The conclusion is based on the reported strong runtime–energy correlation.Source: §5.4 · p. 57

5.5 Memory Usage

All evaluated strategies reduce memory relative to vanilla FP32, with the largest average gains from 4-bit methods. Memory efficiency also varies across models, so method and architecture both matter.

5.5 Memory Usage

  • EETQ int8 achieves a 3.87× memory efficiency ratio, above Bitsandbytes int8 at 3.53× and Quanto int8 at 2.87×.HQQ int8 has the lowest reported memory savings overall, at 2.52×.Source: §5.5 · p. 57
  • AutoAWQ int4 and TorchAO int4 achieve the highest average memory efficiency ratios across models, at 5.73× and 5.55× relative to vanilla FP32.The table aggregates the ratios across all models.Source: §5.5 · p. 57, Table 5.2 · p. 62
  • Memory-constrained settings such as edge deployments or limited-VRAM servers can benefit substantially from quantization, with computational-overhead and accuracy trade-offs.The section advises considering both quantization method and model architecture when targeting these settings.Source: §5.5 · p. 62
  • AutoAWQ int4 reduces memory 6.7× on Phi 3.5 Mini but 5.1× on Llama 2 7B, illustrating model-specific variation.The passage also reports that TorchAO int4 performs consistently well across models, while HQQ int4 and Quanto int8 fluctuate more.Source: §5.5 · p. 62

5.6 Cost & Emissions

Estimated inference costs and emissions differ substantially by quantization strategy, with Bitsandbytes int8 and EETQ int8 lowest across datasets. These estimates use specified AWS us-east-1 electricity and carbon-intensity assumptions.

5.6 Cost & Emissions

  • The estimates assume AWS us-east-1 electricity at $0.093 USD per kWh and emissions of 0.393 kg CO2 per kWh.Figure 5.11 reports cost and emissions per billion queries on an A6000, with both axes logarithmic.Source: §5.6 · p. 62
  • The paper states that choosing an efficient quantization scheme can reduce CO2 emissions by orders of magnitude without sacrificing accuracy.This is presented as a consequence of the reported cost and emissions comparisons.Source: §5.6 · p. 64
  • Bitsandbytes int8 and EETQ int8 have the lowest estimated costs and emissions across datasets, while HQQ int4 can be substantially more expensive.The relative ordering is reported as consistent across datasets, with magnitude differences driven largely by per-query runtime and energy use.Source: Figure 5.11 · p. 64, §5.6 · p. 64

Conclusion

Quantization’s strongest energy advantage appears on long-input, short-output workloads, where prefill can amortize dequantization overhead. Fused int8 methods combine substantial energy savings with accuracy close to full precision, and runtime tracks energy use in the evaluated settings.

Conclusion

  • Up to 4× energy savings for single-token outputs came from fused low-bit methods, with EETQ sustaining nearly 2× savings on long sequences.EETQ int8 and Bitsandbytes int8 provided the best energy-accuracy trade-offs in the evaluated strategies.Source: Conclusion · p. 65
  • Int8 quantization preserved accuracy within 2.5% of full-precision baselines, although the conclusion notes that 4-bit methods still pose risks.Source: Conclusion · p. 65
  • Latency serves as a reliable proxy for total energy consumption in the evaluated results, a relationship attributed to modern GPUs’ high static power draw.Reducing total runtime can matter more than minimizing instantaneous power.Source: Conclusion · p. 65
  • Larger batch sizes can moderately amplify energy-efficiency gains by improving hardware utilization and distributing fixed architectural overhead across more tokens.Source: Conclusion · p. 65
  • Quantization reaches peak energy efficiency on long-input, short-output workloads, as prefill amortizes dequantization overhead across many floating-point operations.Autoregressive decoding uses smaller matrix-vector operations, where dequantization costs form a larger share of execution.Source: Conclusion · pp. 65–66

6.1 Future Work

The future-work directions address resource management, broader quantization, hardware advances, and quantization configuration as routes toward more sustainable LLM deployment.

6.1.1 Context-Aware Scheduling and Precision Allocation

  • Schedulers could route short or simple requests to quantized models and longer or mission-critical requests to full precision, while shifting flexible jobs to cleaner times or regions.Source: §6.1.1 · pp. 66–67

6.1.2 Extending Quantization Beyond Model Weights

  • An 8-bit KV cache reduces the stated 10,000-token, 7B FP32 cache estimate from ∼10 GB to ∼2.5 GB; activation quantization could save more but risks numerical error from outliers.Source: §6.1.2 · p. 67

6.1.3 Low-Bit Hardware and Accelerator Optimization

  • Future evaluations could test quantized inference on TPUs and emerging AI accelerators optimized for low-bit operations.Source: §6.1.3 · p. 67

6.1.4 Sources of Energy Efficiency in Quantized Inference

  • The relative energy contributions of reduced memory traffic and lower-precision arithmetic remain unclear; systematic memory, compute, and power breakdowns could guide future designs.Source: §6.1.4 · p. 68

6.1.5 Quantization Configuration

  • The study does not fully explore granularity or symmetry, whose optimal settings may depend on the task and interact with other hyperparameters.Source: §6.1.5 · p. 68

6.1.6 Overall Outlook

  • The overall roadmap combines smarter scheduling, broader quantization, hardware specialization, and configuration choices to balance efficiency with expected service quality.Source: §6.1.6 · p. 68

Related papers

Loading 2610.17531v1…