PaperlayerView exact revision on arXiv

Source-linked AI summary

Activation Denoising: A Robustness View on Parallel vs Sequential LLM Quantization

Yan Scholten, Rachel Lawrence, James Hensman, Stephan Günnemann, Alicia Curth, Riccardo Grazzi

arXiv:2610.07522v1cs.LG

TL;DR

Parallel quantization is efficient but ignores upstream errors that compound, while sequential calibration addresses them at a serial bottleneck. This paper makes layers robust to modeled noise through activation denoising and parallel rounding; it recovers much of sequential quantization’s benefit, though hyperparameters require tuning.

  • Problem

    Parallel methods ignore upstream quantization errors that compound, while sequential methods correct them through layer-by-layer calibration that becomes a bottleneck at scale.

    Source: §1 · p. 1

  • Method

    Activation denoising models upstream error as noise, constructs a linear denoising filter for each layer, and rounds the transformed weights independently in parallel.

    Source: §1 · pp. 1–2

  • Results

    At 3 bits, parallel denoising closes 18–31% of the perplexity gap across all four scales; at 2 bits it outperforms sequential quantization on Llama-3-8B and Llama-2-13B.

    Source: §6.1 · p. 7

  • Takeaways & Limitations

    Treating upstream quantization errors as a robustness problem offers a more efficient alternative to sequential quantization for LLM compression.

    Source: §8 · p. 9

    The regularization introduces noise-scale and injection-magnitude hyperparameters that require tuning.

    Source: §7 · p. 9

Abstract

from arXiv · show

Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but quantization errors then compound through the residual stream, as no layer corrects for the errors of the layers before it. Sequential quantization accounts for this error compounding by re-calibrating each layer on the already-quantized outputs of its predecessors, yielding stronger results but at the cost of a serial schedule that becomes a bottleneck at scale. As a solution, we propose parallel quantization with activation denoising, which recovers much of the sequential benefit while keeping quantization fully parallel. Rather than re-calibrating layer-by-layer, we take a robustness perspective and model the upstream error as noise, regularizing to be robust to it through a preprocessing step followed by metric-weighted rounding. Applied at every layer, this regularization forms a depth-compounding smoothness penalty that dampens how strongly quantization errors amplify through the model. Unlike orthogonal rotations commonly used in quantization, which must preserve the model's function, we multiply the weights by a more general linear transformation. We find that the two are complementary and their effects compound. Empirically, our robustness regularization recovers a significant part of sequential quantization's benefit in a single parallel pass, at a fraction of its time. Overall, by treating compounding quantization errors as a robustness problem, we offer a principled foundation for more efficient and accurate LLM quantization at scale.

1 INTRODUCTION

Parallel post-training quantization is efficient but ignores upstream errors that accumulate through layers, while sequential calibration corrects them at a serial bottleneck. Activation denoising instead regularizes layers against noise in parallel, combining a denoising transform with noise-aware rounding.

1 INTRODUCTION

  • Existing parallel methods calibrate on clean inputs and ignore upstream quantization errors, while sequential methods account for them layer by layer at a scale-limiting serial bottleneck.Recovery fine-tuning can partially recover accuracy, but adds a separate costly training stage.Source: §1 · p. 1
  • Activation denoising injects random noise in one extra full-precision forward pass, uses its moments to construct a per-layer filter, and absorbs that filter into weights before rounding.The approach replaces sequential measurement and correction with noise-based regularization of every layer.Source: §1 · p. 1, Figure 1 · p. 2
  • The denoising filter acts as a depth-compounding smoothness penalty that limits how strongly quantized-layer errors amplify at inference.Modified weights are rounded independently and in parallel in a noise-aware metric.Source: §1 · p. 2
  • Activation denoising adds weighted column-wise L2 regularization through a general linear weight transform, complementing rotations that preserve the model’s function.The paper reports lower Frobenius norm and improvements over both random and learned rotations.Source: §1 · p. 2
  • Parallel activation denoising consistently improves 3- and 4-bit scalar and 2-bit vector quantization across models from 1B to 13B parameters, recovering a significant part of sequential quantization’s benefit.All layers remain quantized in parallel, and the remaining gap can be closed through cheaper trade-offs rather than a full sequential pass.Source: §1 · p. 2

2 RELATED WORK

Related work spans layer-wise PTQ, transformations that address outliers, and methods that compensate for errors across layers. Activation denoising differs by estimating upstream noise in parallel and incorporating it into rounding.

2 RELATED WORK

  • PTQ compresses LLMs without retraining; complementary approaches protect salient weights, allocate mixed precision, learn block-wise parameters, or use floating-point, vector, and trellis formats.The proposed framework supports both scalar and vector quantization and avoids recovery fine-tuning or full-precision low-rank inference corrections.Source: §2 · pp. 2–3
  • Orthogonal or Hadamard rotations redistribute outliers while preserving the full-precision output, whereas denoising changes each layer’s mapping to compensate for upstream errors and reduce weight Frobenius norm.The paper frames these approaches as complementary.Source: §2 · p. 3
  • Sequential compensation methods measure accumulated error in a fully sequential pass; activation denoising replaces that measurement with a parallel noise model used in the rounding target and metric.The framework recovers GPTQ, GPTAQ, and QEP objectives as special cases and requires no gradient-based optimization.Source: §2 · p. 3

3 BACKGROUND

The background defines PTQ as low-bit weight rounding using calibration data and explains why layer-wise output error, rather than independent weight rounding, is commonly optimized. It distinguishes independent parallel calibration from sequential calibration on perturbed inputs.

3 BACKGROUND

  • PTQ maps trained-model weights from high precision to a few bits without retraining, using uniform scalar rounding or joint vector rounding on groups of weights.The layer-wise objective minimizes output error over a small calibration set.Source: §3 · p. 3
  • Metric-weighted rounding minimizes distance from a continuous target under a positive-definite metric, which determines the cost of weight perturbations.In the standard layer-wise objective, the target is the original weights and the metric is the input covariance; GPTQ approximately solves the combinatorial problem column by column.Source: §3 · p. 3
  • Parallel quantization calibrates layers on clean inputs independently, whereas sequential quantization uses perturbed inputs produced after earlier layers have been quantized.Sequential objectives may use only perturbed inputs or compare perturbed-input quantized outputs with clean-input full-precision outputs.Source: §3 · p. 3

4 ACTIVATION DENOISING QUANTIZATION: A FILTERING PERSPECTIVE

Activation denoising treats upstream quantization error as input perturbation and seeks to denoise each layer in parallel. Its objective is converted into a weight pretransform followed by rounding in a new metric, without sequential computation or additional inference cost.

4 ACTIVATION DENOISING QUANTIZATION: A FILTERING PERSPECTIVE

  • Quantization errors propagate downstream, so activation denoising makes each layer denoise its input in parallel and reduces the objective to a linear weight transform followed by rounding in a new metric.The transformed procedure has zero additional inference cost.Source: §4 · p. 4

4.1 QUANTIZATION WITH ACTIVATION DENOISING

  • The objective compares quantized outputs on noisy inputs with full-precision outputs on target inputs, while a proximity penalty regularizes the weights.The input noise scale λ controls noise carried by the quantized layer’s input; target noise scale ρ controls how much noise the target retains.Source: §4.1 · p. 4
  • Partial denoising, with less noise in the target than in the input, typically performs best, even when the perturbation is the exact sequential error.ρ = 0 requires removing noise entirely, while ρ = λ adapts to it without removing it.Source: §4.1 · p. 4
  • The objective recovers parallel GPTQ at λ=0 and ρ=0, and several sequential objectives when λ=1 with corresponding target-noise settings.It also supports general noise models and flexible noise scales beyond those sequential objectives.Source: §4.1 · p. 4

4.2 DENOISING QUANTIZATION IN PRACTICE: PREPROCESS THEN ROUND

  • The pretransform uses input covariance H, signal-noise cross-covariance C, and noise covariance N; at ρ=0 and α=0 when the inverse exists, F is a generalized Wiener filter.In that case, F maps corrupted activations to estimates of clean activations.Source: §4.2 · pp. 4–5
  • The denoising objective is implemented as a per-layer weight pretransform followed by standard rounding in a new metric, using existing approximate rounding solvers.Given an estimate of upstream noise, the layers require no sequential computation and the method adds no inference cost.Source: §4.2 · pp. 4–5

5 ACTIVATION NOISE MODEL FOR EFFICIENT PARALLEL DENOISING

The method estimates upstream quantization error by injecting and propagating random activation noise, then uses the resulting statistics to denoise and quantize layers independently. Local noise encourages smaller weight columns, while propagated noise regularizes against errors compounding through depth.

5 ACTIVATION NOISE MODEL FOR EFFICIENT PARALLEL DENOISING

  • Independent, zero-mean per-channel Gaussian noise is scaled to each channel’s largest calibration activation, giving larger outlier channels proportionally more noise.The channel-specific scale is multiplied by a model-wide constant γ.Source: §5 · p. 5
  • Each layer receives propagated upstream noise plus a new, independent local injection, so early perturbations accumulate through depth.Sequential methods instead use the true upstream error, available only after quantizing preceding layers in sequence.Source: §5 · p. 5
  • One clean and one noise-injected full-precision pass provide moments for each layer, after which pretransforms and rounding can run independently without extra optimization or inference cost.The moments are used before each weight is rounded.Source: §5 · p. 5

5.1 RANDOM NOISE INJECTION AS ROBUSTNESS REGULARIZATION

  • Local noise combines with the proximity ridge to impose weighted column-wise L2 penalties toward down-scaled weight targets.In the denoising regime, this encourages smaller column-wise norms.Source: §5.1 · p. 6
  • Local and propagated noise together regularize layers for robustness, dampening how strongly quantization errors compound while keeping quantization parallel.The local penalty reduces column norms; propagated noise accounts for errors accumulating across layers.Source: §5.1 · p. 6

6 EXPERIMENTAL EVALUATION

Across four model scales, activation denoising recovers part of sequential quantization’s perplexity benefit while keeping quantization parallel. The evaluation also finds that denoising complements rotations and explores sequential prefixes and higher-bit modules as ways to close the remaining gap.

6 EXPERIMENTAL EVALUATION

  • The evaluation quantizes four Llama models at 3 and 2 bits, using GPTQ and QuIP# and comparing parallel and sequential baselines on perplexity and zero-shot accuracy.Weights are quantized while activations and KV cache remain at 16 bits; calibration uses WikiText-2 sequences.Source: §6 · p. 6

6.1 ACTIVATION DENOISING IMPROVES THE QUALITY OF PARALLEL QUANTIZATION

  • At 3 bits, denoising closes 18–31% of the perplexity gap across four scales; at 2 bits, it closes 38% and 86% on Llama-3.2-1B and 3B, and outperforms sequential quantization on Llama-3-8B and Llama-2-13B.Table 1 compares 3-bit GPTQ and 2-bit QuIP#; the gap is measured from parallel quantization to sequential asymmetric correction.Source: §6.1 · p. 7, Table 1 · p. 7
  • Denoising retains fully parallel quantization and projects a 7.1–7.5× time reduction on eight A100s versus the single-GPU sequential baseline.The projection assumes balanced work allocation and negligible communication overhead; the two forward passes take under 1% of Llama-2-13B quantization time.Source: §6.1 · p. 7, Figure 2 · p. 7

6.2 PARALLEL DENOISING COMPOSES EFFECTIVELY WITH ORTHOGONAL LLM ROTATIONS

  • Denoising lowers the Frobenius norm, which rotations preserve, and the two approaches are complementary.The denoised-to-original norm ratio is below 1 for all 112 linears, with the strongest reduction in attention output projections.Source: §6.2 · pp. 7–8
  • Across 100 draws, denoising composes effectively with random or learned rotations and is insensitive to the random noise draw, reducing the parallel–sequential gap.Parallel GPTQ varies widely across random rotations, whereas sequential GPTAQ is more robust to them.Source: §6.2 · p. 8, Figure 3 · p. 8

6.3 TWO EFFICIENT ALTERNATIVES TO FULL SEQUENTIAL QUANTIZATION

  • Sequentially quantizing the first quarter of blocks closes 40–60% of the gap, while closing the last few percent requires a prefix spanning most of the model depth.The benefit is front-loaded across all models.Source: §6.3 · p. 8, Figure 4 · p. 9
  • Promoting only value projections to 4 bits matches sequential quality at an average width of 3.03 bits, just 0.03 bits above W3.Under grouped-query attention, this option outperforms finer group sizes and other module prefixes in the comparison.Source: §6.3 · p. 8, Figure 4 · p. 9

7 DISCUSSION

Activation denoising recovers much of sequential quantization’s benefit in parallel, but a gap remains; measured upstream error and scale up to 13B models point to ways to extend the approach. The method adds tuning parameters, and adversarial robustness remains future work.

7 DISCUSSION

  • Extensions to activation and KV-cache quantization, stronger rounding schemes, and joint learning of noise injections and rotations remain future work.Recovery fine-tuning is also identified as an orthogonal future direction.Source: §7 · p. 9
  • Measured upstream error from an already parallel-quantized model can improve a second parallel pass, and the efficiency advantage should grow at larger scale.The paper reports demonstrations up to 13B parameters and says a short sequential prefix or a few 4-bit modules can close the remaining gap when a small trade-off is acceptable.Source: §7 · p. 9
  • The regularizer adds noise-scale and injection-magnitude hyperparameters, whose optimal values are reported as similar across models.The paper frames the method as average-case robustness to random perturbations; worst-case adversarial robustness remains a future direction.Source: §7 · p. 9

8 CONCLUSION

The conclusion presents activation denoising as a parallel alternative that recovers much of sequential post-training quantization’s benefit by making layers robust to noise. The appendix adds experiments, setup details, ablations, and theoretical material.

8 CONCLUSION

  • Activation denoising recovers much of sequential post-training quantization’s benefit while keeping quantization fully parallel.The conclusion frames robustness to noise as an alternative to expensive layer-by-layer recalibration for upstream quantization errors.Source: §8 · p. 9

APPENDIX OVERVIEW

  • The appendix provides additional experimental results and ablations, detailed experimental setup, links to sequential GPTAQ and QEP corrections, and full proofs and derivations.These materials are organized across Appendices A–D.Source: APPENDIX OVERVIEW · p. 14

A ADDITIONAL EXPERIMENTAL RESULTS AND ABLATIONS

A.1 NOISE-MODEL MECHANISM ABLATION: LOCAL VS. PROPAGATED NOISE INJECTION

The appendix ablates how noise injection works, explores noise-scale choices, and provides model- and quantization-specific results and sweeps. It also reports injection-scale and noise-model comparisons.

A.1 NOISE-MODEL MECHANISM ABLATION: LOCAL VS. PROPAGATED NOISE INJECTION

  • 28% of the parallel–sequential gap closes with propagated noise, versus 11% with local injection alone; without injected noise, the objective reduces to parallel GPTQ.Both noise-injection effects contribute, with the propagated depth-compounding term providing the majority of the benefit.Source: Figure 5 · p. 14, §A.1 · p. 14

A.2 NOISE-SCALE LANDSCAPE: PARTIAL DENOISING AND GENERALIZING SEQUENTIAL QUANTIZATION

  • 15.2 average perplexity is the best result at λ=1, while λ=0.8 and ρ=0.4 reach 15.12 using the same sequential upstream error.The broader objective can scale down the upstream error, improving on the best asymmetric correction reported for that error.Source: §A.2 · p. 15
  • The best configurations lie below ρ=λ: the objective shrinks weights there, whereas ρ>λ grows them and is associated with poorer performance.On the diagonal, the noise changes the rounding metric but does not correct the weights.Source: §A.2 · p. 15

A.3.1 W3A16 GPTQ SUMMARY ACROSS MODELS

  • The W3A16 GPTQ summary covers four Llama models, compares fixed random-Hadamard and learned SpinQuant rotations, and includes GPTQ-W3 proxy noise.Source: Table 2 · p. 16

A.3 FULL RESULT TABLES WITH HYPERPARAMETER SWEEP HEATMAPS

  • Detailed W3A16 tables cover Llama-3.2-1B, 3B, Llama-3-8B, and Llama-2-13B; a separate W4A16 table covers Llama-3.2-1B.The tables identify Gaussian noise and GPTQ proxy noise variants, including their combination.Source: Table 3 · p. 17, Table 4 · p. 18, Table 5 · p. 20
  • W2A16 QuIP# tables cover Llama-3.2-1B and 3B, Llama-3-8B, and Llama-2-13B; the Llama-3.2-1B table defines accuracy and the parallel-to-sequential gap.Its comparison includes parallel, denoising, and sequential correction variants with E8 rounding.Source: Table 8 · p. 25, Table 9 · p. 26, Table 10 · p. 27

A.4 INJECTED-NOISE SCALE ABLATION

  • 16.59 average perplexity is reached at γ=0.053 in the injected-noise-scale sweep; nearby values perform similarly, while γ above 0.07 increases perplexity quickly.Source: Figure 21 · p. 29, §A.4 · p. 29

A.5 RANDOM NOISE MODEL UNDER FORWARD PASS MODIFICATIONS

  • Per-channel noise improves over the parallel baseline for both W3 GPTQ and W2 QuIP#, while isotropic noise improves only for GPTQ.For QuIP#, isotropic noise stays above the parallel baseline across tested ρ and ridge α; the authors attribute the difference to unrotated activations and outlier channels.Source: Figure 22 · p. 29, §A.5 · p. 29

A.6 GENERALIZATION ABLATION: HELD-OUT EVALUATION

The appendix checks held-out generalization, extends trade-off diagnostics across models, and reports runtime measurements and experimental settings. It also documents the rotation choices and hyperparameter sweeps used for the main tables.

A.6 GENERALIZATION ABLATION: HELD-OUT EVALUATION

  • The held-out protocol selects configurations by WikiText-2 test perplexity but reports the average on C4 and FineWeb, and the qualitative conclusions remain unchanged.Calibration and validation use disjoint WikiText-2 splits, while evaluation uses separate datasets.Source: §A.6 · p. 30, Table 12 · p. 30
  • At 3 bits, denoising closes 34/21/32/12% of the parallel-to-sequential gap on Llama-3.2-1B/3B, Llama-3-8B, and Llama-2-13B under held-out evaluation.At 2 bits, it closes a large fraction of the gap on smaller models and matches or surpasses the sequential baseline on larger models.Source: §A.6 · p. 30, Table 12 · p. 30

A.7 SEQUENTIAL-PREFIX AND 4-BIT-PREFIX TRADE-OFFS ACROSS MODELS

  • Across all four models, the main-text conclusions hold for both sequential-prefix quantization diagnostics and promoting a prefix of modules to 4 bits.Figure 24 shows per-model perplexity and residual-stream SNR diagnostics; Figure 23 extends the effective-bit-width versus perplexity plot.Source: §A.7 · p. 31, Figure 23 · p. 31, Figure 24 · p. 32

A.8 RUNTIME MEASUREMENTS AND PROJECTED MULTI-GPU TIME

  • On 8 GPUs, the projection gives denoising a ≈7.5× reduction versus sequential quantization for Llama-2-13B, with projected reductions of ≈7.1×–≈7.5× across models.These are projections from single-GPU measurements, assume even layer splitting, and ignore communication overhead.Source: §A.8 · p. 33, Figure 25 · p. 34
  • Denoising’s extra noisy calibration pass adds only a few seconds, negligible beside per-layer rounding, although overall overhead depends on the rounding solver.With GPTQ, denoising takes 29.4 minutes versus 11.6 for parallel GPTQ and 24.1 for sequential GPTAQ on Llama-3-8B; QuIP# rounding is roughly an order of magnitude more expensive.Source: §A.8 · p. 33, Table 15 · p. 34

B.1 FIXED SETTING

  • Experiments use 128 WikiText-2 calibration sequences, symmetric per-channel weights, 16-bit activations and KV cache, and one GPU per run unless otherwise stated.Evaluation includes perplexity on WikiText-2, C4, and FineWeb and zero-shot accuracy on eight listed tasks.Source: §B.1 · p. 35

B.2 QUAROT ROTATIONS AND LEARNED SPINQUANT ROTATIONS

  • The GPTQ experiments compare fixed randomized-Hadamard and learned SpinQuant-style rotations, while QuIP# uses its own incoherence transform without an external rotation.Learned rotations are optimized once per model and bit-width, then reused across experiments.Source: §B.2 · pp. 35–36

B.3 MAIN TABLES HYPERPARAMETER SWEEPS

  • The main-table sweeps vary solver damping or correction strength for baselines and sweep ρ and δ across denoising configurations, using Gaussian-noise and/or parallel-GPTQ error proxies.W3/W4 use GPTQ; W2 uses QuIP#.Source: §B.3 · p. 36

B.4 RANDOMNESS ABLATION

The appendix specifies how randomness is resampled in the ablation, gives the Frobenius-norm analysis setting, and relates GPTAQ and QEP to the denoising objective. It describes their shared correction target and differing implementations.

B.4 RANDOMNESS ABLATION

  • The randomness ablation resamples rotations and injected noise over 100 seeds, with separate conditions for fixed learned rotations and random Hadamard rotations.The setup is on Llama-3.2-1B at W3A16; the denoising runs fix α=0.01, ρ=0.5, and δ=0.05.Source: §B.4 · pp. 36–37

B.5 FROBENIUS-NORM ANALYSIS SETUP

  • The Frobenius-norm analysis uses Llama-3.2-1B with ρ=0.5 and δ=0.05 to compare weights before and after the pretransform.Source: §B.5 · p. 37

C GPTAQ AND QEP AS SPECIAL CASES

  • GPTAQ and QEP are special cases of activation denoising when the sequential upstream error is used as noise.The appendix presents their closed forms before establishing this relationship.Source: C GPTAQ AND QEP AS SPECIAL CASES · p. 38

C.1 BACKGROUND ON GPTAQ AND QEP

  • GPTAQ folds the correction into GPTQ’s column sweep, whereas QEP applies a closed-form least-squares weight pretransform before standard GPTQ rounding.Both use a correction strength β for the asymmetric correction.Source: §C.1 · p. 38

C.2 GPTAQ AND QEP AS SPECIAL CASES OF ACTIVATION DENOISING

With sequential upstream error as the noise and λ=1, the denoising objective recovers GPTAQ and QEP’s rounding problem through β=1−ρ. The appendix then states the setup and proves that the objective is equivalent to pretransforming weights and rounding in a metric.

C.2 GPTAQ AND QEP AS SPECIAL CASES OF ACTIVATION DENOISING

  • With η=˜x−x, λ=1, and α=0 for simplicity, activation denoising yields the GPTAQ and QEP rounding problem with correction strength β=1−ρ.The equivalence assumes D=E[˜x˜x⊤] is invertible.Source: §C.2 · p. 39

D FULL PROOFS AND DERIVATIONS

D.1 SETUP AND NOTATION

  • The notation defines noisy and target activations and expresses the matrices G and B using second moments; α>0 makes G positive definite and F well defined.Source: §D.1 · p. 40

D.2 PROOF OF THEOREM 1

  • For α>0, the denoising optimization is equivalent to rounding the pretransformed weights in the stated metric, because the objectives differ only by a constant independent of the candidate weights.Source: §D.2 · p. 40

D.3 PROOF OF THEOREM 2

Theorem 2 characterizes independent local noise injection as an added column-wise regularizer. Depending on its parameters, that penalty can favor down-scaled weight columns.

D.3 PROOF OF THEOREM 2

  • Independent local noise injection makes the objective equal, up to a weight-independent constant, to the reconstruction objective without that injection plus a column-wise penalty.The theorem assumes zero-mean noise with diagonal covariance; Gaussianity is not required.Source: §D.3 · p. 41
  • For ρ<λ and σ_c>0, the penalty favors a down-scaled target s_cW_:,c; for ρ=λ, it is centered at the original weight column.The stated scaling satisfies s_c<1 exactly when ρ<λ for channels with σ_c>0 and λ>0.Source: §D.3 · p. 41
Loading 2610.07522v1…