PaperlayerView exact revision on arXiv

Source-linked AI summary

SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights

Lorenz K. Müller, Philippe Bich, Jiawei Zhuang, Ahmet Çelik, Luca Benfenati, Lukas Cavigelli

arXiv:2509.22944v4cs.LG

TL;DR

Weight quantization helps reduce inference costs, while uncalibrated uniform quantizers seek output quality comparable to calibrated or non-uniform approaches. SINQ adds a second-axis scale and uses Sinkhorn-Knopp-style iterations to approximate activation awareness from weight structure. It improves perplexity over uncalibrated baselines, sometimes reducing the residual gap to the 16-bit baseline by more than half.

  • Problem

    Uncalibrated uniform quantization seeks output quality comparable to calibrated quantizers or non-uniform formats, which can entail additional costs or drawbacks.

    Source: §1 · p. 1

  • Method

    SINQ adds a second-axis scale and uses Sinkhorn-Knopp-style iterations to derive scales from weight structure that mimic activation awareness.

    Source: §1 · p. 1

  • Results

    SINQ outperforms baselines in every uncalibrated case on C4 and WikiText2 perplexity, sometimes reducing the residual difference to the 16-bit baseline by more than half.

    Source: §3.1 · p. 5

  • Takeaways & Limitations

    The paper reports that an uncalibrated uniform quantizer can approach the output quality of calibrated quantizers or non-uniform formats while being simpler.

    Source: §1 · p. 1

    Some model layers must share the second scale, creating a trade-off between output quality and minor inference-time overhead.

    Source: §2.3.1 · p. 4

Abstract

from arXiv · show

Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision. Still, current methods show perplexity degradation at bit-widths less than or equal to 4, partly because representing outliers causes precision issues in parameters that share the same scales as these outliers. This problem is especially pronounced for calibration-free, uniform quantization methods. We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn-Knopp-style algorithm that finds scales to normalize per-row and per-column variances. We show that this approximates activation-aware quantization by recovering column scales from the weight matrix structure that are predictive of the typical activation magnitudes the matrix received during training. Our method has no interactions between layers and can be trivially applied to new architectures to quantize any linear layer. We evaluate our method on the Qwen3 model family, among others. SINQ reduces the perplexity gap on WikiText2 and C4 by over 50% against uncalibrated uniform quantization baselines, incurs zero to negligible compute overhead, and can be further enhanced by combining it with calibration and non-uniform quantization levels. Code is available at https://github.com/huawei-csl/SINQ.

1. Introduction

Weight quantization reduces inference storage, memory, and data movement, but calibration and non-uniform formats can add costs. SINQ uses dual scaling and Sinkhorn-Knopp iterations to approximate activation-aware quantization, and the paper reports substantial gains over calibration-free baselines.

1. Introduction

  • Weight quantization reduces the storage, memory, and data movement required for neural-network inference, which can make it useful when those components bottleneck performance.Moving from float16 to int4 weights can yield a potential 4x speedup in memory-bound scenarios.Source: §1 · p. 1
  • Column-wise weight standard deviations predict input scales, providing a calibration-free route to pseudo-activation-aware quantization.Source: §1 · p. 1
  • SINQ adds a second-axis scale and uses fast Sinkhorn-Knopp iterations to find column scales that mimic activation awareness while avoiding row-wise kurtosis increase.Source: §1 · p. 1
  • SINQ substantially improves over state-of-the-art calibration-free quantization baselines across experiments at different model scales.The paper reports that its uncalibrated uniform quantizer can approach the output quality of calibrated quantizers or non-uniform formats.Source: §1 · p. 1
  • Dual scaling trades quantization-error impact between rows and columns, unlike single-scale quantization.Source: Figure 1 · p. 2

2. Methods

SINQ adds a second-axis scale to quantized weight tiles and uses weight structure to approximate activation-aware scaling without calibration. Its normalization balances row and column standard deviations, and it can also be combined with calibration methods such as AWQ.

2.1.1. PARAMETERIZATION PER TILE

  • Dual scales let quantization trade row-wise error against column-wise error, while dual-scale-plus-shift tiles add a small auxiliary-parameter overhead.For an outlying weight, scaling its row up and its column down shifts where quantization error occurs; rectangular tiling uses 2 × N × M/T + M auxiliary parameters.Source: §2.1.1 · p. 2, §2.1.2 · p. 2

2.2.1. PSEUDO-ACTIVATION-AWARE QUANTIZATION FROM WEIGHT STRUCTURE

  • SINQ uses per-column weight standard deviations as predictors of input magnitudes, but balances column normalization with row stability to avoid new quantization errors.Directly scaling columns by inverse standard deviation can worsen row outliers; the proposed Sinkhorn-Knopp-style method iteratively normalizes row and column standard deviations.Source: §2.2.1 · p. 3
  • SINQ’s input-side scales have a higher coefficient of determination with average input magnitudes than raw column standard deviations.The weight-structure relationship is reported across Qwen, Phi, and Llama models.Source: §2.2.1 · p. 3
  • Unlike Hadamard rotation, which gives better matrix reconstruction, SINQ reduces activation reconstruction error in most evaluated attention-layer cases.The comparison is against RTN on Qwen3-1.7B attention layers.Source: §2.2.1 · p. 3, Figure 3 · p. 4

2.2.2. ACTIVATION-AWARE CALIBRATION WITH SINQ

  • Combining SINQ with AWQ as ASINQ helps prevent row-wise kurtosis from increasing too much when AWQ scales are applied.AWQ determines input scales by minimizing the difference between original and scaled, quantized layer outputs.Source: §2.2.2 · pp. 3–4

2.3. Implementation Considerations

  • The dual-scale formulation applies its second scale to layer inputs, adding one element-wise activation scaling similar to AWQ.The reported overhead of this additional scaling is small in practice.Source: §2.3 · p. 4

2.3.1. NO-OVERHEAD SINQ

  • No-overhead SINQ absorbs input scales into preceding layers, but some model layers must share the second scale, trading output quality against minor inference-time overhead.The paper gives Q, K, and V layers, and Gate and Up-Projection, as shared-scale groups in Qwen-3 models.Source: §2.3.1 · p. 4

3. Experiments

Experiments evaluate SINQ across Qwen3 models and quantization settings, including uncalibrated uniform and non-uniform quantization, calibration, inference, and quantization time. Results show improved uncalibrated perplexity, compatibility with other quantization approaches, and small inference overhead.

3.1. Uncalibrated Uniform Quantization

  • SINQ outperforms baselines in every uncalibrated C4 and WikiText2 perplexity case, sometimes reducing the residual difference to the 16-bit baseline by more than half.Table 1 reports uncalibrated uniform PTQ on Qwen3 models at 3 and 4 bits.Source: Table 1 · p. 5, §3.1 · p. 5

3.1.1. RESULTS ON LARGE MODELS

  • On Qwen3-235B-A22B and DeepSeek-V2.5-236B, both MoE models, results support SINQ's robustness across different architectures.DeepSeek-V2.5-236B also uses Multi-head Latent Attention.Source: §3.1.1 · p. 5

3.2. Uncalibrated Non-Uniform Quantization

  • SINQ improves over the NF4 baseline; for the 32B model, SINQ with INT4 slightly outperforms SINQ with NF4.The method is compatible with non-uniform levels by replacing RoundToNearest with the NF4 quantizer.Source: §3.2 · p. 5

3.3. Calibrated Uniform Quantization

  • In several cases, uncalibrated SINQ outperforms calibrated baselines, while adding AWQ calibration brings further improvements.Table 4 compares 3-bit and 4-bit Qwen3 quantization.Source: §3.3 · p. 5

3.4. Inference Time

  • Figure 4 places SINQ within < 0.01ppl of the 4-bit Pareto front; its no-overhead form matches standard inference time, while a naive second scale adds 1.8% at batch size 1.The 1.8% overhead is reported for the most challenging setting.Source: §3.4 · pp. 5–6, Figure 4 · p. 6

3.5. Quantization Time

  • SINQ quantization averages 1.1× the runtime of RTN, compared with > 2× for HQQ and > 30× for AWQ.These runtimes were measured on identical hardware.Source: §3.5 · p. 6

3.6. Ablation Studies

  • Across Qwen3 Pareto fronts, both auxiliary-variable precisions work well, while using shifts appreciably improves the front.The authors use shifts and quantize the auxiliary variables in subsequent experiments.Source: Figure 5 · p. 6, §3.6 · p. 6

4. Related Work

Related work spans uncalibrated uniform and non-uniform quantization, calibration-based methods, and weight-space transformations. SINQ is positioned as a dual-scaling, matrix-balancing approach that remains uniform, calibration-free, and architecture agnostic.

4.1. Uncalibrated, Uniform Integer Quantization

  • Uncalibrated uniform methods include RTN, HQQ's optimized shifts, and Hadamard transforms that can normalize weight distributions.HQQ minimizes a p-norm error, usually with p = 0.7; Hadamard normalization is described by Tseng et al. (2024a).Source: §4.1 · p. 6

4.2. Non-Uniform Quantization

  • Non-uniform levels can reduce quantization error for non-uniform trained weights; NF4 uses normal-distribution quantiles, and SINQ is compatible with NF4.Hadamard transformations can reduce variation in optimal levels across layers, as used with non-uniform quantization in HIGGS.Source: §4.2 · p. 7

4.3. Calibration

4.4. Weight Space Transformations

  • Weight-space methods combine transformations with calibration or non-uniform levels; unlike rotated-model methods, SINQ uses dual-scaling and matrix balancing so layers can be treated independently.The paper describes this as supporting uniform, calibration-free quantization and generalization to new architectures.Source: §4.4 · p. 8

5. Conclusion

The conclusion presents SINQ as a calibration-free quantization method that uses weight structure to approximate activation awareness and can combine with calibrated or non-uniform methods.

5. Conclusion

  • Column-wise standard deviations of LLM weight matrices are predictive of input activations.A modified Sinkhorn iteration normalizes row and column standard deviations, balancing pseudo-activation-aware column scaling against row-wise kurtosis impact.Source: §5 · p. 8
  • Experiments show SINQ is fast and outperforms state-of-the-art methods for uniform quantization without calibration.SINQ can also be combined with widely used calibrated and/or non-uniform methods.Source: §5 · p. 8
  • The additional scale has negligible computational overhead at both the individual-layer level and in end-to-end measurements.Source: §5 · p. 8

A.2. Reproducibility

The reproducibility account describes the implementation and evaluation materials and the licensing of the code and models used.

A.2. Reproducibility

  • Supplementary materials include the code used to derive the LLM quantization results and a full implementation of SINQ.Source: §A.2 · p. 10
  • Perplexity evaluations use open-source code to ensure reproducible details such as context length.The code and external code it builds on are permissively licensed to facilitate follow-up research.Source: §A.2 · p. 10
  • Experiments use permissively licensed open-weight models to further promote reproducibility.Source: §A.2 · p. 10

A.3. Weight Standard Deviation Plots for Further Models

The reported plots show that column-wise weight standard deviation and input magnitude have high R² values in Llama2, Phi-2, and further Qwen3 models.

A.3. Weight Standard Deviation Plots for Further Models

  • Column-wise weight standard deviation and input magnitude have high R² values in Llama2, Phi-2, and further Qwen3 models.Source: §A.3 · p. 10

A.4. AWQ vs. ASINQ Row-Wise Kurtosis

On Qwen3-1.7B, applying SINQ before AWQ reduces average row-wise kurtosis, accompanying better quantized performance.

A.4. AWQ vs. ASINQ Row-Wise Kurtosis

  • 1.32× lower average row-wise kurtosis accompanies better quantized performance when SINQ precedes AWQ on Qwen3-1.7B.The authors attribute the performance improvement to this reduction.Source: §A.4 · p. 10

A.5. Results on Reasoning

Preliminary pass@1 findings on Qwen3-14B suggest the method sustains reasoning capabilities without increasing reasoning-trace length.

A.5. Results on Reasoning

  • Reasoning-trace length is included to check whether longer traces negate quantization’s efficiency gains.Source: §A.5 · pp. 10–11
  • The evaluation reports reasoning performance for Qwen3-14B with 4-bit weight-only post-training quantization.Source: §A.5 · p. 10, Table 7 · p. 11
  • Preliminary pass@1 findings suggest the method sustains reasoning capabilities without increasing reasoning-trace length.Source: §A.5 · p. 11, Table 7 · p. 11

A.6. No-Overhead variant

The appendix reports improved quality for no-overhead SINQ and extends its evaluations across formats, model families, timing, and alternative quantization methods.

A.6. No-Overhead variant

  • The no-overhead SINQ variant produces better-quality outputs than comparable prior methods on Qwen3 models with 4-bit weight-only uncalibrated uniform quantization.Source: §A.6 · p. 11, Table 8 · p. 12

A.7. Improving GGUF with no-overhead SINQ

  • No-overhead SINQ reduces perplexity over standard GGUF while preserving inference speedups on Q4_0 and Q3_KS.The evaluation reports WikiText-2 perplexity and prefill and decode throughput.Source: §A.7 · p. 11, Table 9 · p. 12

A.8. Timing Results

  • Quantization times are comparable to RTN, faster than HQQ in the reported comparison, and substantially faster for A-SINQ than GPTQ and AWQ.The timing results cover multiple models on a single GPU; the table reports average times across 10 runs for Qwen3 models.Source: §A.8 · p. 11, Figure 8 · p. 11, Table 10 · p. 13

A.9. Results on Llama Models; A.10. Results on DeepSeek-V3 and other large models; A.12. Results on Phi Models

  • Results on Llama, DeepSeek, and Phi models further validate SINQ’s effectiveness across these model families.The DeepSeek comparisons include DeepSeek-V3-685B, DeepSeek-V2.5-236B, and Qwen3-235B-A22B; the Phi results cover 3-bit and 4-bit quantization.Source: §A.9 · p. 11, §A.10 · p. 11, §A.12 · p. 11

A.11. Accuracy Results

  • The QA evaluation reports accuracy on HellaSwag, PIQA, and MMLU, while identifying flips as the more reliable quality metric for QA tasks.Source: §A.11 · p. 11, Table 14 · p. 15

A.13. Comparison to CrossQuant in W4A8 setting; A.14. Comparison to Code-Book-Based Methods

  • CrossQuant results use W4A8G128 rather than the main results’ W4A16G64 setting and are not directly comparable; code-book methods require non-standard kernels and are incompatible with activation quantization.The code-book comparison reports WikiText perplexity on Llama2 models.Source: §A.13 · p. 12, §A.14 · p. 12, Table 16 · p. 12

A.15. Further Comparison to HIGGS

A.16. Additional Results on MoE Models

  • On MoE models, SINQ outperforms state-of-the-art calibration-free weight-quantization methods in the reported perplexity results.The evaluation includes DeepSeek-V2-Lite and Qwen3-30B-A3B at 3-bit and 4-bit, with perplexity and actual memory usage reported.Source: §A.16 · p. 14, Table 19 · p. 16

Related papers

Loading 2509.22944v4…