PaperlayerView exact revision on arXiv

Source-linked AI summary

SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs

Ruixuan Huang, Hao Zeng, Hantao Huang, Jinyuan Shi, Minghui Yu, Ian En-Hsu Yen, Shuai Wang

arXiv:2512.05409v1cs.CL

TL;DR

Existing low-bit quantization and sparsification face an accuracy-efficiency gap, compounded by limited hardware support for hybrid precision. The paper proposes SQ-format, which combines sparse high-precision values with low-precision computation and static activation quantization. It reports higher throughput than W4A8 baselines with less than a 1% model performance gap, and presents hardware designs for current and future accelerators.

  • Problem

    Low-bit quantization and sparsification struggle to balance accuracy and efficiency, while GPUs lack native support for hybrid-precision operations.

    Source: §1 · pp. 1–2

  • Method

    SQ-format combines sparse high-precision and low-precision values, with a static activation strategy that selects components using calibration data.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    SQ-format delivers higher throughput than W4A8 baselines with a model performance gap of less than 1%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents SQ-format as a practical approach for accelerating LLMs on current hardware and a co-design blueprint for next-generation accelerators.

    Source: §6 · p. 11

    Under the assumption that W8A8 has four times W4A4's computational power, sparsity needs to be at least 0.75 to balance computation.

    Source: §5.2 · p. 10

Abstract

from arXiv · show

Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.

1 Introduction

Existing low-bit quantization and sparsification struggle to balance accuracy and efficiency, in part because GPUs lack native hybrid-precision operations. The paper proposes SQ-format and reports accuracy-throughput gains alongside a static activation strategy.

1 Introduction

  • GPUs lack native support for hybrid-precision operations, so W4A8 schemes may use higher-precision paths that erase their potential speed gains.W4A4 can also significantly degrade model performance compared with W8A8.Source: §1 · pp. 1–2
  • SQ-format combines sparsification and quantization, preserving critical values at higher precision while compressing most values to lower precision.The proposal targets hardware-friendly computation that can shift work toward faster 4-bit tasks.Source: §1 · p. 2

Achieving Pareto Improvement between Accuracy & Throughput

  • SQ-format delivers higher throughput than W4A8 baselines with a model performance gap of less than 1%.The paper reports W4A4-level throughput with near-W4A8-level accuracy.Source: §1 · p. 2

Quantization for Static Activation Splitting

  • Static activation splitting selects high-precision components using calibration-set activation-weight contributions, avoiding costly runtime selection while matching dynamic-strategy performance.The paper also presents SQ-format as a co-design blueprint for future accelerators.Source: §1 · p. 2

2 Related Works

Related work covers PTQ methods for compressing LLMs and data formats that trade precision, range, and hardware support. The section identifies a limitation of applying one precision scheme uniformly to non-uniform information.

Post-Training Quantization

  • GPTQ [10] quantizes weights column by column; AWQ [9] and Wanda [16] account for activation interaction, while SpQR [8] and QUIK [17] isolate outliers but face hardware-efficiency challenges.SmoothQuant [4] transfers activation-quantization difficulty to weights, and SparseGPT [7] formulates pruning as layer-wise sparse regression.Source: §2 · p. 2

Data Representations for Quantization

  • INT8 [18] benefits from broad hardware support and throughput, FP8 [19] captures activation outliers, and NVFP4 uses microblock scaling for 4-bit compression on Blackwell.MX formats [20] adapt to local data ranges through shared scaling.Source: §2 · p. 2
  • HiFloat8 [21] uses tapered precision, but current formats apply one precision scheme to all values despite LLMs' non-uniform information.Source: §2 · p. 3

3 Methodology

The methodology defines a bank-based hybrid-precision format, gives example weight and activation algorithms, and describes hardware paths for using SQ-format. Its design pairs compact high-precision values with lower-precision computation and supports static activation masks.

3.1 SQ-format Definition

  • SQ-format divides one matrix-multiplication operand into banked high- and low-precision components, with fixed bank size and sparsity to avoid load imbalance and distributed accumulators.Its framework includes quantized values, scaling, masks, precision levels, bank size, and sparsity.Source: §3.1 · p. 3

3.2 Example Algorithms with SQ-Format

  • The example algorithms assign SQ-format to one operand and classic quantization to the other.Source: §3.2 · p. 3

3.2.1 SQ-format on Weights

  • For weights, the algorithm ranks values within each bank using weight magnitude and model sensitivity, then selects the top (1 −s) fraction per column for high precision.The weight example stores high-precision values compactly and marks their corresponding low-precision positions with a reserved value.Source: §3.2.1 · p. 3, Figure 1 · p. 4

3.2.2 SQ-format on Activations

  • For activations, static masks select important channels from calibration-set average activation-weight contributions, and weight-column reordering can improve hardware data locality.The method reports a 5.94 MB mask for Llama-3-70B; dynamic activation selection otherwise adds runtime TopK overhead.Source: §3.2.2 · p. 4

3.3 Hardware Design

  • Weight computation uses parallel precision paths, while static activation computation uses precomputed masks and two precision paths; GPU implementation throughput exceeds W8A8 and rises with sparsity.The high-precision weight path gathers corresponding activation values; dynamic masks could be computed by a pipelined hardware unit.Source: §3.3 · pp. 5–6, Figure 2 · p. 5, Figure 3 · p. 6
  • RTL synthesis using TSMC 12nm reports 35.8% less silicon area than a standard integer MAC array under 12-bit I/O, including the dynamic-masking gather unit.Source: §3.3 · p. 6

4 Experiments & Findings

Experiments compare SQ-format with quantization baselines across several LLMs and benchmarks, reporting accuracy-throughput gains and similar accuracy for static and dynamic activation strategies. Results also show that sparse high-precision components can improve W4A4 accuracy.

4.1 Experimental Setups

  • Experiments cover Llama-3-8B, Llama-3-70B, and Qwen3-30B-A3B, using six non-generative datasets, two generative datasets, and two perplexity benchmarks.Calibration uses 32 randomly sampled Wikitext segments, each 2048 tokens; baselines include W4A8, W4A4, GPTQ, SmoothQuant, and SpinQuant.Source: §4.1 · pp. 6–7

4.2 Finding 1: SQ-Format Achieves Accuracy-Throughput Pareto Improvement

  • W4A(SQ6) and W4A(SQ5) match GPTQ in average accuracy and perplexity on Llama-3; on Qwen-3, average benchmark accuracy improves by 3.87% with better perplexity.Source: Table 1 · p. 7, §4.2 · p. 8
  • For Llama-3-70B, SQ-format achieves a 1.71× speedup in prefilling latency measurements using the static activation strategy, capturing ∼89% of theoretical W4A4 acceleration while maintaining higher model performance.Source: §4.2 · p. 8
  • W(SQ)A4 improves average accuracy over W4A4 by 0.11% to 3.54%; at 4x sparsity, W(SQ6)A4 approaches W4A8 accuracy.The W(SQ)A4 results use GPU simulation and do not include latency comparisons because dedicated hardware support is required.Source: §4.2 · p. 8

4.3 Finding 2: SQ-Format Supports Static Activation Quantization

  • Static activation SQ-format retains most dynamic-strategy performance, with average benchmark accuracy varying within ±1%.The static strategy uses activation-weight product contributions from a calibration set; accuracy trends remain relatively stable across calibration-set sizes.Source: §4.3 · pp. 8–9

5 Hardware-Algorithm Co-design & Design Exploration

The design exploration studies bank size, sparsity, and precision choices to guide SQ-format’s hardware-algorithm co-design. Its results identify bank-size requirements and trade-offs for lower low-precision bit-widths.

5.1 Bank Size & Sparsity

  • At s = 3/4, bank_size = 32 separates most important activations, whereas bank_size = 4 leaves per-channel outliers in the low-precision part.The example is from Layer 30 of Llama-3-8B.Source: §5.1 · p. 9, Figure 6 · p. 10
  • For weights, optimal bank_size increases with sparsity; supporting at least 16x sparsity requires bank_size of at least 64, while activation hardware should support bank_size 64 or lower.Source: §5.1 · p. 10, Figure 7 · p. 10, Figure 8 · p. 11

5.2 High/Low Precision Choice & Sparsity

  • With hlow = INT2, weight SQ-format generally struggles to maintain accuracy, while dynamic activation SQ-format performs better and supports up to 4x sparsity.For weights, only B −(8/2) −0.5 achieves usable performance; hlow = INT3 shows weight-performance trends similar to (8/4).Source: §5.2 · p. 10
  • Weight SQ-format requires at least 4x sparsity under the assumed fourfold W8A8-to-W4A4 compute-power ratio; lower low-precision bits require still higher minimum sparsity.Source: §5.2 · pp. 10–11

6 Conclusion

SQ-format sparsely mixes high-precision elements into low-precision computation in a hardware-friendly manner, achieving a Pareto improvement between precision and throughput. The paper also presents static activation quantization and a co-design blueprint for future AI accelerators.

6 Conclusion

  • SQ-format achieves a Pareto improvement between precision and throughput by sparsely mixing high-precision elements into low-precision computation.The mixing is described as hardware-friendly.Source: §6 · p. 11
  • The paper presents a static activation quantization algorithm and shows its potential for hardware deployment.Source: §6 · p. 11
  • The authors position SQ-format as a practical solution for accelerating LLMs on current hardware and a blueprint for co-designing next-generation AI accelerators.Source: §6 · p. 11

A.1 Complete Results for Performance Comparison

The appendix provides performance-comparison tables for several model and quantization settings and reports that performance is relatively stable across calibration-set sizes.

A.1 Complete Results for Performance Comparison

A.2 Impact Calibrate Set Size

  • Performance is relatively stable across different calibration-set sizes, indicating that activation outlier distributions can be easily modeled.Source: §A.2 · p. 18, Table 7 · p. 18

B FP Quantization Results

On DeepSeek-R1, SQ-format uses FP8 activations with FP8/FP4 weight quantization and an effective 5-bit width. It maintains near-lossless performance against BF16 across generative and non-generative benchmarks.

B FP Quantization Results

C Hardware Synthesis Analysis

The hardware synthesis analysis evaluates an SQ-format unit for 4x sparse weights and INT8 activations against an INT6 MAC baseline. The authors characterize the design as physically efficient for increasing compute density.

C Hardware Synthesis Analysis

D End-to-End Latency Analysis

The analysis profiles SQ-format’s end-to-end prefilling latency and effective bandwidth on GPUs. It reports that SQ-format significantly outperforms W4A8 while moving toward theoretical W4A4 performance and maintaining accuracy.

D End-to-End Latency Analysis

Loading 2512.05409v1…