PaperlayerView exact revision on arXiv

Source-linked AI summary

QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization

Yipin Guo, Arun M George, Jie Fu, Tareq Mahmoud, Sixue Xing, Siddharth Joshi

arXiv:2609.00224v2cs.LGcs.AI

TL;DR

Sub-2-bit PTQ can reduce LLM serving costs but often loses accuracy and relies on irregular sparsity. QTEA combines ternary weights with sparse salient residuals, adaptive rescaling, and error decay, achieving stronger accuracy–perplexity results across model families and practical kernel speedups.

  • Problem

    Sub-2-bit PTQ often suffers severe accuracy degradation, while unstructured sparsity reduces regularity and GPU-friendly execution.

  • Method

    QTEA quantizes all weights into a ternary base, adds column-semi sparse residual compensation, refines column scales, and applies error decay during GPTQ-style quantization.

  • Results

    QTEA improves accuracy and reduces perplexity over prior sub-2-bit PTQ methods across Qwen3-14B and Llama3-8B, while its kernel achieves practical GPU speedups.

  • Takeaways & Limitations

    QTEA demonstrates a practical accuracy–compression trade-off for sub-2-bit LLM deployment, with acceleration dependent on the target inference system.

  • Takeaways & Limitations

    QTEA does not quantize activations or KV cache, so its benefits may be smaller when those costs or other system overheads dominate runtime.

Abstract

from arXiv · show

Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existing PTQ methods often fail to generalize across models and suffer severe accuracy loss below 2 bits. Many leverage unstructured sparsity to mitigate this loss, but at the cost of regularity and GPU-friendly execution. We present QTEA, a sub-2-bit PTQ framework that quantizes weights into ternary values and uses salient weights as residual error compensators. To maintain hardware efficiency, residuals are assigned to selected columns with semi-structured $1:4$ sparsity within the salient columns. We further add column-wise rescale refinement to GPTQ-style column-by-column quantization, alternately updating per-column scales and ternary assignments to reduce reconstruction error. We also identify order-dependent error propagation in GPTQ and introduce error decay to attenuate late-stage error accumulation. On Qwen3-14B, QTEA compresses all weights to an effective 1.7 bits per weight while improving average accuracy over the strongest ternary PTQ baseline by 16.7%. It also achieves 1.40$\times$ and 2.61$\times$ lower perplexity on WikiText and C4 respectively. This trend holds on Llama3-8B, where QTEA obtains a 6.6% accuracy gain and 1.34$\times$ / 1.95$\times$ lower perplexity on the same datasets. Finally, we develop a lookup-table based kernel that achieves 7.2$\times$ faster per-token generation over an FP16 baseline. Code is available at https://github.com/Intelligent-Microsystems-Lab/QTEA.

1 Introduction

QTEA targets the memory-bandwidth costs and accuracy degradation of sub-2-bit weight-only PTQ with a ternary representation, sparse residual compensation, adaptive refinement, and error decay. It reports improved accuracy, perplexity, and generation speed across Qwen3 and Llama3 models.

  • Motivation: A 70-billion-parameter model requires roughly 140 GB for BF16 weights, making low-batch decoding memory-bandwidth bound.Weight-only PTQ directly reduces the memory footprint and bandwidth pressure caused by repeatedly loading model weights.
  • QTEA: QTEA quantizes all weights into a ternary base and uses salient weights as residual error compensators instead of unquantized bypasses.Residuals are assigned to important columns, with a sparse subset of high-impact entries retained within those columns.
  • QTEA: Column-wise rescale refinement alternates per-column scale updates and ternary reassignment to better match local weight distributions during GPTQ-style quantization.The refinement addresses magnitude drift after Hessian-aware error propagation.
  • QTEA: Error decay attenuates propagation strength along the column order to reduce late-stage error accumulation caused by GPTQ’s order-dependent imbalance.Later columns have fewer remaining columns available to absorb propagated errors, making uniform propagation overly aggressive.
  • Results: 16.7% relative accuracy gain on Qwen3-14B accompanies 1.40× lower WikiText perplexity and 2.61× lower C4 perplexity versus the strongest ternary PTQ baseline.On Llama3-8B, QTEA reports a 6.6% relative accuracy gain and 1.34× / 1.95× lower WikiText and C4 perplexity.
  • Results: 7.2× faster per-token generation over FP16 with CUDA Graphs is enabled by QTEA’s efficient computation kernels.Compared with FP16 inference without CUDA Graphs, the reported speedup reaches 13.3×.

2 Related Work

Related work spans PTQ, QAT, and hardware-oriented methods for low-bit LLMs. QTEA is positioned as a retraining-free, sub-2-bit PTQ approach that addresses accuracy and regularity challenges.

  • Post-training quantization: GPTQ minimizes layer-wise reconstruction error using approximate second-order information, while AWQ and OmniQuant modify salient-channel scaling or quantization transformations.These methods represent established PTQ strategies for reducing LLM weight costs.
  • Ultra-low-bit QAT: Ultra-low-bit QAT methods improve quantizability through fine-tuning or retraining, including ternary and sparse ternary approaches.QTEA differs by recovering accuracy through adaptive ternary error compensation without retraining.
  • Ultra-low-bit PTQ: Sub-2-bit PTQ methods use rotations, salient-weight preservation, mixed precision, grouping, or ternarization to limit accuracy degradation.Many such methods rely on unstructured designs, motivating hardware-friendlier regularity in QTEA.

3 Quantized Ternary Error Adaptation

QTEA combines a ternary base with sparse residual compensation and adaptive GPTQ-style refinement to preserve accuracy while supporting efficient computation. Its kernel implements the representation through lookup-table computation and structured residual accumulation.

  • 3.2 Column-Semi Sparse Salient Weights: QTEA ranks salient columns by maximum individual weight impact, favoring the most damaging entry rather than average column importance.The score combines weight magnitude with Hessian-based sensitivity and is used only to order residual allocation.
  • 3.2 Column-Semi Sparse Salient Weights: Column-semi sparse residuals retain corrections only in selected salient columns with a 1:4 pattern, improving regularity for hardware execution.The residual structure avoids a fully unstructured mask.
  • 3.3 Column-Wise Rescale Refinement: Column-wise rescale refinement alternates per-column scale updates with ternary reassignment after Hessian-aware error propagation.The factor v_j corrects local magnitude drift while preserving shared group-wise scale and bias parameters.
  • 3.4 Error Decay: Error decay weakens GPTQ propagation along the quantization order to reduce late-stage over-compensation caused by shrinking remaining-column capacity.The damping uses column position and normalized inverse-Hessian information.
  • 3.5 Efficient Ternary Computation: The CUDA kernel avoids full dequantization by using lookup-table ternary computation and parallel semi-sparse residual accumulation.Column-wise rescaling is fused into LUT activation precomputation.

4 Experiments

Experiments evaluate QTEA across Qwen3 and Llama3 backbones using perplexity and zero-shot accuracy under a 1.7-bit weight budget. QTEA consistently outperforms competing sub-2-bit methods and maintains strong scaling across model families and sizes.

  • 4.2 Perplexity and Zero-shot Accuracy Results: 1.7 bits per weight: QTEA achieves the best average zero-shot accuracy among sub-2-bit methods across evaluated Qwen3 and Llama3 models.The evaluation uses block size 128 and compares perplexity on WikiText2 and C4 with seven downstream benchmarks.
  • 4.2 Perplexity and Zero-shot Accuracy Results: 16.7% relative gain: QTEA raises Qwen3-14B average downstream accuracy from 45.11% to 52.65% over the strongest baseline.WikiText2 perplexity falls from 16.48 to 11.78, while C4 perplexity falls from 68.13 to 26.14.
  • 4.2 Perplexity and Zero-shot Accuracy Results: 6.6% relative gain: QTEA improves Llama3-8B average zero-shot accuracy from 37.79% to 40.29% over the best competing sub-2-bit method.WikiText2 and C4 perplexity decrease by 1.34× and 1.95×, respectively.
  • 4.3 Scaling Across Models: Across OPT, Gemma3, Llama2, and Mistral, QTEA remains the most stable method and competitive with the strongest ternary PTQ baseline.It slightly outperforms PT2-LLM on Llama2-7B and remains competitive on larger models.
  • 4.3 Scaling Across Models: 2.29×: QTEA reduces Qwen3 perplexity versus PT2-LLM by up to this factor while remaining best from 0.6B to 14B parameters.Perplexity decreases smoothly from 63.58 on Qwen3-0.6B to 11.78 on Qwen3-14B.

4.4 Efficiency

QTEA’s GPU implementation delivers substantial end-to-end per-token latency reductions, while semi-sparse residual computation adds negligible overhead. Customized-hardware estimates also indicate latency and energy benefits.

  • 3.62× and 7.22× speedups reduce FP16 latency on Qwen3-14B and Llama2-70B with CUDA Graph, respectively.Latency falls from 23.75 to 6.56 ms/token on Qwen3-14B and from 41.13 to 5.70 ms/token on Llama2-70B.
  • QTEA matches the ternary-only kernel’s latency, with residual computation increasing latency by only 0.01 ms/token on Qwen3-14B and 0.03 ms/token on Llama2-70B.Measurements include fused column-wise rescaling and semi-sparse salient-residual computation.
  • 3.83× average latency speedup and 69.4% average energy reduction are estimated for the ternary-LUT engine over dense FP16 matrix multiplication on customized hardware.The estimates cover four evaluated matrix shapes and attribute benefits to LUT accumulation over compact ternary weights.

4.5 Ablation Study

Ablations support QTEA’s salient-weight structure, integrated rescale refinement, and error decay. The method is stable with only a few refinement iterations, while fixed decay settings perform well across model families.

  • Salient scoring: QTEA achieves the best average zero-shot accuracy on Qwen3-14B and Llama3-8B, favoring protection of extreme high-impact values over columns with large average importance.
  • Salient-residual structures: QTEA’s column 5% + 1:4 semi-sparse residual structure trails the global 1.5% unstructured upper bound by only 0.91 and 0.57 points on Qwen3-14B and Llama3-8B.It trades 0.31pp and 0.29pp of accuracy for 4× lower residual storage while retaining hardware-friendly access.
  • Rescale refinement: Rescale refinement is most effective inside GPTQ’s column-by-column quantization, while removing ternary reassignment reduces accuracy.The results support joint alternating optimization of rescale factors and ternary assignments.
  • Rescale refinement: Two refinement iterations provide the strongest overall performance on Qwen3-14B and Llama3-8B, with Qwen3-14B varying by only 0.29 points across one to four iterations.Additional iterations do not improve Llama3-8B, so two iterations are fixed across models.
  • Error decay: Disabling error decay consistently reduces average zero-shot accuracy, with a particularly large drop on Llama3-8B.
  • Error decay: Qwen3-14B remains stable across tested positive decay coefficients, while λdecay = 1 provides a balanced choice across both model families.No single coefficient dominates across all perplexity and zero-shot metrics.

5 Conclusion

QTEA is a sub-2-bit weight-only PTQ framework that combines ternary weights with column-semi-sparse FP8 residuals, rescale refinement, and error decay. Across model families, it improves the accuracy–compression trade-off and enables practical acceleration through a LUT-based kernel.

  • QTEA quantizes all weights into a ternary base and uses column-semi-sparse FP8 residuals to compensate harmful errors instead of bypassing quantization.
  • Column-wise rescale refinement inside GPTQ-style quantization and error decay improve the ternary representation and stabilize order-dependent error propagation.
  • Experiments across multiple model families show improved zero-shot accuracy, reduced perplexity, and a strong accuracy–compression trade-off over prior sub-2-bit PTQ methods.
  • A LUT-based kernel translates QTEA’s compact representation into GPU speedups and promising latency and energy reductions on customized hardware.

Limitations

QTEA’s reported benefits primarily target weight-memory bottlenecks in decoder-only, low-batch inference and depend on fixed settings and specialized hardware implementations. Customized-hardware results are architectural estimates rather than final silicon measurements, and deployment requires broader evaluation.

  • QTEA does not quantize activations or KV cache, so benefits may be smaller when activation computation, KV-cache traffic, or other overheads dominate runtime.
  • Evaluations remain limited to decoder-only language models, excluding instruction-tuned, mixture-of-experts, multimodal, and longer-context workloads.
  • Most experiments keep salient-column ratio, 1:4 residual sparsity, group size, and calibration settings fixed, although adapting them may improve the accuracy–efficiency trade-off.
  • QTEA’s practical speedup depends on specialized kernels, target hardware, memory hierarchy, runtime integration, and CUDA Graph support.
  • Customized-hardware results omit post-layout effects and workload-specific switching activity, so they indicate architectural potential rather than final silicon performance.
  • Quantized models should be re-evaluated on task-specific and safety-critical benchmarks because standard perplexity and zero-shot tests may miss behavior changes.

A Hardware Evaluation Details

The hardware evaluation models a ternary-LUT matrix-vector engine against a dense FP16 baseline, including compute, memory, latency, and energy costs. Across evaluated shapes, the design reduces latency and energy while relying on compact packed ternary weights and estimated hardware assumptions.

  • Evaluation setup: The evaluated workload is matrix-vector multiplication, comparing a ternary-LUT engine with dense FP16 matrix computation.The input has shape [1, K], the weight matrix [K, N], and the output [1, N].
  • Hardware assumptions: The ternary engine partitions each 128-column group into 26 five-column packs and uses packed ternary weights as lookup-table addresses.The design operates at a 1GHz target frequency and includes a pre-compute stage and half-LUT construction.
  • Compute and memory: The ternary compute logic uses about 80% of the dense baseline area before SRAM accesses.The reported areas are 28,454.25µ2 for ternary compute logic versus 35,573.98µ2 for the dense 26-lane baseline.
  • Compute and memory: The ternary engine replaces 128 FP16 weight reads per row-group with 26 packed 8-bit weight-address reads, equivalent to 13 16-bit SRAM reads.The latency model includes precomputation, LUT construction, and lookup/accumulation for each 128-column group.
  • End-to-end results: 3.83× average latency speedup and 69.4% average energy reduction are estimated across four matrix shapes versus the dense FP16 baseline.The benefit is attributed to LUT accumulation over compact ternary weights, despite added LUT scratch-SRAM traffic.
  • Storage: 1.6 bits per weight are required for packed ternary weights before scale and offset metadata, making storage about 9.85× smaller than dense FP16.Five ternary weights are packed into one 8-bit address for each 128-column group.
  • Limitations: The hardware results are architecture-level estimates that omit clock-tree power, routing parasitics, post-layout wire effects, and workload-specific switching activity.Compute power uses synthesized arithmetic units, while SRAM capacity and read/write energy are estimated separately from SRAM macro data.

B Model size.

The model-size comparison evaluates QTEA on Llama2-7B against prior low-bit quantization methods. QTEA substantially reduces checkpoint storage while operating below the size of compared 2-bit methods.

  • Model size: QTEA reduces the Llama2-7B checkpoint from 13.48 GB in FP16 to 2.12 GB, a 6.35× storage reduction.The setting uses approximately 1.7 bpw payload plus 0.3 bpw metadata, embeddings, LayerNorm, and LM-head storage.
  • Model size: QTEA achieves a smaller model size than the compared 2-bit methods GPTQ and Slim-LLM.The comparison uses baseline numbers taken from PT2-LLM.

C Matched-Budget Comparison

The matched-budget experiment tests whether QTEA’s accuracy gains arise only from extra high-precision salient-weight storage. QTEA remains substantially more accurate than an augmented PT2-LLM at the same additional memory footprint.

  • Matched-budget comparison: Adding FP8 column-wise salient weights improves PT2-LLM average zero-shot accuracy by 1.90 points on Qwen3-8B and 2.67 points on Qwen3-14B.The comparison uses an iso-memory footprint variant of PT2-LLM.
  • Matched-budget comparison: QTEA remains 6.08 points higher on Qwen3-8B and 4.87 points higher on Qwen3-14B than the matched-budget PT2-LLM variant.The reported comparison indicates that QTEA’s gain is not explained solely by the additional high-precision salient-weight budget.
Loading 2609.00224v2…