PaperlayerView exact revision on arXiv

Source-linked AI summary

NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models

Hyochan Chong, Dongkyu Kim, Changdong Kim, Minseop Choi

arXiv:2602.06694v3cs.LG

TL;DR

Existing quantization methods either cannot efficiently reach sub-1-bit storage or require substantial data, compute, or metadata. NANOQUANT addresses this with low-rank binary factorization, ADMM initialization, and reconstruction-based tuning, achieving compact 70B deployment on consumer hardware.

  • Problem

    Sub-1-bit PTQ remains an open challenge because binary PTQ has structural and metadata limitations, while binary QAT requires extensive data and computation.

  • Method

    NANOQUANT uses low-rank binary factorization with Hessian-aware ADMM initialization, block reconstruction, and model-level calibration.

  • Results

    NANOQUANT compresses a 70B LLM from 137.95 GB to 5.75 GB and runs it on a consumer 8 GB GPU at up to 20.11 tokens per second.

  • Takeaways & Limitations

    NANOQUANT establishes a low-memory PTQ Pareto frontier and lowers the hardware barrier for some large-model inference.

  • Takeaways & Limitations

    Further work is needed to outperform higher-bit 2- or 3-bit PTQ performance in the sub-binary regime.

Abstract

from arXiv · show

Weight-only quantization has become a standard approach for efficiently serving large language models (LLMs). However, existing methods fail to efficiently compress models to binary (1-bit) levels, as they either require large amounts of data and compute or incur additional storage. In this work, we propose NanoQuant, the first post-training quantization (PTQ) method to compress LLMs to both binary and sub-1-bit levels. NanoQuant formulates quantization as a low-rank binary factorization problem, and compresses full-precision weights to low-rank binary matrices and scales. Specifically, it utilizes an efficient alternating direction method of multipliers (ADMM) solver to precisely initialize latent binary matrices and scales, and then tunes the initialized parameters through a block and model reconstruction process. Consequently, NanoQuant establishes a new Pareto frontier in low-memory post-training quantization, and enables sub-1-bit compression. NanoQuant makes large-scale deployment feasible on consumer hardware. For example, it compresses Llama2-70B by 25.8$\times$ in just 13 hours on a single H100, enabling a 70B model to operate on a consumer 8 GB GPU. Code is available at https://github.com/SamsungLabs/NanoQuant.

1. Introduction

NANOQUANT addresses the difficulty of data- and compute-efficient sub-1-bit PTQ by combining low-rank binary representations with precise initialization and reconstruction. It achieves compact storage and competitive performance using limited calibration resources.

  • 1. Introduction: Existing binary PTQ methods are structurally bounded near 1 bit per parameter and incur metadata overhead that can exceed 2 or 3 effective bits.They use in-place binarization with full-precision scales and grouping metadata.
  • 1. Introduction: Binary QAT reaches 1-bit and sub-1-bit compression with low-rank representations but requires hundreds of millions or billions of tokens, multiple GPUs, and multiple days.These resource demands limit application to larger models such as 70B-parameter LLMs.
  • 1. Introduction: NANOQUANT uses Hessian-aware ADMM initialization followed by block reconstruction and model-level calibration.The method uses only 128 calibration samples, equivalent to 0.26M tokens, and 1 GPU.
  • 1. Introduction: 137.95 GB to 5.75 GB: NANOQUANT compresses a 70B LLM and runs it on a consumer 8 GB GPU at up to 20.11 tokens per second.The reported compression uses 1 GPU.
  • 1. Introduction: NANOQUANT is a PTQ method that compresses LLMs to both 1-bit and sub-1-bit levels.It addresses structural limitations in existing binary quantization frameworks.
  • 1. Introduction: Experiments across model families and language tasks show competitive performance with higher-bit PTQ and binary QAT methods despite limited calibration data.Custom binary CUDA kernels target higher throughput, smaller memory footprints, and improved energy efficiency.

2. Related Work

Prior work spans binary PTQ, low-rank binary QAT, and ADMM-based compression. PTQ remains constrained by storage overhead, while QAT reaches sub-1-bit levels at substantially higher data and compute costs.

  • 2. Related Work: Binary PTQ commonly combines sign-based binarization with full-precision scales, while some methods add sparsity to reduce memory.These approaches still incur storage for scaling factors and grouping bit-masks.
  • 2. Related Work: Low-rank binary QAT methods reach binary and sub-1-bit compression through end-to-end training on larger datasets.Their resource demands limit practicality for constrained deployment and larger models.
  • 2. Related Work: ADMM alternates augmented-Lagrangian subproblems with dual-variable updates and supports compression problems combining continuous reconstruction with hard structural constraints.NANOQUANT uses ADMM to decouple weight reconstruction from discrete low-rank binary structure.

3. NANOQUANT

NANOQUANT formulates sub-1-bit compression as low-rank binary factorization and integrates initialization, block reconstruction, refinement, and packing into a sequential pipeline. Its solver and calibration procedures target stable, efficient optimization under limited PTQ budgets.

  • 3.1. Quantization Scheme: NANOQUANT decomposes each dense weight matrix into two low-rank binary matrices and two full-precision channel-scale vectors.The binary matrices contain values from {−1, +1}.
  • 3.1. Quantization Scheme: The compression scheme factorizes continuous latent factors, binarizes them, and packs −1 and +1 values into bits for memory efficiency.Packing maps −1 to 0 and +1 to 1.
  • 3.2. Block Reconstruction Pipeline: NANOQUANT integrates initialization within sequential block reconstruction rather than treating it as separate preprocessing.Each block applies error mitigation, LB-ADMM initialization with magnitude balancing, and factorized component refinement.
  • 3.2. Block Reconstruction Pipeline: Hessian-aware preconditioning uses activation and gradient statistics, with shrinkage regularization to balance curvature information against calibration noise.Smaller γ values are reported for Llama and Qwen, while larger values are reported for Gemma 3 and Rnj-1.
  • 3.2. Block Reconstruction Pipeline: LB-ADMM alternates continuous-factor, auxiliary-proxy, and dual-variable updates while using ridge regularization to handle the nonconvex factorization.Stabilized Cholesky reduces the stated complexity to O(r^3/3) from O(2r^3/3) for general LU factorization.
  • 3.2. Block Reconstruction Pipeline: Factorized component refinement jointly tunes latent proxies and scales with STE, then fixes binary factors by their signs and packs the resulting weights.The refinement aligns the compressed block with full-precision block outputs.

4. Experiments

Experiments evaluate NANOQUANT across model families, quantization baselines, accuracy tasks, storage efficiency, inference hardware, and ablations. Results show competitive accuracy and resource-efficient sub-1-bit compression, with initialization and reconstruction contributing to performance.

  • Experimental Setup: Evaluations cover Llama-2, Llama-3, Gemma 3, Qwen3, and Rnj-1 models ranging from 0.6B to 70B parameters.Calibration used 128 WikiText-2 samples with sequence length 2048, and evaluation included perplexity and six zero-shot commonsense tasks.
  • Accuracy Analysis: NANOQUANT maintains functional perplexity across diverse model families while using fewer bits than competing binary PTQ methods.Table 2 evaluates 1-bit and sub-1-bit methods across five model families.
  • Accuracy Analysis: NANOQUANT achieves accuracy competitive with higher-bit binary PTQ baselines and approaches binary QAT performance on zero-shot commonsense reasoning.The comparison uses Llama-3 and Qwen3 models; the method uses orders of magnitude less data and compute than typical QAT approaches.
  • Compression vs. Model Size: 2.88 and 4.13 bits per weight are required by BiLLM and STBLLM, respectively, versus 2.28 bits for GPTQ (W2g64), while NANOQUANT achieves genuine 1-bit and sub-1-bit storage.NANOQUANT minimizes metadata overhead and maintains perplexity competitive with resource-intensive QAT methods.
  • Ablation Studies: NANOQUANT establishes a new low-memory Pareto frontier and consistently outperforms previous binary PTQ baselines across the Qwen3 family.The authors identify low-rank binary representations as a viable alternative to low-bit integer quantization for memory-critical applications.

5. Conclusion

NANOQUANT enables 1-bit and sub-1-bit weight quantization for LLMs up to 70B parameters using a single GPU, reducing memory requirements for large-model inference.

  • NANOQUANT enables 1-bit and sub-1-bit weight quantization for LLMs of up to 70B parameters using only a single GPU.
  • Up to 24× model compression makes it feasible to run a 70B-parameter LLM on an 8 GB GPU.
  • Custom binary CUDA kernels improve energy efficiency and decoding speed during inference.
  • NANOQUANT lowers the hardware barrier for some large-model inference in resource-constrained settings.

Impact Statement

NANOQUANT is presented as a sub-1-bit PTQ algorithm that enables massive LLMs such as Llama-2-70B to run on consumer hardware and edge devices.

  • NANOQUANT enables deployment of massive models such as Llama-2-70B on a single 8 GB GPU.
  • The method lowers the barrier to advanced AI research and applications for individuals and institutions with limited computational resources.
  • NANOQUANT contributes to the democratization of AI by expanding access to state-of-the-art LLMs.

A. Theoretical Analysis: Magnitude Balancing

NANOQUANT resolves scale ambiguity in low-rank factorization by balancing latent factors before extracting channel-wise scales, improving numerical stability without changing the reconstructed weight matrix.

  • Scale ambiguity: Low-rank factorization W ≈ UV⊤ is scale-invariant, allowing infinitely many equivalent latent-factor rescalings.
  • Magnitude balancing: Magnitude balancing selects equivalent factors with comparable Frobenius norms before channel-wise scale extraction.
  • Scale extraction: The continuous latent variables are decomposed into channel-wise scales and binary directions, with s1 and s2 capturing output- and input-channel magnitudes.
  • Magnitude balancing: The balanced representative minimizes total factor magnitude among equivalent rescalings.
  • Scale extraction: Balancing prevents one factor from absorbing most magnitude, reducing sensitivity to numerical range and rounding effects during scale extraction.
  • Numerical stability: Magnitude balancing avoids extreme factor regimes and provides a better-conditioned parameterization for scale extraction and straight-through-estimator refinement.

B. Stability Analysis of Low-Rank Binary Initialization

The stability analysis supports LB-ADMM as an efficient initialization procedure: robust preconditioning controls target scale, while regularized continuous updates are unique SPD solves, without implying global convergence.

  • Scope of analysis: Because the discrete factorization is nonconvex, LB-ADMM is used as a stable initializer rather than a globally convergent solver.
  • Initialization formulation: LB-ADMM initializes a rank-R factorization W ≈ UV⊤ using auxiliary variables for structured binary proxy constraints.
  • Spectral control: Robust diagonal preconditioning uses clipping and shrinkage to limit the influence of outlier calibration statistics.
  • Spectral control: The preconditioned target has controlled spectral scale, helping avoid extreme numerical ranges in subsequent linear solves.
  • Proxy updates: The SVID step maps continuous factors into structured sign-value proxies for latent binary initialization.
  • Continuous updates: The continuous U and V update matrices are symmetric positive definite, yielding strongly convex quadratic subproblems with unique solutions.
  • Continuous updates: The ADMM penalty and ridge regularization improve worst-case conditioning through a shared positive diagonal shift.
  • Refinement: Magnitude balancing and block reconstruction subsequently refine the initialized factors using calibration data.

C. Implementation Details

NanoQuant’s experiments use unified compression settings on one H100 and examine how calibration budget and composition affect reconstruction and zero-shot performance.

  • Experimental Setup: All NanoQuant compression experiments ran on one NVIDIA H100 (80 GB) with unified hyperparameters.The reported tuning uses learning rates of 1e-4, 1e-5, and 1e-6 across reconstruction stages.
  • Calibration Budget: More calibration data during block reconstruction produces greater performance gains.This result is reported for Llama-2-7B across different reconstruction data budgets.
  • Calibration Composition: Increasing calibration-data diversity mitigates overfitting and can improve zero-shot performance.The composition study varies C4 and WikiText-2 samples while keeping the total calibration set fixed at 128.

D.3. Analysis of Latent Weight Dynamics

Latent-weight analysis shows that refinement usually preserves LB-ADMM signs but selectively adjusts near-zero boundary weights, while ADMM settings trade speed against reconstruction quality.

  • Stability of Initialization: 0.47%–6.82% sign-flip ratios show that most latent weights retain their LB-ADMM signs during refinement.The range spans k_proj and gate_proj layers, respectively.
  • Refinement of Boundary Weights: Near-zero initial magnitudes receive the largest updates and are most likely to flip sign during refinement.These weights lie near decision boundaries, so refinement selectively modifies ambiguous latent factors.
  • Refinement of Boundary Weights: Targeted boundary adjustments compensate for discretization errors introduced during initial factorization.The refinement phase therefore acts as a margin-maximization process.
  • ADMM Ablations: Reducing ADMM outer iterations accelerates convergence but increases final reconstruction error.The ablation also evaluates penalty scheduling as a separate optimization choice.

E.1. Kernel Benchmarking Details

NanoQuant evaluates custom binary GEMV and GEMM kernels across high-end, consumer, and edge GPUs using throughput, memory, and decoding comparisons against established baselines.

  • Hardware and Protocol: The benchmark varies output-token counts for batch-1 GEMV and batch sizes for GEMM inference.The evaluation fixes input tokens at 128 and uses torch.compile, StaticCache, and CUDA 12.4.
  • Hardware and Protocol: Four GPUs spanning high-end, consumer, and edge devices are used to test custom binary CUDA-kernel limits.The Jetson TX2 lacks NVIDIA Tensor Cores, providing an edge-device case.
  • Kernel Design: The GEMV kernel uses two-stage low-rank binary matrix–vector multiplication with signs packed as one bit per weight.It retains FP16/BF16 tensor arithmetic while reducing memory traffic through bit packing.
  • GEMV Results: 12.2× speedup in inference throughput is achieved over PyTorch FP16 on an NVIDIA Jetson TX2.The result is attributed to NanoQuant’s extreme compression capability.
  • Kernel Design: Batched GEMM targets compute-bound serving by overlapping data movement and computation to keep tensor-core units busy.The implementation packs binary values into 32-bit words and performs fast dequantization with FP16/BF16 computation.
  • Cross-Kernel Results: Custom binary GEMV kernels outperform both BF16 and GemLite in decoding operations without NVIDIA Tensor Cores.The comparisons are reported in Figures 12 and 13.

F. Detailed Model Size Analysis of Binary Weight Quantization Methods

The storage analysis counts binary factors, FP16 scales, and auxiliary metadata to compare effective bits per weight across binary PTQ, QAT, and NanoQuant representations.

  • Unified Storage Metric: Storage accounting includes binarized weights or factors, FP16 reconstruction coefficients, and auxiliary flags or bitmaps.This provides a common memory basis across binary PTQ and low-rank binary QAT methods.
  • Unified Storage Metric: 1 BPW represents direct ±1 binarization of a full-precision matrix without additional metadata.The example derives the ratio from 1-bit quantized weights versus 16-bit full-precision weights.
  • Binary PTQ Methods: Binary PTQ baselines use salient-column handling, grouping, sparsity, scales, means, and metadata that contribute to storage overhead.The formulas explicitly account for these components across BiLLM, STBLLM, ARB-LLM, and HBLLM variants.
  • Binary QAT Methods: Low-rank binary QAT methods store two 1-bit factor matrices and FP16 scales, with DBF requiring r(n + m) + 16(n + r + m) bits.The corresponding BPW expression normalizes this storage by mn full-precision weights.
  • NanoQuant: NanoQuant removes the rank-wise scale from the factorization and uses a two-scale system.This scalar-overhead reduction yields lower BPW than DBF at the same rank r.
  • Model-Level Comparison: Model-level BPW sums layer storage across all decoder-block linear layers and divides by the total number of full-precision weights.Baseline bounds use at most 50 salient columns and block size k = 128.
Loading 2602.06694v3…