PaperlayerView exact revision on arXiv

Source-linked AI summary

All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs

Zhixiong Zhao, Zukang Xu, Guangyu Sun, Lifeng Liu, Dawei Yang

arXiv:2609.06161v1cs.LGcs.AI

TL;DR

LLM deployment is constrained by storage and memory-bandwidth demands, while existing binary PTQ methods struggle with representational capacity and hidden structural overhead. AF1 combines NABF for Hessian- and null-space-aware binary reconstruction with HiSA for hierarchical capacity allocation, achieving strong accuracy under genuine 1-bit constraints and outperforming existing binary PTQ methods. Its scope is weight-only compression, so activations and KV-cache costs remain outside the method.

  • Problem

    LLM deployment faces substantial storage and memory-bandwidth demands, while existing binarization-based PTQ methods are limited by binary reconstruction capacity and heterogeneous component sensitivity.

  • Method

    AF1 combines NABF for Hessian-aware, null-space-aware binary reconstruction and scale-only refinement with HiSA for hierarchical structural-budget allocation.

  • Results

    AF1 achieves a superior accuracy–memory trade-off over existing binarization-based PTQ methods while maintaining a strict 1.0 effective-BPW budget for target linear weights.

  • Takeaways & Limitations

    AF1 provides a scalable and deployment-friendly pathway toward genuine 1-bit LLM compression with practical memory reduction and decoding acceleration.

  • Takeaways & Limitations

    AF1 compresses target linear weights but does not directly compress activations or the KV cache, and its efficiency evaluation uses a representative optimized GPU implementation.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable progress, yet their massive storage and memory-bandwidth demands still hinder efficient deployment. Weight binarization is a promising solution, but existing binarization-based post-training quantization (PTQ) methods usually far exceed the nominal 1-bit storage target due to hidden overhead. To address this gap, we propose All for 1-Bit (AF1), a genuine 1-bit PTQ framework for LLMs. AF1 comprises two complementary components: (1) Null-space-Aware Binary Factorization (NABF) for improving binary reconstruction through Hessian-aware surrogate reparameterization, null-space-aware binary factorization, and scale-only global reconstruction; and (2) Hierarchical Shapley Allocation (HiSA) for assigning structural capacity using hierarchical Shapley sensitivity. Together, they preserve model accuracy under a strict 1.0-BPW budget in the PTQ setting. Experiments on LLaMA, Qwen, and Gemma families show that AF1 consistently outperforms existing binarization-based PTQ methods in perplexity and zero-shot accuracy. Compared with BF16, AF1 achieves an average 2.5 times inference speedup and over 90% memory reduction across evaluated models, providing a practical path toward deployable genuine 1-bit compression for LLMs. The code for reproducibility is available at https://github.com/Kishon-zzx/AF1.

1 Introduction

AF1 targets genuine 1-bit PTQ for LLMs, motivated by deployment costs and limitations in existing binary reconstruction and structural allocation. It combines NABF and HiSA while maintaining a strict 1.0 effective-BPW budget.

  • Motivation: LLM scaling creates substantial storage and inference costs, motivating compression that preserves model performance.Models approaching 400B parameters may require about 800 GB for weights under BF16/FP16 alone.
  • Challenges: Existing binarization methods pursue single-bit storage but face limited representational capacity and stringent PTQ error-control requirements.A single binary matrix struggles to capture complex weight magnitudes and structural correlations.
  • Challenges: Uniform quantization configurations can misallocate capacity because layers and submodules have heterogeneous sensitivity to binarization errors.This mismatch can degrade full-model performance.
  • Approach: AF1 combines NABF for expressive, error-suppressed binary factorization with HiSA for hierarchical, interaction-aware structural budget allocation.The framework addresses both reconstruction capacity and non-uniform component sensitivity.
  • Result: AF1 maintains a strict 1.0 effective-BPW storage budget while achieving a superior accuracy–memory trade-off over existing binarization-based PTQ methods.The result is reported across LLM families in extensive experiments.

2 Related Work

Prior work reduces LLM cost through binarization and mixed-precision quantization, but commonly relies on local sensitivity, saliency, or activation statistics rather than layer–submodule interactions.

  • Binarization for LLMs: LLM binarization represents weights with binary values to reduce storage, memory bandwidth, and inference costs.Existing methods compensate reconstruction error using Hessian-aware compensation, weight distributions, residual approximation, structured sparsity, and refined binarization.
  • Mixed-Precision Quantization: Mixed-precision quantization assigns different bit-widths to components to improve the accuracy–efficiency trade-off.Prior methods use Hessian, attention, saliency, activation distributions, or AutoML search for allocation.
  • Research gap: Related low-bit methods often model local sensitivity, saliency, or activation statistics but rarely capture interactions between layers and submodules.This limitation motivates interaction-aware structural allocation.

3 Preliminaries

The preliminaries define null-space perturbations and Shapley values as the mathematical foundations for error compensation and structural-capacity allocation in AF1.

  • Null Space: A null-space perturbation is canceled by a linear mapping and therefore does not affect the corresponding matrix-multiplication output.The paper uses such perturbations to compensate discrete projection errors in binary factorization.
  • Shapley Value: The Shapley value measures a participant’s importance by averaging its marginal contribution across all possible coalitions.Participants may be layers, blocks, or submodules, while the value function measures reconstruction change under compression.

4 Method

AF1 combines NABF, which improves binary reconstruction through Hessian-aware and null-space-aware procedures, with HiSA, which allocates structural capacity hierarchically under a strict 1.0 effective-BPW budget.

  • NABF: NABF constructs a Hessian-aware surrogate space so reconstruction emphasizes task-sensitive input and output channels.The surrogate reparameterization converts Hessian-weighted reconstruction into a Frobenius objective.
  • NABF: NABF factorizes the surrogate matrix into two binary sign factors with continuous scaling vectors to increase representation capacity.The factorization is optimized using a continuous proxy and discrete SVID projection.
  • NABF: NABF guides projection-error compensation toward null-space directions so errors are largely canceled after multiplication by the fixed factor.The relevant propagated perturbation is EAB rather than EA alone.
  • NABF: Scale-only global reconstruction freezes binary sign matrices and recalibrates continuous scales using a calibration set and teacher-distribution matching.Trainable parameters per linear layer are reduced from O(doutr+rdin) to O(dout+r+din).
  • HiSA: HiSA addresses heterogeneous compression sensitivity by modeling global layer interactions and within-layer submodule differences under fixed storage constraints.It generates adaptive BPW configurations for NABF.
  • HiSA: HiSA estimates layer importance by progressively degrading intermediate dimensions and measuring induced token-level NLL changes.Monte Carlo permutations estimate Shapley scores because enumerating all capacity states is infeasible.

5 Experiments

AF1 is evaluated across multiple LLM families, benchmarks, ablations, hyperparameter settings, and efficiency comparisons. Results show strong genuine 1-bit accuracy, reconstruction, quantization-time, memory, and throughput performance.

  • Main Results: AF1 improves average zero-shot accuracy by over 20% and reduces average WikiText2 PPL by over 90% versus HBLLM across evaluated models.
  • Main Results: At 1.0 effective BPW for target linear weights, AF1 achieves 61.80 zero-shot accuracy and 8.91 WikiText2 PPL on LLaMA-3-70B.
  • Instruction-tuned Models: AF1 uses 2.92 GB versus HBLLM’s 7.43 GB on Qwen3-14B-Instruct, while averaging about 9% lower performance.
  • Ablation Studies: NABF reduces ablation PPL from 8.9e2 to 30.47 and raises average accuracy to 40.79, while HiSA further reaches 24.21 PPL and 43.09 accuracy.
  • Ablation Studies: HiSA performs best at τ = 1.0 and α = 0.2, with Shapley sampling stabilizing around 60 layer and 40 block samples.
  • Efficiency Analysis: AF1 reduces LLaMA-2-7B PPL from 16.44 for ARB-LLM to 11.20 using 5.9 GPU hours, and is about 40× faster than LittleBit and 228× faster than OneBit.
  • Efficiency Analysis: AF1 reduces memory and improves decoding throughput versus BF16 and GPTQ-W4 under batch size 1, 1024-token prefill, and 4096-token decoding.

6 Conclusion

AF1 is presented as a genuine 1-bit PTQ framework that addresses binary reconstruction and heterogeneous structural sensitivity without costly QAT. Its experiments report stronger binary-PTQ performance, practical memory reduction, and decoding acceleration, while its scope remains weight-only compression with broader deployment validation left open.

  • Conclusion: AF1 combines NABF for binary reconstruction with HiSA for allocating limited structural budgets across layers and submodules.NABF addresses reconstruction, while HiSA handles non-uniform structural sensitivity.
  • Conclusion: AF1 substantially outperforms existing binary PTQ methods under comparable or lower BPW and approaches QAT-based binarization accuracy with lower quantization cost.
  • Conclusion: AF1 provides practical memory reduction and decoding acceleration as a scalable, deployment-friendly pathway toward genuine 1-bit LLM compression.
  • Limitations: AF1 compresses target linear weights at 1.0 effective BPW but does not directly compress activations or the KV cache.These uncompressed components may matter in long-context or large-batch serving.
  • Limitations: The evaluation covers representative LLaMA, Qwen, and Gemma models and standard benchmarks, while broader architectures, instruction-tuned variants, and real workloads remain for future validation.
  • Ethical Considerations: More efficient deployment may lower barriers to misuse, so AF1-compressed models should retain the safety, monitoring, and access-control mechanisms of full-precision models.

Appendix Overview

The appendix collects detailed algorithm descriptions, proofs, effective-bit-width analysis, additional experiments, dialogue examples, and information about large-language-model use.

  • Appendix contents: Appendix Section A.1 documents AF1's detailed algorithms, including the overall workflow, NABF, HiSA, and ADMM-SVID framework.
  • Appendix contents: Appendix Section A.2 provides proofs covering binary parameterization capacity, null-space properties, residual-energy cutoff, and scale compensation.
  • Appendix contents: Appendix Sections A.3 and A.4 analyze effective bit-width and provide more experimental results, including detailed results and multiple ablations.The ablations cover residual-energy thresholds, allocation-sensitivity metrics, HiSA hyperparameters, and calibration.
  • Appendix contents: Appendix Sections A.5 and A.6 contain dialogue examples and discuss the use of large language models.

A.1 Detailed Algorithms of AF1

The appendix presents AF1's algorithmic workflow through three pseudocode blocks covering the complete PTQ pipeline, NABF, and HiSA.

  • A.1 Detailed Algorithms of AF1: The appendix provides the algorithmic workflow of AF1.
  • A.1 Detailed Algorithms of AF1: One pseudocode block corresponds to the overall PTQ pipeline.
  • A.1 Detailed Algorithms of AF1: Separate pseudocode blocks correspond to Null-space-Aware Binary Factorization and Hierarchical Shapley Allocation.

A.1.1 Overall Workflow of AF1

AF1 combines hierarchical structural allocation with operator-level binary reconstruction, using a pretrained model and calibration set to assign budgets, reconstruct operators, and perform scale-only global reconstruction.

  • A.1.1 Overall Workflow of AF1: Given a pretrained model and calibration set, AF1 first collects statistics required by HiSA and NABF.
  • A.1.1 Overall Workflow of AF1: HiSA determines layerwise and module-wise structural budgets, while NABF reconstructs each target linear operator under its assigned intermediate dimension.
  • A.1.1 Overall Workflow of AF1: After replacing all target operators, AF1 freezes the binary sign matrices and performs scale-only global reconstruction.

A.1.2 Null-space-Aware Binary Factorization Hessian-aware surrogate space.

NABF transforms Hessian-weighted reconstruction into a surrogate-space problem, then combines binary factorization, null-space compensation, and scale optimization within AF1’s PTQ pipeline.

  • NABF scales weights using input-activation and output-gradient statistics to preserve task-sensitive channel information in a Hessian-aware surrogate space.
  • NABF uses two binary sign matrices with continuous scale vectors, updating the continuous factors through alternating ADMM steps before SVID projection.
  • Null-space compensation folds discrete projection residuals into existing intermediate scaling vectors using an approximate projector derived from the fixed factor.
  • ColScaleFit and RowScaleFit provide column-wise and row-wise least-squares scaling for matching factor representations.
  • AF1 collects calibration statistics, applies HiSA allocations to target operators, freezes binary matrices, and optimizes only scale vectors with a KL objective.

A.1.3 Hierarchical Shapley Allocation

HiSA allocates structural capacity hierarchically by estimating sensitivity at the layer and submodule levels, then converts module budgets into NABF intermediate dimensions.

  • Layer-level allocation: HiSA estimates layer sensitivity by progressively degrading layers under sampled permutations and measuring end-to-end NLL changes.
  • Block-level allocation: HiSA assigns submodule capacity by separately measuring directional and magnitude output distortions, normalizing them by MAD, and fusing them with Soft-OR.
  • Block-level allocation: The fused submodule score becomes a size-aware density and then a module-wise budget.
  • Budget-to-rank conversion: AF1 converts each HiSA module budget into the intermediate dimension used by NABF, with dominant double-factor storage proportional to r_i(d_out + d_in) bits.

A.1.4 ADMM-SVID Framework in DBF

AF1 combines double-factor binary parameterization, null-space compensation, and hierarchical dual-distortion allocation to improve reconstruction and assign capacity under genuine low-bit budgets.

  • NABF: NABF uses ADMM-SVID as its discrete optimization framework and adds null-space-aware compensation to suppress projection residuals.The compensation exploits perturbation directions that do not affect the matrix output.
  • Representation capacity: Double-factor binary parameterization represents each connection as a sum of r binary paths with continuous path coefficients, enabling multiple effective magnitudes and correlation patterns.This expands capacity beyond the separable magnitude of a single-factor representation while keeping the dominant factors binary.
  • HiSA: HiSA allocates structural capacity using hierarchical Shapley sensitivity, accounting for interactions among compressed and uncompressed layers.Its layer-level scores reflect downstream propagation, prediction-level changes, and different structural coalitions.
  • HiSA: HiSA separates directional and magnitude distortions because a single MSE score can hide which representation failure mode dominates.The two terms measure angular deviation and representation-magnitude mismatch, respectively.
  • Storage budget: AF1 stores binary sign factors with lightweight BF16 scale vectors, while prior binary PTQ methods incur additional effective-storage overhead.The overhead in prior methods includes residual tensors, reconstruction coefficients, saliency masks, group bitmaps, and sparsity metadata.
  • Results: Under 1.00-bit budgets, AF1 remains competitive with or superior to several higher-bit baselines and improves robustness across tasks and allocation variants.HiSA achieves the lowest WikiText2 perplexity in the reported allocation comparisons, while dual-distortion fusion outperforms single-distortion variants.
Loading 2609.06161v1…