All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs

Motivation

• LLM scaling creates substantial storage and inference costs, motivating compression that preserves model performance.Models approaching 400B parameters may require about 800 GB for weights under BF16/FP16 alone.Source: §1 · pp. 1–2

Challenges

• Existing binarization methods pursue single-bit storage but face limited representational capacity and stringent PTQ error-control requirements.A single binary matrix struggles to capture complex weight magnitudes and structural correlations.Source: §1 · p. 2
• Uniform quantization configurations can misallocate capacity because layers and submodules have heterogeneous sensitivity to binarization errors.This mismatch can degrade full-model performance.Source: §1 · p. 2

Approach

• AF1 combines NABF for expressive, error-suppressed binary factorization with HiSA for hierarchical, interaction-aware structural budget allocation.The framework addresses both reconstruction capacity and non-uniform component sensitivity.Source: §1 · pp. 2–3

Result

• AF1 maintains a strict 1.0 effective-BPW storage budget while achieving a superior accuracy–memory trade-off over existing binarization-based PTQ methods.The result is reported across LLM families in extensive experiments.Source: §1 · p. 3

Binarization for LLMs

• LLM binarization represents weights with binary values to reduce storage, memory bandwidth, and inference costs.Existing methods compensate reconstruction error using Hessian-aware compensation, weight distributions, residual approximation, structured sparsity, and refined binarization.Source: §2 · p. 3

Mixed-Precision Quantization

• Mixed-precision quantization assigns different bit-widths to components to improve the accuracy–efficiency trade-off.Prior methods use Hessian, attention, saliency, activation distributions, or AutoML search for allocation.Source: §2 · p. 3

Research gap

• Related low-bit methods often model local sensitivity, saliency, or activation statistics but rarely capture interactions between layers and submodules.This limitation motivates interaction-aware structural allocation.Source: §2 · p. 3

Null Space

• A null-space perturbation is canceled by a linear mapping and therefore does not affect the corresponding matrix-multiplication output.The paper uses such perturbations to compensate discrete projection errors in binary factorization.Source: §3 · p. 3

Shapley Value

• The Shapley value measures a participant’s importance by averaging its marginal contribution across all possible coalitions.Participants may be layers, blocks, or submodules, while the value function measures reconstruction change under compression.Source: §3 · p. 3

NABF

• NABF constructs a Hessian-aware surrogate space so reconstruction emphasizes task-sensitive input and output channels.The surrogate reparameterization converts Hessian-weighted reconstruction into a Frobenius objective.Source: §4.1 · p. 4
• NABF factorizes the surrogate matrix into two binary sign factors with continuous scaling vectors to increase representation capacity.The factorization is optimized using a continuous proxy and discrete SVID projection.Source: §4.1 · p. 4
• NABF guides projection-error compensation toward null-space directions so errors are largely canceled after multiplication by the fixed factor.The relevant propagated perturbation is EAB rather than EA alone.Source: §4.1 · pp. 4–5
• Scale-only global reconstruction freezes binary sign matrices and recalibrates continuous scales using a calibration set and teacher-distribution matching.Trainable parameters per linear layer are reduced from O(doutr+rdin) to O(dout+r+din).Source: §4.1 · p. 5

HiSA

• HiSA addresses heterogeneous compression sensitivity by modeling global layer interactions and within-layer submodule differences under fixed storage constraints.It generates adaptive BPW configurations for NABF.Source: §4.2 · p. 5
• HiSA estimates layer importance by progressively degrading intermediate dimensions and measuring induced token-level NLL changes.Monte Carlo permutations estimate Shapley scores because enumerating all capacity states is infeasible.Source: §4.2 · pp. 5–6

Main Results

• AF1 improves average zero-shot accuracy by over 20% and reduces average WikiText2 PPL by over 90% versus HBLLM across evaluated models.Source: §5.2 · p. 7
• At 1.0 effective BPW for target linear weights, AF1 achieves 61.80 zero-shot accuracy and 8.91 WikiText2 PPL on LLaMA-3-70B.Source: §5.2 · p. 7

Instruction-tuned Models

• AF1 uses 2.92 GB versus HBLLM’s 7.43 GB on Qwen3-14B-Instruct, while averaging about 9% lower performance.Source: §5.2 · p. 8

Ablation Studies

• NABF reduces ablation PPL from 8.9e2 to 30.47 and raises average accuracy to 40.79, while HiSA further reaches 24.21 PPL and 43.09 accuracy.Source: §5.3 · p. 8
• HiSA performs best at τ = 1.0 and α = 0.2, with Shapley sampling stabilizing around 60 layer and 40 block samples.Source: §5.3 · p. 8

Efficiency Analysis

• AF1 reduces LLaMA-2-7B PPL from 16.44 for ARB-LLM to 11.20 using 5.9 GPU hours, and is about 40× faster than LittleBit and 228× faster than OneBit.Source: §5.4 · p. 9
• AF1 reduces memory and improves decoding throughput versus BF16 and GPTQ-W4 under batch size 1, 1024-token prefill, and 4096-token decoding.Source: §5.4 · p. 9

Conclusion

• AF1 combines NABF for binary reconstruction with HiSA for allocating limited structural budgets across layers and submodules.NABF addresses reconstruction, while HiSA handles non-uniform structural sensitivity.Source: §6 · p. 9
• AF1 substantially outperforms existing binary PTQ methods under comparable or lower BPW and approaches QAT-based binarization accuracy with lower quantization cost.Source: §6 · p. 9
• AF1 provides practical memory reduction and decoding acceleration as a scalable, deployment-friendly pathway toward genuine 1-bit LLM compression.Source: §6 · p. 9

Limitations

• AF1 compresses target linear weights at 1.0 effective BPW but does not directly compress activations or the KV cache.These uncompressed components may matter in long-context or large-batch serving.Source: Limitations · p. 10
• The evaluation covers representative LLaMA, Qwen, and Gemma models and standard benchmarks, while broader architectures, instruction-tuned variants, and real workloads remain for future validation.Source: Limitations · p. 10

Ethical Considerations

• More efficient deployment may lower barriers to misuse, so AF1-compressed models should retain the safety, monitoring, and access-control mechanisms of full-precision models.Source: Ethical Considerations · p. 10

Appendix contents

• Appendix Section A.1 documents AF1's detailed algorithms, including the overall workflow, NABF, HiSA, and ADMM-SVID framework.Source: Appendix Overview · p. 14
• Appendix Section A.2 provides proofs covering binary parameterization capacity, null-space properties, residual-energy cutoff, and scale compensation.Source: Appendix Overview · p. 14
• Appendix Sections A.3 and A.4 analyze effective bit-width and provide more experimental results, including detailed results and multiple ablations.The ablations cover residual-energy thresholds, allocation-sensitivity metrics, HiSA hyperparameters, and calibration.Source: Appendix Overview · p. 14
• Appendix Sections A.5 and A.6 contain dialogue examples and discuss the use of large language models.Source: Appendix Overview · p. 14

A.1 Detailed Algorithms of AF1

• The appendix provides the algorithmic workflow of AF1.Source: §A.1 · p. 14
• One pseudocode block corresponds to the overall PTQ pipeline.Source: §A.1 · p. 14
• Separate pseudocode blocks correspond to Null-space-Aware Binary Factorization and Hierarchical Shapley Allocation.Source: §A.1 · p. 14

A.1.1 Overall Workflow of AF1

• Given a pretrained model and calibration set, AF1 first collects statistics required by HiSA and NABF.Source: §A.1.1 · p. 14
• HiSA determines layerwise and module-wise structural budgets, while NABF reconstructs each target linear operator under its assigned intermediate dimension.Source: §A.1.1 · p. 14
• After replacing all target operators, AF1 freezes the binary sign matrices and performs scale-only global reconstruction.Source: §A.1.1 · p. 14

• NABF scales weights using input-activation and output-gradient statistics to preserve task-sensitive channel information in a Hessian-aware surrogate space.Source: §A.1.2 · p. 14
• NABF uses two binary sign matrices with continuous scale vectors, updating the continuous factors through alternating ADMM steps before SVID projection.Source: §A.1.2 · p. 14
• Null-space compensation folds discrete projection residuals into existing intermediate scaling vectors using an approximate projector derived from the fixed factor.Source: §A.1.2 · p. 14
• ColScaleFit and RowScaleFit provide column-wise and row-wise least-squares scaling for matching factor representations.Source: §A.1.2 · pp. 14–15
• AF1 collects calibration statistics, applies HiSA allocations to target operators, freezes binary matrices, and optimizes only scale vectors with a KL objective.Source: §A.1.2 · p. 15

Layer-level allocation

• HiSA estimates layer sensitivity by progressively degrading layers under sampled permutations and measuring end-to-end NLL changes.Source: §A.1.3 · p. 15

Block-level allocation

• HiSA assigns submodule capacity by separately measuring directional and magnitude output distortions, normalizing them by MAD, and fusing them with Soft-OR.Source: §A.1.3 · p. 15
• The fused submodule score becomes a size-aware density and then a module-wise budget.Source: §A.1.3 · p. 15

Budget-to-rank conversion

• AF1 converts each HiSA module budget into the intermediate dimension used by NABF, with dominant double-factor storage proportional to r_i(d_out + d_in) bits.Source: §A.1.3 · p. 15

NABF

• NABF uses ADMM-SVID as its discrete optimization framework and adds null-space-aware compensation to suppress projection residuals.The compensation exploits perturbation directions that do not affect the matrix output.Source: §A.1.4 · p. 17, §A.2.1 · p. 19

Representation capacity

• Double-factor binary parameterization represents each connection as a sum of r binary paths with continuous path coefficients, enabling multiple effective magnitudes and correlation patterns.This expands capacity beyond the separable magnitude of a single-factor representation while keeping the dominant factors binary.Source: §A.2.1 · p. 18

HiSA

• HiSA allocates structural capacity using hierarchical Shapley sensitivity, accounting for interactions among compressed and uncompressed layers.Its layer-level scores reflect downstream propagation, prediction-level changes, and different structural coalitions.Source: §A.2.6 · p. 22
• HiSA separates directional and magnitude distortions because a single MSE score can hide which representation failure mode dominates.The two terms measure angular deviation and representation-magnitude mismatch, respectively.Source: §A.2.7 · p. 22

Storage budget

• AF1 stores binary sign factors with lightweight BF16 scale vectors, while prior binary PTQ methods incur additional effective-storage overhead.The overhead in prior methods includes residual tensors, reconstruction coefficients, saliency masks, group bitmaps, and sparsity metadata.Source: §A.3 · p. 25

Results

• Under 1.00-bit budgets, AF1 remains competitive with or superior to several higher-bit baselines and improves robustness across tasks and allocation variants.HiSA achieves the lowest WikiText2 perplexity in the reported allocation comparisons, while dual-distortion fusion outperforms single-distortion variants.Source: §A.4.1 · p. 26, §A.4.3 · pp. 26–27