PaperlayerView exact revision on arXiv

Source-linked AI summary

BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models

Junyu Chen, Jungang Li, Jing Xiong, Wenjie Wang, Qingyao Yang, He Xiao, Zhen Li, Taiqiang Wu, Mengzhao Chen, Zhen Peng, Chaofan Tao, Long Shi, Hongxia Yang, Ngai Wong

arXiv:2602.04163v2cs.LG

TL;DR

Low-bit post-training quantization is constrained by rigid, shape-invariant grids. BPDQ uses bit-plane decomposition to build and refine variable grids, enabling 2-bit Qwen2.5-72B inference on one RTX 3090 with 83.85% GSM8K accuracy.

  • Problem

    Shape-invariant quantization grids restrict the feasible set for optimization-based post-training quantization in low-bit regimes.

  • Method

    BPDQ constructs variable grids from bit-planes and scalar coefficients, then refines them in Hessian-induced geometry with delta correction.

  • Results

    83.85% GSM8K accuracy enables serving Qwen2.5-72B at 2 bits on a single RTX 3090.

  • Takeaways & Limitations

    BPDQ supports high-fidelity 2-bit inference for 72B models on consumer-grade GPUs while retaining a hardware-friendly format.

  • Takeaways & Limitations

    BPDQ retains a fidelity gap relative to vector quantization, which has higher overhead and limited hardware support.

Abstract

from arXiv · show

Large language model inference is often bounded by memory footprint and bandwidth in resource-constrained deployments, making quantization fundamental to efficient serving. While post-training quantization (PTQ) maintains high fidelity at 4-bit, it deteriorates at 2-3 bits. In essence, existing methods enforce a shape-invariant quantization grid (e.g., the fixed uniform intervals of UINT2) for each group, severely restricting the feasible set for error minimization. To address this, we propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients, and iteratively refines them using second-order information while progressively compensating for quantization errors to minimize output discrepancy. In the 2-bit regime, BPDQ enables serving Qwen2.5-72B on a single RTX 3090 with 83.85\% GSM8K accuracy (vs. 90.83\% at 16-bit). Moreover, we theoretically show that the variable grid expands the feasible set, and that the quantization process consistently aligns with the optimization objective in Hessian-induced geometry. The code is available at https://github.com/KingdalfGoodman/BPDQ.

1. Introduction

Low-bit PTQ is limited by the shape invariance of fixed quantization grids, especially at 2–3 bits. BPDQ constructs variable grids with bit-plane coefficients and Hessian-guided refinement, expanding the feasible set while retaining accuracy and enabling efficient 72B deployment.

  • Motivation: 2-bit quantization provides only four distinct values, causing representational loss and severe model-quality degradation.
  • Problem: Fixed quantization grids impose shape invariance across groups, restricting feasible values and misaligning optimization with the quantizer’s feasible set.
  • Method: BPDQ constructs variable grids through bit-plane decomposition and scalar coefficients, allowing group-specific spacing patterns and enlarging the feasible set.
  • Results: 83.85% accuracy on GSM8K enables serving Qwen2.5-72B at 2-bit precision on a single RTX 3090 with 22.69 GB VRAM.BPDQ also achieves 86.13% on GSM8K in the cited 2-bit Qwen2.5-72B comparison and uses a bit-plane look-up-table kernel for low-latency decoding.
  • Method: BPDQ iteratively refines bit-planes and scalar coefficients within Hessian-induced geometry so optimization remains aligned with the output-aligned objective.

2. Related Work

Prior low-bit quantization methods trade training cost, quantization overhead, or fidelity, while optimization-based PTQ is constrained by rigid fixed grids. BPDQ addresses this limitation with a variable grid that expands the feasible solution set for extreme compression.

  • Low-bit Quantization for LLMs: QAT methods optimize in the Boolean domain or use factorized representations, but incur substantial training costs.
  • Low-bit Quantization for LLMs: Vector quantization preserves high fidelity through codebooks but suffers from prohibitive quantization overheads.
  • Low-bit Quantization for LLMs: Other approaches typically rely on progressive residual correction or fine-tuning without a rigorous output-aligned objective.
  • Optimization-based PTQ for LLMs: Optimization-based PTQ minimizes output discrepancy using objectives such as ∥WX −c WX∥ and builds on second-order sensitivity analyses and OBC.
  • Optimization-based PTQ for LLMs: Fixed quantization grids restrict the feasible solution set, degrading low-bit optimization-based PTQ; BPDQ uses a variable grid to expand it.

3. Methodology

BPDQ minimizes calibration-data output reconstruction error in Hessian-induced geometry using a variable grid formed from bit-planes and group-wise scalar coefficients. It initializes with dominant bit-planes and closed-form coefficient fitting, then alternates exact bit-plane updates, coefficient refitting, and delta correction to preserve error-propagation consistency.

  • Variable Grid Representation: The variable grid combines k binary bit-planes with group-wise scale coefficients and a group-wise bias coefficient, expanded across each group’s input columns.The operator REP repeats each group coefficient across its g columns.
  • Optimization Objective: BPDQ minimizes output reconstruction error over admissible low-bit weights using H = XX^T as a calibration-induced approximate Hessian metric.The objective is defined for a linear layer with weights W and calibration activations X.
  • Bit-Plane Selection: BPDQ initializes each group by retaining the k most significant bit-planes from an unsigned 8-bit RTN quantizer and discarding the remaining least significant planes.The retained MSB planes capture dominant magnitude information, while discarding LSB planes introduces only a small truncation error.
  • Scalar Coefficient Fitting: With fixed bit-planes, BPDQ fits scalar coefficients by closed-form row-wise weighted least squares under the Hessian-induced geometry.This fitting is an optimal projection for the fixed bit-planes and uses damping α = 10^-4 for numerical stability.
  • Iterative Refinement and Error Compensation: BPDQ alternates column-wise exact bit-plane updates and group-wise coefficient refitting, then applies delta correction so propagation states remain consistent after refitting.Bit-plane updates enumerate 2^k candidates per row, and the experiments consistently use 10 iterations while retaining the iterate with minimum group-wise propagation error.

4. Experiments

Experiments across multiple language models and benchmarks show that BPDQ’s variable-grid quantization is especially effective at 2–3 bits, preserving reasoning and long-context performance while enabling extreme compression. It also retains activation outliers and remains substantially more efficient than vector quantization, despite requiring iterative refinement.

  • Experimental Setup: BPDQ is evaluated across Qwen-3, Qwen-2.5, and Ministral-3 models using lm-evaluation-harness on language understanding, reasoning, and other benchmarks.All methods use asymmetric quantization calibrated on 1024 C4 samples, with larger BPDQ group sizes used to maintain similar bits-per-weight.
  • Benefits of Variable Grid: BPDQ yields the best performance in most tested settings, with its strongest advantage over GPTQ and AWQ in the 2-bit regime, where reasoning accuracy otherwise collapses.For example, 2-bit AWQ reaches 0.00% on the 72B model in a cited reasoning setting, while 2-bit GPTQ also deteriorates severely.
  • Benefits of Variable Grid: 83.85% GSM8K accuracy at W2-G256 compresses Qwen2.5-72B to 22.69 GB for deployment on a single RTX 3090.This retains 92.32% of baseline accuracy; general benchmarks preserve over 91.01%, with BoolQ reaching 99.15%.
  • Comparison and Efficiency: At 2 bits, BPDQ, AnyBCQ, and VPTQ consistently outperform GPTQ and AWQ, while BPDQ remains highly efficient at approximately 3× GPTQ quantization time.VPTQ achieves the highest accuracy but requires approximately 40× GPTQ’s quantization time; BPDQ uses 10 iterations across experiments.
  • Activation Outlier Statistics: BPDQ preserves essential activation outliers under 2-bit quantization by extending the feasible set with a variable grid, and outlier preservation positively correlates with downstream performance.The analysis reports outlier intensity using layerwise DiagR P95 and outlier quantity using Cnt10 across 128 WikiText-2 sequences.
  • Long-Context Capabilities: At 2-bit retrieval, BPDQ sustains 53.75% LongBench performance versus GPTQ’s 4.98%, while remaining competitive with baseline performance on summarization and classification.At 3–4 bits, all methods are generally comparable to baseline across most evaluated long-context tasks; VPTQ is more resilient at 2 bits but has prohibitive overhead.

5. Conclusion

BPDQ relaxes shape-invariant quantization grids through bit-plane decomposition, theoretically expanding the feasible solution set and enabling rigorous Hessian-geometry refinement. This approach supports high-fidelity 2-bit inference for 72B models on consumer-grade GPUs.

  • BPDQ relaxes the shape-invariant grid constraint that hampers optimization-based PTQ in low-bit regimes.
  • BPDQ constructs a variable quantization grid through bit-plane decomposition, theoretically expanding the feasible solution set.
  • BPDQ enables rigorous quantization refinement within Hessian-induced geometry.
  • BPDQ unlocks high-fidelity 2-bit inference for 72B models on consumer-grade GPUs.

6. Limitations and Future Work

BPDQ retains a fidelity gap relative to vector quantization, while its bit-plane structure offers promising directions for solver enhancements and efficient, flexible hardware deployment.

  • Fidelity Gap and Enhancements: BPDQ retains a fidelity gap relative to vector quantization, motivating rotation techniques or enhanced sequential solvers such as Qronos.Vector quantization often incurs high overhead and has limited hardware support.
  • Hardware Efficiency on FPGA/ASIC: Binary bit-planes enable FPGA or ASIC implementations that replace expensive floating-point multiplications with simple additions.This can significantly improve energy and area efficiency.
  • Mixed- and Multi-Precision: BPDQ supports mixed precision by allocating more or fewer bit-planes instead of requiring hardware support for diverse data types.Its unified basis inherently surpasses conventional mixed-precision schemes.
  • Mixed- and Multi-Precision: BPDQ naturally supports multi-precision serving from a single on-device model, enabling dynamic accuracy-latency trade-offs.This structure is suited to serving multiple precisions without separate models.

A. Analysis of the Variable Grid · A.1. Optimization-based PTQ as an H-Metric Projection

Optimization-based PTQ can be viewed as projecting each weight vector onto a feasible quantization set under a Hessian-induced metric. In low-bit settings, fidelity is limited by the rigidity of shape-invariant grids rather than by the optimization objective itself.

  • A.1. Optimization-based PTQ as an H-Metric Projection: A Hessian matrix H defines the geometry for a weight vector w within a quantization group of size g.The corresponding vector and matrix lie in R^g and R^{g×g}, respectively.
  • A.1. Optimization-based PTQ as an H-Metric Projection: Optimization-based PTQ selects a quantized vector bw from a feasible set Q ⊂ Rg to minimize output discrepancy.The feasible set contains the quantized candidates available to the method.
  • A.1. Optimization-based PTQ as an H-Metric Projection: This optimization is equivalent to projecting w onto Q under the H-metric.The projection interpretation recasts quantization as a geometric nearest-point problem.
  • A.1. Optimization-based PTQ as an H-Metric Projection: Equation (10) establishes optimization-based PTQ as a nearest-point projection problem.The selected quantized vector is therefore determined by proximity under the Hessian-induced geometry.
  • A.1. Optimization-based PTQ as an H-Metric Projection: Quantization quality is constrained by the geometric richness of the feasible set Q.A more restrictive feasible set limits which quantized vectors can be selected by the projection.
  • A.1. Optimization-based PTQ as an H-Metric Projection: In the 2-3-bit regime, fidelity degradation arises from the rigidity of Q induced by a shape-invariant grid, not from failure of the objective.The passage identifies feasible-set rigidity as the source of low-bit degradation.

A.2. Feasible Set Comparison at 2-Bit Precision … B.2. Consistency of Bit-Plane Update

BPDQ expands the 2-bit quantization feasible set beyond fixed grids and can achieve strictly lower error on an open set of weights. Its coefficient fitting and bit-plane updates are each theoretically aligned with the Hessian-induced optimization geometry.

  • A.2. Feasible Set Comparison at 2-Bit Precision: BPDQ’s independent bit-plane coefficients expand fixed-grid quantization from a lower-dimensional manifold to a higher-dimensional geometry.Fixed grids preserve template level ratios, whereas BPDQ allows group-specific relative spacings.
  • A.2. Feasible Set Comparison at 2-Bit Precision: BPDQ’s feasible set strictly contains the feasible set of standard UINT2 grids.Choosing c1 = s and c2 = 2s exactly reproduces {0, s, 2s, 3s}, while other coefficient choices yield non-uniform spacings.
  • A.2. Feasible Set Comparison at 2-Bit Precision: BPDQ achieves strictly lower quantization error than fixed-template grids on a non-empty open set of weight vectors.Under group size g ≥3 and H ≻0, continuity of the error difference establishes an open neighborhood where Ffix(w) > Fvar(w).
  • A.2. Feasible Set Comparison at 2-Bit Precision: BPDQ’s bit-plane parameterization provides up to three affine-subspace dimensions, compared with at most two for fixed templates.The additional dimension arises from spanning 1, b1, and b2 rather than only 1 and z.
  • B.1. Consistency of Coefficient Fitting: The coefficient fitting objective is theoretically equivalent to minimizing the current group’s local contribution to the output reconstruction objective under Hessian-induced geometry.The equivalence follows by substituting the inverse-Hessian Cholesky factor and decomposing the Frobenius norm row-wise.
  • B.1. Consistency of Coefficient Fitting: The coefficient-fitting formulation parameterizes each quantized row as Brcr and reduces optimization to weighted least squares over group coefficients.A damping factor α = 10−4 is applied to the diagonal for numerical stability.
  • B.2. Consistency of Bit-Plane Update: During bit-plane updates with fixed coefficients, element-wise enumeration within each column minimizes the column-wise optimization error.With H ≻0, Ul,l is a strictly positive choice-independent scalar, making the projected residual minimization equivalent to Euclidean distance minimization.
  • B.2. Consistency of Bit-Plane Update: Fixed-coefficient bit-plane updates decouple into dout independent scalar nearest-neighbor projections and maintain consistency with Hessian-induced geometry.Each coordinate selects its optimal bit vector from values generated by b ∈ {0, 1}k.

B.3. Consistency of Delta Correction

The delta correction preserves error-propagation consistency after coefficient refitting. The proof establishes consistency for both current group columns and tail columns, showing that applying ΔE synchronizes the propagation state.

  • The delta correction preserves error-propagation consistency following coefficient refitting.This is stated as the section’s proposition.
  • Upper-triangular residual decomposition preserves local consistency within the current group columns.The residual separates historical-error contributions from the local group term during coefficient refitting.
  • Differencing old and refitted states strictly derives the delta correction ΔE Uloc = c Wold −c Wnew.The left side remains constant, forcing equality of the corresponding right-side expressions across states.
  • Applying the update ΔE exactly synchronizes propagation for the tail columns.The tail proof treats preceding-group weights and errors as constant while the current group varies.

C. Additional Evaluation Results

Additional evaluations report results for Qwen3 models from 0.6B to 14B, Llama3.1-8B, Gemma2-9B, and Phi4-14B across seven benchmarks, alongside comparisons with bit-plane and VQ-based methods on LLaMA2-7B.

  • Additional model evaluations: Table 4 evaluates Qwen3-0.6B, Ministral3-3B, and Qwen3-4B across seven benchmarks.The table highlights the best and second-best results.
  • Additional model evaluations: Table 5 reports Qwen3-8B and Qwen3-14B results across seven benchmarks.
  • Additional model evaluations: Table 6 evaluates Llama3.1-8B, Gemma2-9B, and Phi4-14B.
  • Method comparisons: Table 7 compares additional bit-plane and VQ-based methods on LLaMA2-7B.
Loading 2602.04163v2…