Source-linked AI summary
MPQ: A Message-Passing View of Post-Training Quantization
Yanshuo Chen, Chenxi Liu, Reza Shirkavand, Heng Huang
TL;DR
Existing PTQ solvers commit to a single codeword for each block before considering neighboring blocks, despite plausible alternatives and inter-block coupling. MPQ treats row-wise PTQ as posterior inference and propagates soft codeword information before hard projection and local refinement. It improves over strong solver baselines across model families and quantization regimes, with the clearest gains at 2 bits.
Problem
Existing PTQ solvers commit to one codeword per block before neighboring blocks are processed, although low-bit assignments can have near-tied alternatives and strong coupling.
Source: §1 · p. 2Method
MPQ casts row-wise PTQ as posterior inference over discrete codebooks, uses finite-temperature VAMP to propagate soft blockwise beliefs, then refines a hard initialization with block coordinate descent.
Source: §1 · p. 2Results
MPQ improves over strong solver baselines across aggressive low-bit regimes, with the clearest gains at 2 bits; downstream accuracy gains over the matched CD-polished greedy baseline range from 0.4 to 1.7 percentage points.
Source: §6.3 · p. 8, §7 · p. 10Takeaways & Limitations
Finite-temperature message passing offers a solver-side way to propagate codeword uncertainty across blocks before committing to a quantized solution.
Source: §7 · p. 10The VAMP state-evolution analysis assumes an i.i.d. uniform finite codebook and a right-orthogonally-invariant operator.
Source: §5.1 · p. 6
Abstract
from arXiv · showhide
Post-training quantization (PTQ) is a key technique for efficiently deploying large language models, as it compresses weights into low-bit representations without retraining. Existing PTQ solvers, including GPTQ, LDLQ, and block coordinate descent, commit each weight block to a single codeword before its neighbors are considered, leaving near-optimal alternatives unexplored. In this work, we cast row-wise PTQ as posterior inference over discrete codebooks, in which existing solvers correspond to a zero-temperature, hard-decision limit retaining only the top codeword per block. We propose Message-Passing Quantization (MPQ), which preserves soft posterior information across blocks during inference and commits to hard codewords only at the output stage. MPQ first uses vector approximate message passing to compute approximate blockwise posteriors, then refines a hard-decision initialization by exact block coordinate descent. Across 2-bit lattice and 3- and 4-bit scalar quantization on dense language models from Llama, Qwen, and Mistral families, MPQ improves perplexity and zero-shot accuracy over strong hard-decision baselines, and composes favorably with preprocessing techniques such as activation-aware scaling and incoherence processing. Our implementation is available at https://github.com/poseidonchan/MPQ.
1 Introduction
PTQ supports deployment of large language models under tight memory and compute budgets, but existing solvers commit to one codeword per block before considering its neighbors. MPQ instead propagates distributions over plausible codewords, improving over strong solver baselines, especially at low bit widths.
1 Introduction
- PTQ compresses pretrained model weights into low-bit discrete codes for deployment under tight memory and compute budgets without retraining.Prior work has improved preprocessing and objective design through activation scaling, incoherence processing, structured codebooks, and inter-layer error propagation.Source: §1 · p. 1
- Existing solvers such as GPTQ, LDLQ, and block coordinate descent make hard assignments before neighboring blocks are processed, leaving plausible alternatives unexplored.This is especially consequential at low bit widths, where multiple codewords can have similar local error and block choices are coupled.Source: §1 · p. 2
- The paper casts row-wise PTQ as posterior inference over a discrete codebook with a Gaussian likelihood, interpreting existing solvers as zero-temperature schemes that propagate point estimates.A finite-temperature alternative lets messages carry distributions over likely codewords before commitment.Source: §1 · p. 2
- MPQ first uses finite-temperature VAMP to aggregate information about near-tied assignments across blocks, then converts the posterior into a hard initialization for block coordinate descent.The method explores uncertainty before committing to a discrete solution.Source: §1 · p. 2
- MPQ improves over strong solver baselines in perplexity and zero-shot accuracy across Llama, Qwen, and Mistral, with the largest gains at 3-bit scalar and 2-bit lattice precision.It combines finite-temperature vector approximate message passing with exact block coordinate descent.Source: §1 · p. 2
2 Related Work
Related work improves PTQ through preprocessing, codebook design, and objectives that account for error propagation, while solver-side methods largely retain sequential hard assignments. MPQ connects this solver problem to discrete message passing.
2 Related Work
- Preprocessing and objective-design methods rescale or transform weights, improve low-bit codebooks, or account for quantization errors propagating across layers.Examples include activation scaling, incoherence processing, structured codebooks, and inter-layer objectives.Source: §2 · p. 2
- Solver-side PTQ spans GPTQ, LDLQ, and block coordinate descent, while OJBKQ samples multiple Babai trajectories that still commit sequentially within each trajectory.The paper unifies these solvers as zero-temperature limits of a posterior view.Source: §2 · pp. 2–3
- AMP methods maintain soft beliefs over latent blocks, and VAMP provides rigorous state evolution for large right-orthogonally-invariant operators.The paper uses this model as qualitative guidance for PTQ with a random Hadamard transform, which improves incoherence but does not make a finite calibration Hessian exactly right orthogonally invariant.Source: §2 · p. 3
3 Preliminaries
The preliminaries formulate weight-only PTQ as a row-wise discrete optimization problem whose blocks are coupled by the calibration matrix. A finite-temperature Gibbs posterior provides a probabilistic view that recovers the original minimizers in the zero-temperature limit.
3 Preliminaries
- Weight-only PTQ minimizes the layer-wise reconstruction error between activations applied to the original and quantized weights, and the objective decomposes over output rows.Source: §3 · p. 3
- Each output row is partitioned into blocks assigned to finite codebooks, covering scalar grids and structured vector codebooks such as E8P12.The per-row assignment ranges over the product of the block codebooks.Source: §3 · p. 3
- Off-diagonal terms in the calibration Gram matrix couple blocks, so changing one block perturbs the residual seen by every other block.This makes the row-wise assignment a combinatorial problem over the product codebook.Source: §3 · p. 3
- The finite-temperature Gibbs measure is equivalently a posterior under a Gaussian likelihood and a codebook-supported reference measure.The reference measure uses per-block anchoring widths derived from fixed scales and a global multiplier.Source: §3 · p. 3
- As inverse temperature tends to infinity, the posterior concentrates on minimizers of the original per-row PTQ objective; at finite temperature it retains probability on near-best codewords.Hard-assignment solvers approximate the zero-temperature MAP problem, while message passing can propagate finite-temperature blockwise means, variances, and probability mass.Source: §3 · p. 3
4 Message-Passing Quantization
MPQ delays hard assignments by first passing soft blockwise beliefs through VAMP, then projecting to a discrete initialization and refining it with exact block coordinate descent. The VAMP stage can operate in rank space, while the final local search remains dependent on the current residual.
4 Message-Passing Quantization
- MPQ applies finite-temperature inference to the Gibbs posterior, then refines the resulting hard assignment with exact block coordinate descent on the row-wise PTQ objective.The first stage preserves uncertainty over near-tied codewords; the second performs local repair after projection.Source: §4 · pp. 3–4
- Figure 1 contrasts greedy sequential fixing and error passing with MPQ’s alternating denoising and Gaussian updates before marginal MAP projection and exact block coordinate descent.Rows represent weight blocks, colors represent codewords, and bar heights represent codeword probabilities.Source: Figure 1 · p. 4
4.1 Finite-temperature VAMP in rank space
- The VAMP stage alternates a blockwise discrete denoiser with a Gaussian linear module, keeping every block soft-valued throughout its iterations.Its computations use the compact SVD rank-space representation rather than the original calibration matrix.Source: §4.1 · pp. 4–5
4.2 Zero-temperature exact local refinement
- After VAMP, MPQ projects blockwise marginal modes to a hard initialization and runs exact block coordinate descent, although marginal modes need not form a joint MAP assignment.The block search is exact for each update but remains local because it evaluates candidates against the current residual; the VAMP warm start enters refinement from a better basin.Source: §4.2 · p. 5
5 Theoretical Analysis
MPQ’s analysis links soft posterior information to more reliable hard decisions, then traces how preprocessing affects the VAMP linear module. It treats the theory as qualitative guidance because the analysis uses an idealized surrogate and an unanchored limit.
5 Theoretical Analysis
- MPQ retains near-tie information with finite-temperature VAMP, then projects blockwise posterior modes and refines the hard initialization with exact block coordinate descent.The analysis treats its results as qualitative guidance, not a tight characterization of MPQ’s finite-temperature operation.Source: §5 · p. 5, §5.1 · p. 6
5.1 Posterior concentration under VAMP
- At a VAMP state-evolution fixed point, each block sees an effective AWGN channel, and Theorem 5.1 bounds its MAP error probability.The bound becomes less informative for dense codebooks with many nearby competitors; residual errors are left to block coordinate descent.Source: §5.1 · p. 6
5.2 Preprocessing and the linear module
- For fixed codebook geometry, preprocessing enters the surrogate linear module through Elin, which depends on the Hessian spectrum.Proposition 5.2 characterizes how diagonal preprocessing changes Elin along a path from the baseline Hessian to the scaled Hessian.Source: §5.2 · p. 7
- The analysis extends to AWQ+RHT because orthogonal conjugation preserves the Hessian eigenvalues on which Elin depends; the realized AWQ contribution is typically negative.Coordinates with positive log-scales decrease Elin, while negative log-scales increase it, weighted by resolvent factors.Source: §5.2 · p. 7
6 Experiments
Across Llama sizes and bit budgets, MPQ improves perplexity over strong baselines, and its advantage also appears in zero-shot accuracy across model families. Solver time varies by quantization regime, while CD refinement and preprocessing diagnostics probe the method’s behavior.
6.1 Setup
- Evaluation uses 128 C4 calibration sequences of length 2048, with 16-bit activations; scalar experiments apply activation-aware scaling and RHT preprocessing.The study measures WikiText-2 and C4 perplexity and zero-shot accuracy on five multiple-choice tasks; the default MPQ setting uses 16 VAMP iterations and three CD passes.Source: §6.1 · p. 7
6.2 Llama-family perplexity scaling; 6.3 Zero-shot transfer across model families
- Across Llama sizes and bit budgets, MPQ stays ahead of OJBKQ; it is best or tied best in nine of ten zero-shot model/bit-width cells.The zero-shot comparison covers five model families at 3-bit scalar and 2-bit E8P12 quantization.Source: §6.2 · p. 8, Table 1 · p. 8, §6.3 · p. 8
6.4 Solver scalability
- Across Llama-family scales from 1B to 13B, MPQ matches GPTQ+CD3 at 3-bit but remains several times slower than greedy GPTQ; at 2-bit it stays within 3× of LDLQ and is faster than OJBKQ.On Meta-Llama-3-8B, VAMP alone nearly saturates scalar quality, whereas each additional CD sweep lowers 2-bit E8P12 perplexity.Source: §6.4 · p. 9, Figure 2 · p. 9, Figure 3 · p. 9
6.5 Ablation and mechanism study
- With the D4 codebook, MPQ has the lowest objective after one CD sweep and the lowest perplexity after three; with E8P12, its perplexity remains lower than LDLQ’s after 20 sweeps.These controls test whether the result holds across codebooks and under extended refinement.Source: §6.5 · p. 10
- Under AWQ+RHT on Meta-Llama-3-8B, AWQ decreases Elin for most scale groups, and the leading-order integrand’s sign agrees with the observed change on nearly all groups.Source: §6.5 · p. 10
7 Conclusion
The conclusion presents MPQ as a solver-side approach that carries codeword uncertainty across blocks before committing, with its clearest gains in aggressive low-bit regimes. The appendix reports stable pilot defaults and describes the full seven-model perplexity matrix.
7 Conclusion
- MPQ’s benefits are clearest at 2 bits, where it improves over strong solver baselines across aggressive low-bit regimes.The conclusion frames finite-temperature message passing as a way to propagate codeword uncertainty before committing to a quantized solution.Source: §7 · p. 10
A.1 Pilot sweep for frozen VAMP hyperparameters
- The selected VAMP setting, (16, 0.5, 0.2), lies in the sweep’s stable region, and 3-bit results remain stable across 32–128 calibration sequences and three seeds.Figure 5 summarizes sweeps over iterations, damping, anchoring multiplier τ, and calibration budget.Source: §A.1 · p. 14, Figure 5 · p. 14
A.2 Full perplexity matrix
- The full perplexity matrix covers five Llama models, Qwen3-8B, and Mistral-7B, with WikiText-2 and C4 entries and codebook/preprocessing regimes specified by section headers.MPQ denotes the final CD-polished configuration.Source: Table 3 · p. 14, §A.2 · p. 15
A.3 Full zero-shot task matrix
The appendix provides a retained zero-shot task matrix, recipe and speed-quality ablations, and diagnostics of cleanup and preprocessing proxies. Its cleanup diagnostic distinguishes nearly inactive CD in 3-bit scalar quantization from substantial local repair in 2-bit E8P12.
A.3 Full zero-shot task matrix
- The retained zero-shot task matrix marks the best quantized result within each model, bit width, and task in bold, and the second-best by underlining.The supplied material identifies the matrix but does not provide its task-cell outcomes.Source: §A.3 · pp. 15–16, Table 4 · p. 15
A.4 Ablation study
- Figure 6 compares recipe variants for 3-bit scalar preprocessing/CD choices and 2-bit E8P12 VAMP/CD-polished recipes; AWQ is excluded from the 2-bit preprocessing ablation because it disrupts lattice block-level isotropy and makes quantization fail.Table 5 separately tests AWQ under the same RHT convention.Source: Figure 6 · p. 16, §A.4 · p. 16
A.5 Speed-quality tradeoff and memory
- The speed-quality section reports solver-side costs on one RTX 5090, excluding Hessian construction, evaluation, and export, and includes time-scaling and speed-quality tables.The appendix also supplies full Llama-3-8B grids corresponding to Figure 3.Source: §A.5 · p. 16, Table 5 · p. 17, Table 6 · p. 17
A.6 Theory-guided cleanup diagnostic
- In 2-bit E8P12, the median cleanup fraction is 1.27 × 10^-1 and the global fraction is 1.29 × 10^-1; in 3-bit scalar quantization, the median is zero and the global fraction is 6.44 × 10^-7.The corresponding median precision proxies are 1.30 × 10^1 over 128 fused-module checkpoints and 1.77 × 10^5 over 224 modules, respectively.Source: §A.6 · p. 17
- CD makes only small assignment changes in 3-bit scalar GPTQ, but performs local repair and substantially improves PPL in 2-bit E8P12 LDLQ; the remaining gap to MPQ is attributed to VAMP basin selection, not simply too few CD sweeps.This is the diagnostic’s interpretation of why additional refinement alone does not close the gap.Source: §A.6 · p. 20
A.7 Calibration-resampling robustness
The appendix reports calibration-resampling, matched-refinement, and matched-budget comparisons, then documents implementation details and sensitivity/search identities. The reported D4 and E8P12 results show MPQ retaining advantages under matched CD refinement and extended refinement.
A.7 Calibration-resampling robustness
- MPQ has the lowest mean perplexity across the calibration-resampling settings, with larger gains at 3 bits.The comparison uses three independently sampled C4 calibration sets, each with 128 sequences of length 2048; methods share each draw.Source: §A.7 · pp. 20–21
A.8 D4 codebook and matched refinement
- With the 2-bit D4 codebook, MPQ starts with the highest objective but has the lowest after one CD sweep, and has the lowest WikiText-2 and C4 perplexity after three sweeps.The comparison uses identical exact block-CD updates across LDLQ, OJBKQ (K = 5), and MPQ.Source: §A.8 · p. 21, Table 11 · p. 21, Table 12 · p. 21
A.9 Matched solver budgets and extended refinement
- At the higher matched solver-time budget, MPQ reaches lower perplexity than LDLQ with slightly less solver time, and its advantage persists after 20 CD sweeps.Tables 13 and 14 report solver time/WikiText-2 PPL and extended-refinement perplexity, respectively.Source: §A.9 · p. 21, Table 13 · p. 22, Table 14 · p. 22
B Implementation
- The implementation appendix collects exact update formulas and module identities referenced in Section 4, followed by Python-style reference pseudocode.It also specifies deterministic helper routines for the listing.Source: B Implementation · p. 22
B.1 Extrinsic-message updates, initialization, and β update
- The VAMP updates damp means and precisions and clip precisions; initialization scales precision per row, while an online Gaussian-likelihood M-step updates β and removes it as a free hyperparameter.The paper reports more stable behavior than fixing β to a constant.Source: §B.1 · p. 22
B.2 Exact denoiser sensitivity
- The appendix states an exact discrete-denoiser Jacobian and derives the average sensitivity used by MPQ for finite codebooks and finite positive precision.The proof differentiates the normalized discrete posterior.Source: §B.2 · p. 23
B.3 Exact linear-module sensitivity
- For the linear module, the appendix gives a Jacobian with respect to r2 and obtains average sensitivity by averaging its diagonal entries.The map is affine, so its Jacobian is the coefficient of r2.Source: §B.3 · p. 23
B.4 Block coordinate descent search as Mahalanobis nearest-codeword search
- When the block Gram matrix is nonsingular, exact block search is equivalent to nearest-codeword search in its induced Mahalanobis metric; with a singular matrix, the quadratic objective is minimized directly.The search metric depends on the current residual, so later refinement remains local and initialization still matters.Source: §B.4 · p. 24
B.5 Reference implementation
The reference implementation runs finite-temperature VAMP in rank space, projects blockwise posteriors to hard codewords, and then applies exact block coordinate descent. It includes safeguards, best-iterate selection, and deterministic helper routines.
Stage II: MAP projection and exact block coordinate descent
- After VAMP, the implementation takes a blockwise MAP projection and uses that hard assignment to initialize exact block coordinate descent sweeps with residual updates and early stopping.A best-iterate rule passes the MAP projection with the smallest rank-space surrogate error to CD initialization; the RTN baseline initializes the selection.Source: §B.5 · pp. 24–26
Stage I: finite-temperature VAMP
- Stage I initializes from round-to-nearest codes and per-row error scales, then iterates VAMP denoising and linear-module updates in rank space.The pseudocode applies retry/clipping safeguards to precisions and sensitivity values, with damping where configured.Source: §B.5 · pp. 24–25
Helper contracts and safeguards
- The helper contracts define deterministic routines for posterior evaluation, MAP projection, codebook enumeration during CD, β updates, and precision/sensitivity clipping.The listing also specifies retry rescaling, log-space damping, and block-slice indexing.Source: §B.5 · p. 26
C Limitations
The appendix qualifies the asymptotic analysis, discusses broader impacts, and supplies proof details for the ROI state-evolution results.
C Limitations
- The asymptotic results are qualitative guidance: the analysis uses an unanchored τ→∞ ROI surrogate, unlike MPQ’s finite-τ, RHT-conditioned Hessians.A finite-τ and finite-dimensional analysis remains open.Source: C Limitations · p. 26
D Broader impact
- Lower deployment costs may broaden access and reduce inference energy, but may also facilitate harmful uses shared with weight quantization generally.The submission releases an offline solver, not pretrained models, fine-tuned weights, or training data.Source: D Broader impact · p. 27
E Detailed proofs
- The appendix provides full proofs for the two main theorems and records supporting results and implementation identities.The right-orthogonally invariant surrogate is restated for analysis of one row-wise PTQ subproblem.Source: E Detailed proofs · p. 27, §E.1 · p. 27
E.1 Auxiliary specialization of the right-orthogonally invariant (ROI) state evolution
- Under a fixed-size block-separable ROI surrogate, the state-evolution precisions follow the MPQ denoiser’s error maps and VAMP updates.The effective-channel representation follows from the scalar-equivalent model, with local convergence when initialized in the selected attractive fixed point’s basin.Source: §E.1 · pp. 27–28
E.2 Proof of Theorem 5.1
- The proof bounds MAP projection errors using Gaussian tail bounds and a union bound, then bounds expected incorrect blocks by summing across blocks.It conditions on a fixed true codeword before averaging over the uniform codeword draw.Source: §E.2 · p. 28
E.3 Proof of Proposition 5.2
The proof uses orthogonal invariance to reduce the AWQ+RHT comparison, then records assets and checklist support for the paper’s claims.
E.3 Proof of Proposition 5.2
- Orthogonal invariance reduces the AWQ+RHT comparison to the unrotated Hessian path, and the trace response is rewritten coordinatewise with nonnegative resolvent-weighted factors.The linear-module error depends on the Hessian through its eigenvalues; the coordinate form makes the sign explicit.Source: §E.3 · pp. 28–29
F Existing Assets
- The asset list names the evaluated datasets, models, and tooling, including WikiText-2, Llama and Qwen models, and lm-evaluation-harness.The listed assets include C4, ARC, HellaSwag, PIQA, Winogrande, Mistral, GPTQModel, and QuIP#.Source: F Existing Assets · p. 30
NeurIPS Paper Checklist
- The checklist affirms that the abstract and introduction reflect the contributions and scope, citing results across seven dense models and three bit budgets.Its guidance says claims should match the results and reflect expected generalizability.Source: NeurIPS Paper Checklist · p. 31
2. Limitations
The paper identifies asymptotic-only theory and limited codebook scope for its empirical evaluation as limitations. It also reports that statistical variability was assessed only for selected experiments and that runtime was not recorded for Llama-2-70B runs.
2. Limitations
- The theory is asymptotic-only, and the empirical evaluation has a limited codebook scope.The limitations are identified in Appendix C; the supplied passage does not specify the precise boundaries.Source: §2 · p. 31
- Statistical variability is reported for three paired calibration draws on four Llama models at 3- and 4-bit scalar quantization; other experiments use individual runs.The reported mean and sample standard deviation capture calibration-sampling variability.Source: §2 · pp. 33–34
- Runtime was not recorded for Llama-2-70B runs, limiting compute-time reproducibility for those experiments.The runs used an H200 or L40S cluster; other timing and memory protocols are specified.Source: §2 · p. 34