PaperlayerView exact revision on arXiv

Source-linked AI summary

ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models

Jiaqi Chen, Xianglong Yan, Chengzhu Bao, Xie Chen, Yulun Zhang

arXiv:2610.30779v1cs.AI

TL;DR

Sparse activation reduces MoE computation but leaves substantial expert-weight storage costs, and existing low-bit quantization approaches face allocation and routing challenges. ProQuant combines joint expert pruning and bit allocation with routing-preserving quantization compensation. It consistently outperforms competing methods across six benchmarks.

  • Problem

    Sparse activation does not reduce the need to store all expert weights, while existing allocation and router-tuning approaches face low-bit error-estimation and downstream-generalization challenges.

    Source: §1 · pp. 1–2

  • Method

    ProQuant combines joint expert pruning and bit-width assignment with RoCo, which compensates for quantization errors through equivalent transformations while preserving routing.

    Source: §1 · p. 2

  • Results

    ProQuant consistently outperforms all competing methods across six benchmarks, including tasks where other methods suffer substantial degradation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For extremely low-bit allocation, removing an expert may be preferable to retaining a severely degraded version, and evaluation should emphasize downstream-task generalization.

    Source: §5 · p. 8

    Experiments cover models up to 35 billion total parameters; effectiveness and scalability at hundreds of billions or trillions remain unvalidated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) large language models reduce computation through sparse activation but still incur substantial storage costs. Mixed-precision post-training quantization (PTQ) reduces these costs by quantizing different model components at different bit widths, but existing methods often struggle to preserve generation quality under tight bit budgets. We identify two key challenges: (1) nonlinear quantization errors at extremely low precision are underestimated when cross-layer propagation is ignored; (2) direct router fine-tuning may cause routing drift, and calibration gains may not generalize to downstream tasks. To address these issues, we propose ProQuant, a weight-only PTQ framework for MoE models. Prune-or-Prioritize Mixed-Precision Allocation (PoP) jointly optimizes expert pruning and bit-width assignment. It allows higher-precision experts to substitute for severely degraded ones while improving bit-budget utilization. Routing-Consistent RMSNorm Finetune (RoCo) adapts expert input representations through mathematically equivalent transformations. It compensates for quantization errors to reduce routing drift. Extensive experiments demonstrate that ProQuant outperforms existing methods across diverse generative tasks. Code is available at https://github.com/ThrowBug/ProQuant.git.

1 Introduction

The introduction frames MoE quantization as a storage problem compounded by low-bit allocation and routing challenges. It presents ProQuant and reports better performance across six benchmarks.

1 Introduction

  • Sparse activation lowers computation but does not reduce the need to store all expert weights, making storage and weight movement deployment bottlenecks.Weight-only PTQ reduces storage and weight-transfer costs during inference without large-scale retraining.Source: §1 · p. 1
  • Existing allocation metrics may miss cross-layer effects of extremely low-bit quantization, while calibration gains from router fine-tuning do not consistently transfer to downstream generation.These challenges motivate evaluating generation quality rather than relying only on calibration or likelihood-based metrics.Source: §1 · p. 2
  • ProQuant consistently outperforms competing methods across all six benchmarks, including tasks where other methods degrade substantially.The comparison is on Qwen3-30B-A3B-Instruct-2507 at 2.5-bit average expert quantization; experiments also cover two models and 1.5-, 2.0-, and 2.5-bit settings.Source: Figure 1 · p. 1, §1 · p. 2
  • RoCo compensates for expert quantization errors through mathematically equivalent transformations while preserving the original routing behavior.This avoids routing drift associated with directly fine-tuning full-precision parameters.Source: §1 · p. 2
  • ProQuant is a weight-only PTQ framework that combines mixed-precision allocation with lightweight post-quantization compensation.Its components are PoP and RoCo.Source: §1 · p. 2
  • PoP jointly prunes or prioritizes experts to avoid severe degradation from extremely low precision and use limited bit budgets more effectively.The approach can remove low-contribution experts while preserving sufficient precision for more valuable ones.Source: §1 · p. 2

2 Related Works

Related work covers general PTQ techniques and methods tailored to MoE quantization. It highlights mixed-precision allocation and error compensation as responses to expert heterogeneity and low-bit degradation.

2.1 Post-Training Quantization

  • General PTQ methods use second-order compensation, saliency-aware scaling, or equivalent transformations, but can underutilize the bit budget on MoE models.Examples include GPTQ, AWQ, SmoothQuant, and rotation-based methods.Source: §2.1 · pp. 2–3

2.2 Quantization of MoE Models

  • MoE quantization methods use token–expert affinity, router alignment, or expert sensitivity to guide calibration and mixed-precision allocation.Other approaches compensate for quantization error or reduce cross-expert redundancy; ProQuant prunes less important experts and uses router-decoupled RMSNorm fine-tuning.Source: §2.2 · p. 3

3 Method

The method combines expert-level mixed-precision allocation and pruning with RMSNorm adaptation that compensates for quantization error while preserving routing behavior. It builds on weight-only quantization and uses evidence about cross-layer error propagation to guide its allocation choices.

3.1 Preliminaries

  • The preliminaries describe top-K routing and grouped asymmetric weight quantization, then use GPTQ’s layer-wise reconstruction loss and Hessian approximation for low-bit quantization.The router selects the experts with the highest scores, while each weight group is mapped to low-bit integers using a scale, zero point, and rounding.Source: §3.1 · pp. 3–4

3.2 Prune-or-Prioritize Mixed-Precision Allocation

  • PoP jointly selects expert pruning and bit widths, using routing-score-weighted quantization cost normalized by expert activation frequency.Mean-routing-score rankings yield more consistent top-32 expert selections across C4 and downstream benchmarks than activation-count rankings.Source: §3.2 · p. 4, Figure 4 · p. 4
  • Because 1-bit errors can persist across layers, ProQuant excludes 1-bit candidates under average expert bit-width budgets of 1.5, 2.0, and 2.5 bits.The motivating experiment quantizes layer 0 at 1–4 bits while keeping subsequent layers at 2 bits; local output-error estimates may understate cross-layer effects.Source: §3.2 · p. 4
  • Replacing 1-bit quantization with 0-bit pruning frees budget for important experts and removes severely compressed experts that higher-precision experts can replace.The allocation ILP jointly determines pruning and retained-expert bit widths, with candidate widths {0, 2, 3} and a bound on the number of pruned experts.Source: §3.2 · pp. 4–5

3.3 Routing-Consistent RMSNorm Finetune

  • RoCo adapts hidden representations through equivalent RMSNorm and router-weight scaling, compensating for quantization errors while preserving original routing behavior.The inverse scaling on router weights cancels the RMSNorm scaling in router logits; the learned scaling is folded into the weights after fine-tuning, adding no inference overhead.Source: §3.3 · pp. 5–6
  • RoCo achieves higher top-32 expert-set Jaccard similarity than the compared strategies across most MoE layers, better preserving full-precision expert-selection preferences.The comparison uses mean-routing-score rankings on three datasets.Source: §3.3 · p. 6, Figure 6 · p. 6

4 Experiments

ProQuant is evaluated on generation tasks, conventional likelihood metrics, and component ablations. It performs especially well at low bit budgets, while the ablations examine the roles of PoP, RoCo, and PoP’s pruning-ratio setting.

4.1 Settings

  • Experiments use two Qwen models, C4 calibration sequences, and EvalScope generation benchmarks, with perplexity and zero-shot results as auxiliary metrics.The comparison methods use GPTQ with their respective allocation or compensation strategies.Source: §4.1 · p. 6

4.2 Main Results

  • Generation-based evaluation shows ProQuant is best or competitive across models and bit budgets, with larger advantages at lower precision.Likelihood-based benchmarks show comparable results, but favorable likelihood scores do not necessarily translate into stronger generation performance.Source: §4.2 · pp. 6–7
  • At 1.5 bits, ProQuant averages 67.30 on Qwen3-30B-A3B-Instruct-2507 and 60.85 on Qwen3.5-35B-A3B, outperforming existing mixed-precision methods.Table 1 covers six generation-based benchmarks across both models and varying average expert bit-widths.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Ablation Study

  • PoP’s full {0, 2, 3} candidate set scores 74.07 on average, versus 68.46 for {1, 2, 3} and 66.98 for {0, 1, 2, 3}.The comparison indicates the gain is not from pruning alone: allowing 1-bit experts can retain severely degraded experts in the forward pass.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • Without RoCo, the average score drops from 67.30 to 64.77; direct RMSNorm fine-tuning reaches 65.91 but remains below full ProQuant.The passage attributes partial recovery to RMSNorm fine-tuning and says direct updates may disturb routing by altering router inputs.Source: §4.3 · p. 8, Table 4 · p. 8
  • On Qwen3-30B-A3B-Instruct-2507, performance varies only slightly as the maximum pruning ratio ρ ranges from 0.10 to 0.30 without RoCo.Above 0.30 the constraint is inactive; a moderate maximum ratio is generally favorable for average accuracy.Source: §4.3 · p. 8, Table 5 · p. 8

5 Conclusion

The conclusion argues that extremely low-bit allocation should discard experts whose functionality cannot be preserved rather than retain severely degraded versions. It also emphasizes downstream generalization and generation-based evaluation.

5 Conclusion

  • When the bit budget cannot preserve an expert’s functionality, removing it may be preferable to retaining a severely degraded version.The conclusion frames this as “discard it rather than destroy it.”Source: §5 · p. 8
  • The authors caution against over-optimizing for calibration data or specific metrics and emphasize generalization to downstream tasks.Source: §5 · p. 8
  • Perplexity and log-likelihood measure probability assignment to reference sequences, so the conclusion says they cannot replace real-world generation evaluations.Source: §5 · p. 8

Limitations

The experiments evaluate models with up to 35 billion total parameters; effectiveness and scalability at hundreds of billions or trillions of parameters remain unvalidated.

Limitations

  • ProQuant is evaluated only up to 35 billion total parameters, leaving its effectiveness and scalability at hundreds of billions or trillions unvalidated.Source: Limitations · p. 9

A ProQuant Pipeline and Algorithms

The pipeline allocates expert bit-widths under a global budget, quantizes weights with GPTQ, then compensates for quantization errors by fine-tuning RMSNorm scaling vectors and fusing them into RMSNorm and router weights.

A ProQuant Pipeline and Algorithms

  • The pipeline first evaluates each expert’s quantization cost across candidate bit-widths to determine allocations under the global bit budget.It then applies GPTQ, freezes other parameters, fine-tunes RMSNorm scaling vectors, and fuses the learned scales and element-wise inverses into RMSNorm and router weights.Source: A ProQuant Pipeline and Algorithms · p. 12

B Additional Implementation Details

This section records the calibration, quantization, and evaluation settings used to produce and assess the quantized models.

B Additional Implementation Details

  • Calibration uses 128 C4 sequences of 2,048 tokens, sampled with fixed random seed 0, for expert-cost estimation, GPTQ calibration, and RoCo fine-tuning.The ILP in Eq. 8 is solved with HiGHS via SciPy; GPTQ uses group size 128 and asymmetric quantization.Source: B Additional Implementation Details · p. 12
  • RoCo freezes model weights and fine-tunes RMSNorm scaling vectors on the calibration set using learning rate α for T steps.The algorithm then forwards the model with the scaling vectors and their element-wise reciprocals; the supplied steps also mention a loss computation and absorbing learned scaling into RMSNorm and router weights.Source: B Additional Implementation Details · p. 12
  • Clipping bounds l and u are selected by minimizing weight quantization error for the final bit-width assignment.Ŵ(l, u) denotes the quantized-dequantized weights under those bounds.Source: B Additional Implementation Details · p. 12
  • For Qwen3.5-35B-A3B, maximum pruning ratios are 0.1, 0.15, and 0.3 at 2.5, 2.0, and 1.5 expert bits; Qwen3 uses ρ = 1.0.RoCo learning rates are 3 × 10−3 for Qwen3.5 and 5 × 10−3 for Qwen3; both use AdamW for one epoch with batch size 1.Source: B Additional Implementation Details · p. 12
  • Downstream evaluation uses EvalScope with official sampling recommendations in non-thinking mode and task-specific maximum generation lengths.Sampling uses temperature 0.7, top-p 0.8, top-k 20, and min-p 0.0; generation limits range from 2,048 to 8,192 tokens across the listed benchmarks.Source: B Additional Implementation Details · pp. 12–13

C Efficiency Analysis

ProQuant's efficiency analysis reports model-size compression and the quantization-time cost of its allocation and fine-tuning steps.

C Efficiency Analysis

  • 7.55× storage compression: ProQuant reduces Qwen3-30B-A3B-Instruct-2507 from 61.06 GB in BF16 to 8.09 GB after quantization.Figure 7(a) compares model size across expert average bit-width settings for Qwen3-30B-A3B-Instruct-2507 and Qwen3.5-35B-A3B.Source: Figure 7 · p. 13, C Efficiency Analysis · p. 13
  • 1.21× overhead over GPTQ: ProQuant's added statistical computation for bit-width allocation and RoCo fine-tuning increases quantization time by this amount.Figure 7(b) compares methods on Qwen3-30B-A3B-Instruct-2507 at a 2.0-bit expert average; ProQuant's allocation is lighter-weight than GEMQ's gradient-based backpropagation.Source: Figure 7 · p. 13, C Efficiency Analysis · p. 13

C.1 Model Size Calculation

The model-size calculation accounts separately for mixed-precision experts, parameters fixed at 4 bits, and parameters retained in BF16.

C.1 Model Size Calculation

  • For asymmetric quantization with group size 128, storage accounting includes two 16-bit metadata values per group: scale and zero-point.Original weights are stored in BF16 at 16 bits per parameter.Source: §C.1 · p. 13
  • Model size at average expert bit-width B is SE r(B) + S4 r(4) + SFP, with each category measured by its original BF16 storage size.When experts are pruned, this formula is a conservative upper bound because it still includes metadata for pruned experts.Source: §C.1 · p. 13
  • Qwen3-30B-A3B-Instruct-2507 has calculated model sizes of 11.717 GB, 9.905 GB, and 8.093 GB at 2.5, 2.0, and 1.5 average expert bits, respectively.The calculation uses 57.982 GB of original BF16 expert weights, 1.812 GB of attention weights, and 1.270 GB of remaining BF16 parameters.Source: §C.1 · p. 13
  • Qwen3.5-35B-A3B model sizes are 14.276 GB, 12.263 GB, and 10.249 GB at 2.5, 2.0, and 1.5 average expert bits, respectively.These sizes include language-model parameters only and exclude the vision encoder.Source: §C.1 · p. 14

D Comparative Analysis of Expert Importance Metrics

The comparison finds that expert-importance metrics materially affect downstream scores under the same allocation settings, while removing activation-frequency influence improves most evaluated benchmarks.

D Comparative Analysis of Expert Importance Metrics

  • At a 2.0-bit budget with {1, 2, 3} candidate widths and no fine-tuning, PoP's importance metric averages 61.40 across six benchmarks, beating MxMoE, PMQ, and GEMQ by 11.51, 11.75, and 11.18 percentage points.The allocation procedure and other quantization settings are held identical for this comparison.Source: D Comparative Analysis of Expert… · p. 14
  • Removing activation-frequency influence raises the average score from 47.84 to 61.40 and improves five of six benchmarks, with a decrease on MMLU-Redux.The authors hypothesize that C4 calibration-set activation patterns may align more closely with MMLU-Redux, while retaining frequency dependence lowers performance on the other evaluated benchmarks.Source: D Comparative Analysis of Expert… · p. 14

E Dialog Examples

The dialog examples compare model responses on mathematical and tool-use prompts, showing both correct answers and failures. The section also records software attribution and the authors’ AI-use statement.

E Dialog Examples

  • For the trigonometric prompt, BF16 and ProQuant return cot x, while other responses loop, call the notation ambiguous, or leave the expression unsimplified.The supplied responses include MxMoE, PMQ, and GEMQ examples of those failures.Source: E Dialog Examples · pp. 15–16
  • ProQuant’s factorial call and BF16’s quadratic call are correct, while shown MxMoE and PMQ factorial calls are malformed.The examples include the factorial-of-5 prompt and a quadratic-root prompt.Source: E Dialog Examples · p. 16
  • In the rotation example, ProQuant and BF16 answer −2 + 7i correctly, while other shown responses give the wrong sign or no final answer.The table caption describes four examples using Qwen3-30B-A3B-Instruct-2507, with quantized methods at an average expert bit-width of 2.5 bits.Source: E Dialog Examples · pp. 14–15, Table 7 · p. 17

F License and Attribution

G AI Use Statement

  • AI assistants were used only for instructed code modifications and manuscript wording; the authors conducted the literature review and developed the research contributions.Source: G AI Use Statement · p. 17
Loading 2610.30779v1…