PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs

Deokjae Lee, Sihun Chu, Hyun Oh Song

arXiv:2608.30564v1cs.LGcs.AI

TL;DR

MoE models make mixed-precision quantization difficult because their expert layers create a very large allocation space, while existing methods do not directly optimize coupled block choices. Q-STRATA uses a bi-level design that ranks assignments within blocks with a cheap proxy and allocates budgets across blocks using the model-level objective, achieving lower low-bit WikiText2 perplexity than the compared methods.

  • Problem

    MoE models have many expert linear layers, and existing methods either enforce uniform per-block budgets or allocate globally with an additive proxy rather than directly optimizing coupled model-level choices.

  • Method

    Q-STRATA caches Pareto-frontier within-block assignments over budget levels, then selects one budget per block by directly optimizing the model-level objective with lazy greedy descent.

  • Results

    Q-STRATA achieves lower WikiText2 perplexity than uniform GPTQ, MxMoE, and GEMQ across three MoE LLMs in the low-bit regime.

  • Takeaways & Limitations

    The hierarchical allocation captures inter-block coupling that additive proxies miss, while the outer search also functions as a standalone dense-model allocator with far fewer objective queries than black-box search.

  • Takeaways & Limitations

    The outer stage is heavier at search time than proxy-only allocation because it repeatedly evaluates the assembled quantized model, and reducing this cost remains future work.

Abstract

from arXiv · show

Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-induced quality loss under a fixed budget, but Mixture-of-Experts (MoE) models contain these layers in every expert of every MoE block, so the allocation space grows far larger than in a dense model. Existing methods either allocate within each block under a uniform per-block budget, or allocate across blocks through an additive proxy, and neither directly optimizes a model-level objective over the choices that couple the blocks. We propose Q-Strata, a bi-level allocator that ranks within-block assignments with a cheap proxy and allocates across blocks with a model-level objective evaluated on the assembled quantized model. Its inner stage caches a Pareto frontier of candidates per block over finely spaced budgets, leaving the outer stage to set one budget per block instead of a bitwidth for every linear layer. With the search reduced to one budget per block, the outer stage optimizes this model-level objective directly, capturing the inter-block coupling that additive proxies miss. On Mixtral-8x7B-Instruct, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite, Q-Strata consistently achieves lower WikiText2 perplexity than uniform-bitwidth GPTQ and the state-of-the-art MoE MPQ methods MxMoE and GEMQ in the low-bit regime. The code is available at https://github.com/snu-mllab/Q-Strata/tree/main.

1 Introduction

MoE models make weight-only mixed-precision quantization especially challenging because their many expert layers create a huge allocation space. Q-STRATA addresses this by ranking assignments within blocks cheaply and allocating budgets across blocks using a direct model-level objective.

  • Motivation: MoE models store all experts although only a few are active per token, making weight compression important for practical deployment.The paper focuses on weight-only post-training quantization using a small calibration dataset without retraining.
  • Challenge: 3LE linear layers across L MoE blocks and E experts create an assignment space of |Q|^3LE, far larger than in dense models.Each expert contains gate, up, and down projections.
  • Challenge: Qwen3-30B-A3B has 18,432 MoE-block linear layers versus 448 in dense Qwen3-32B, exceeding forty times the dense count.The comparison excludes attention and routing from the MoE count.
  • Prior limitations: Existing MoE methods improve uniform quantization with proxies but either keep block budgets uniform or use an additive global proxy.These approaches do not directly optimize the coupled model-level objective across blocks.
  • Q-STRATA: Q-STRATA caches Pareto-optimal within-block candidates and selects one budget per block to minimize the assembled model-level objective under a global budget.The outer search therefore operates over L per-block budgets instead of 3LE individual bitwidths.

2 Preliminaries

The paper formulates MoE MPQ as minimizing a model-level quality objective under an average-bitwidth constraint, while exploiting separable within-block structure for tractable candidate generation.

  • Problem formulation: MoE MPQ assigns one candidate quantizer from Q to each expert projection, forming local block assignments and a global assignment.Each block contains E experts with up, gate, and down projections.
  • Problem formulation: The storage cost includes quantized layer bits plus scale and zero-point overhead, and block and model average bitwidths aggregate these costs.For structurally identical blocks, model average bitwidth reduces to the mean of per-block bitwidths.
  • Problem formulation: MPQ minimizes the model-level objective L at target average bitwidth τ, using Jensen–Shannon divergence between quantized and full-precision next-token distributions.Lower JSD indicates higher quality, and full-precision distributions are precomputed once.
  • Problem formulation: The assignment space is especially difficult for MoE models because its dimension N = 3LE scales with the number of experts.This motivates exploiting the model’s two-level block structure.
  • Block reconstruction proxy: Within-block reconstruction error is coupled by gating and routing, so prior methods approximate it with a decomposable sum of isolated layer distortions.The resulting proxy is block-separable and cheap because each term requires only a single-block forward.
  • Q-STRATA allocation: Q-STRATA’s inner stage caches Pareto-optimal assignments across per-block budgets, while its outer stage chooses one cached candidate per block using L directly.The inner stage is a multiple-choice knapsack problem over the 3E layers and candidate quantizers.

3 Method

Q-STRATA solves MoE mixed-precision allocation in two stages: a proxy-based inner stage builds per-block candidates, while a model-level outer stage selects budgets across blocks. Lazy greedy descent makes the outer search tractable by exploiting empirically supported diminishing returns.

  • 3.1 Inner stage: The inner stage precomputes Pareto-optimal assignments for each MoE block across a shared grid of budgets using a cheap proxy.The proxy allocation is solved as a multiple-choice knapsack problem over the block’s linear layers and candidate quantizers.
  • 3.1 Inner stage: Caching one candidate per budget level reduces the global search from Q^3LE assignments to one budget choice per block.The cached candidates form the per-block sets used by the outer stage.
  • 3.2 Outer stage: The outer stage assembles cached candidates and directly minimizes the non-additive model-level objective under a global budget constraint.Because the objective does not decompose across blocks, the outer selection cannot be solved as a multiple-choice knapsack problem.
  • 3.3 Solving the outer problem: Lazy greedy descent starts from the most expensive allocation and repeatedly lowers the block with the smallest marginal loss increase until the target average bitwidth is reached.Each one-level move saves the same budget, and every candidate move is evaluated on the assembled model.
  • 3.3 Solving the outer problem: The lazy implementation caches marginal losses in a min-heap, recomputing stale values before committing a move.It reproduces eager descent when stale losses underestimate current values, a condition justified empirically but not guaranteed for the model objective.
  • 3.3 Solving the outer problem: O(RLK) model evaluations are required in lazy descent versus O(L^2K) for eager descent, with R near 2.5 in practice.The method obtains assignments for every target budget along a single top-to-bottom sweep.

4 Experiments

Experiments show that Q-STRATA improves low-bit MoE quantization, that direct outer-stage optimization captures inter-block coupling, and that its search remains effective for dense models.

  • 4.2 Main results: Q-STRATA achieves the lowest WikiText2 perplexity for every evaluated model and target bitwidth, with the best zero-shot accuracy except one case.At DeepSeek-V2-Lite and 2.25 bits, MxMoE leads accuracy by 0.3 points.
  • 4.2 Main results: At 1.75 bits on Mixtral, Q-STRATA reduces perplexity from MxMoE’s 25.20 to 12.14 and improves average accuracy by almost eight points.
  • 4.2 Main results: Q-STRATA’s gains over MxMoE widen as the budget tightens because the outer stage directly evaluates the model-level objective across blocks.MxMoE shares Q-STRATA’s inner stage but uses uniform block budgets.
  • 4.3 Outer-stage ablation: The outer-stage lazy greedy method attains the lowest JSD and perplexity at every tested bitwidth, with a 74-JSD advantage over one-shot ILP at 1.75 bits.One-shot ILP uses a separable proxy and therefore ignores inter-block coupling.
  • 4.3 Outer-stage ablation: The pruning-free direct cell-level ILP requires more than eight times Q-STRATA’s model-level evaluations, while descent reaches JSD 364 versus 410–419 for both one-shot solvers.
  • 4.4 Robustness checks: With router fine-tuning restored and applied equally, Q-STRATA has lower perplexity than GEMQ in all 18 evaluated pairs.Rotation is not consistently better at low budgets; the strongest Mixtral results there are rotation-free.
  • 4.5 The outer search on dense models: On dense Llama-2-7B, lazy greedy matches or beats AMQ while using 6.9× fewer objective queries at 8K calibration tokens.At 2K tokens, it stays within 0.001 JSD of eager greedy using 51× fewer queries.

5 Related Work

Prior MoE quantization methods use local or additive proxies, while Q-STRATA instead exploits a hierarchical search space and evaluates objective marginals directly at the outer level.

  • MC-MoE and MxMoE score within-block assignments with proxies but keep every MoE block’s budget uniform.
  • GEMQ allocates at expert granularity across blocks using an additive proxy for model quality.
  • ScaleBits also uses greedy allocation and diminishing returns, but searches a much finer tile-level space with sensitivity estimates rather than measured objective marginals.
  • Q-STRATA’s bi-level hierarchy reduces the outer search to per-block budgets, enabling greedy optimization of the model-level objective directly.Lazy evaluation removes redundant objective queries.

6 Conclusion

Q-STRATA separates within-block proxy ranking from across-block model-level allocation, achieving strong low-bit MoE results and efficient dense-model search.

  • Q-STRATA caches one Pareto-optimal within-block candidate per budget level, leaving the outer stage to choose one budget per block.
  • Direct model-level optimization in the outer stage captures inter-block coupling that additive proxies miss.
  • Across three MoE LLMs, Q-STRATA achieves lower WikiText2 perplexity than uniform GPTQ, MxMoE, and GEMQ in the low-bit regime.The gains over MxMoE persist on the larger Qwen3-30B-A3B.
  • The outer search also works as a standalone dense-model allocator, matching AMQ with far fewer objective queries.

Limitations

Q-Strata directly evaluates a model-level objective during outer allocation, increasing search-time cost relative to proxy-only methods.

  • Limitations: The outer stage evaluates the model-level objective with forward passes of the assembled quantized model.This direct evaluation captures the objective after combining block assignments.
  • Limitations: Outer-search evaluation count grows with the number of MoE blocks L rather than the total number of expert linear layers 3LE.Evaluations per move stay near 2.5 and below L.
  • Limitations: Although incurred once before deployment, Q-Strata remains heavier at search time than proxy-only allocation.Reducing this search cost is left to future work.

A Empirical diminishing returns

The lazy descent relies on diminishing returns: marginal loss increases should grow as the model is compressed, an empirical trend observed across five trials.

  • A Empirical diminishing returns: The exactness condition requires per-block marginal loss increases δ_l(β) not to shrink as compression increases.Equivalently, the marginal gain from restoring one level must diminish as the model becomes more precise.
  • A Empirical diminishing returns: Five sampled descent chains measured marginal loss increases by raising one block level and comparing calibration-set objectives.Configurations were minmax-normalized within each chain and plotted against effective bitwidth.
  • A Empirical diminishing returns: All five curves rose near-monotonically as bitwidth fell, so cached marginal losses increase along the descent.This supports using earlier cached values as underestimates for the lazy heap.
  • A Empirical diminishing returns: The model-level objective is not strictly DR-submodular, and a few small reversals occur despite the broadly monotone trend.The cached values nevertheless serve as the lazy heap’s underestimates at the reported per-move cost.

B Experimental details

Experiments use GPTQ-based group-128 asymmetric weight-only quantization, allocate bits only across expert-FFN linears, and evaluate several budgeted baselines and search variants.

  • B Experimental details: Experiments cover Mixtral-8×7B-Instruct, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B for Table 13.Quantizer settings include 1, 2, 3, and 4 bits, with an optional offline random Hadamard rotation.
  • B Experimental details: Only expert-FFN linear layers receive mixed-precision allocation; routing gates, attention projections, and other linear layers remain at 16 bits.This scope is shared by MxMoE, GEMQ, and Q-Strata in Table 1.
  • B Experimental details: WikiText2 calibration uses sequence length 4096, with 128 sequences for final GPTQ, 64 for inner searches, and 32 for Q-Strata’s outer objective.MxMoE and GEMQ use their corresponding stated calibration procedures.
  • B Experimental details: Evaluation reports WikiText2 perplexity and the average of six zero-shot accuracies at context length 4096.The JSD objective uses top-1000 token logits for memory efficiency in MoE experiments.

D.1 Fidelity of the calibration objective

The calibration objective closely tracks final perplexity, Q-Strata remains stable across calibration subsets, and its non-uniform allocations preserve downstream capabilities better than uniform GPTQ at equal budget.

  • D.1 Fidelity of the calibration objective: ρ = 0.98 without rotation and 0.99 with rotation between model-level JSD rankings and final WikiText2 perplexity rankings.The comparison uses 32 random same-budget allocations on DeepSeek-V2-Lite at 2.0 average bits.
  • D.1 Fidelity of the calibration objective: Q-Strata gains 15.7 to 17.9 MMLU points over uniform GPTQ at the same 2.0-bit budget across all three models.Uniform GPTQ remains at chance level, while Q-Strata also raises GSM8K from at most 0.1 to 3.7–8.1.
  • D.1 Fidelity of the calibration objective: Q-Strata achieves the best MMLU among compared quantized methods on all three models and the best GSM8K on two of three.DeepSeek-V2-Lite is statistically tied with MxMoE on GSM8K, 8.1 versus 8.4 within one standard error.
  • D.4 Stability across calibration subsets: Across fresh Qwen1.5-MoE calibration subsets, perplexity agrees within 0.08 at every budget.This indicates the full search outcome is not sensitive to the particular calibration subset in that experiment.
  • D.5 Rotation at extreme bitwidths: At 1.75 bits, rotation-free Q-Strata wins on Mixtral, the settings tie on DeepSeek-V2-Lite, and rotation wins on Qwen1.5-MoE.The Mixtral comparison is 10.22 versus 12.14 perplexity; 1-bit assignments cover 25–59% of expert linears at these budgets.

D.6 Scalability and search cost

Q-Strata’s outer-stage gains persist on Qwen3-30B-A3B, while its search is a one-time cost that produces allocations across the full budget range. Deployment requires one GPTQ pass, and allocation changes quality rather than decode speed at fixed budget.

  • Scalability: Q-STRATA achieves lower WikiText2 perplexity than the inner-only MxMoE baseline at all three budgets on Qwen3-30B-A3B.The model contains 18,432 expert linear layers across 48 blocks.
  • Search cost: One Stage-2 descent sweeps the entire budget range and emits an allocation at every target bitwidth.Thus, the reported row-level search cost covers all budgets for that model.
  • Search cost: The search is a one-time offline cost, while producing the deployed model requires a single GPTQ pass taking less than half an hour on one H100.Final evaluation cost is identical across compared methods, and search cost scales linearly with calibration-set size.
  • Deployment footprint and throughput: At fixed budget, the two allocations differ in decode speed by about 1%, so allocation primarily affects quality in these measurements.Quantized models reduce model size by 5.2 to 6.3× and speed decoding by about 1.46× over BF16.

E Budgeted descent for dense models

For dense models, Q-Strata’s budgeted descent accounts for unequal layer sizes by ranking bitwidth reductions according to loss increase per budget saved. The dense evaluation uses HQQ quantizers and compares methods under a shared JSD objective and calibration data.

  • Budgeted descent: Because dense layers have different parameter counts, lowering a large layer by one level frees more budget than lowering a small layer.This motivates normalizing the loss increase by the budget released.
  • Budgeted descent: Dense descent ranks each one-level reduction by loss increase per unit of budget freed, rather than loss increase alone.The ranking uses ΔL_i/(w_iΔ), where w_i is the layer’s parameter count and Δ is grid spacing.
  • Budgeted descent: The algorithm changes its min-heap key to the loss-increase-per-budget ratio and stops when average bitwidth reaches the target.Lazy refresh and staleness checking remain unchanged.
  • Dense evaluation: The dense protocol uses asymmetric HQQ quantizers at 2, 3, and 4 bits with group size 128 and evaluates WikiText2 perplexity at context length 2048.Methods optimize the same JSD objective on the same calibration data while differing in evaluation count.
  • Search protocol: A single greedy descent from 4.25 to the 2.5-bit target passes through every intermediate budget and yields the whole frontier.One-shot ILP uses (K − 1)L + 1 measurements and re-solves per budget without further query cost.
Loading 2608.30564v1…