PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs

Deokjae Lee, Sihun Chu, Hyun Oh Song

arXiv:2608.30564v1cs.LGcs.AI

TL;DR

MoE models make mixed-precision quantization difficult because their expert layers create a very large allocation space, while existing methods do not directly optimize coupled block choices. Q-STRATA uses a bi-level design that ranks assignments within blocks with a cheap proxy and allocates budgets across blocks using the model-level objective, achieving lower low-bit WikiText2 perplexity than the compared methods.

  • Problem

    MoE models have many expert linear layers, and existing methods either enforce uniform per-block budgets or allocate globally with an additive proxy rather than directly optimizing coupled model-level choices.

    Source: §1 · pp. 1–2

  • Method

    Q-STRATA caches Pareto-frontier within-block assignments over budget levels, then selects one budget per block by directly optimizing the model-level objective with lazy greedy descent.

    Source: §1 · p. 2, §2.2 · p. 3, §3.1 · p. 4

  • Results

    Q-STRATA achieves lower WikiText2 perplexity than uniform GPTQ, MxMoE, and GEMQ across three MoE LLMs in the low-bit regime.

    Source: §4.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The hierarchical allocation captures inter-block coupling that additive proxies miss, while the outer search also functions as a standalone dense-model allocator with far fewer objective queries than black-box search.

    Source: §1 · p. 2, §6 · p. 9

    The outer stage is heavier at search time than proxy-only allocation because it repeatedly evaluates the assembled quantized model, and reducing this cost remains future work.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-induced quality loss under a fixed budget, but Mixture-of-Experts (MoE) models contain these layers in every expert of every MoE block, so the allocation space grows far larger than in a dense model. Existing methods either allocate within each block under a uniform per-block budget, or allocate across blocks through an additive proxy, and neither directly optimizes a model-level objective over the choices that couple the blocks. We propose Q-Strata, a bi-level allocator that ranks within-block assignments with a cheap proxy and allocates across blocks with a model-level objective evaluated on the assembled quantized model. Its inner stage caches a Pareto frontier of candidates per block over finely spaced budgets, leaving the outer stage to set one budget per block instead of a bitwidth for every linear layer. With the search reduced to one budget per block, the outer stage optimizes this model-level objective directly, capturing the inter-block coupling that additive proxies miss. On Mixtral-8x7B-Instruct, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite, Q-Strata consistently achieves lower WikiText2 perplexity than uniform-bitwidth GPTQ and the state-of-the-art MoE MPQ methods MxMoE and GEMQ in the low-bit regime. The code is available at https://github.com/snu-mllab/Q-Strata/tree/main.

1 Introduction

MoE models make weight-only mixed-precision quantization especially challenging because their many expert layers create a huge allocation space. Q-STRATA addresses this by ranking assignments within blocks cheaply and allocating budgets across blocks using a direct model-level objective.

Motivation

  • MoE models store all experts although only a few are active per token, making weight compression important for practical deployment.The paper focuses on weight-only post-training quantization using a small calibration dataset without retraining.Source: §1 · p. 1

Challenge

  • 3LE linear layers across L MoE blocks and E experts create an assignment space of |Q|^3LE, far larger than in dense models.Each expert contains gate, up, and down projections.Source: §1 · p. 1
  • Qwen3-30B-A3B has 18,432 MoE-block linear layers versus 448 in dense Qwen3-32B, exceeding forty times the dense count.The comparison excludes attention and routing from the MoE count.Source: §1 · p. 2

Prior limitations

  • Existing MoE methods improve uniform quantization with proxies but either keep block budgets uniform or use an additive global proxy.These approaches do not directly optimize the coupled model-level objective across blocks.Source: §1 · pp. 1–2

Q-STRATA

  • Q-STRATA caches Pareto-optimal within-block candidates and selects one budget per block to minimize the assembled model-level objective under a global budget.The outer search therefore operates over L per-block budgets instead of 3LE individual bitwidths.Source: §1 · p. 2, Figure 1 · p. 2

2 Preliminaries

The paper formulates MoE MPQ as minimizing a model-level quality objective under an average-bitwidth constraint, while exploiting separable within-block structure for tractable candidate generation.

Problem formulation

  • MoE MPQ assigns one candidate quantizer from Q to each expert projection, forming local block assignments and a global assignment.Each block contains E experts with up, gate, and down projections.Source: §2.1 · p. 3
  • The storage cost includes quantized layer bits plus scale and zero-point overhead, and block and model average bitwidths aggregate these costs.For structurally identical blocks, model average bitwidth reduces to the mean of per-block bitwidths.Source: §2.1 · p. 3
  • MPQ minimizes the model-level objective L at target average bitwidth τ, using Jensen–Shannon divergence between quantized and full-precision next-token distributions.Lower JSD indicates higher quality, and full-precision distributions are precomputed once.Source: §2.1 · p. 3
  • The assignment space is especially difficult for MoE models because its dimension N = 3LE scales with the number of experts.This motivates exploiting the model’s two-level block structure.Source: §2.1 · p. 3

Block reconstruction proxy

  • Within-block reconstruction error is coupled by gating and routing, so prior methods approximate it with a decomposable sum of isolated layer distortions.The resulting proxy is block-separable and cheap because each term requires only a single-block forward.Source: §2.2 · pp. 3–4

Q-STRATA allocation

  • Q-STRATA’s inner stage caches Pareto-optimal assignments across per-block budgets, while its outer stage chooses one cached candidate per block using L directly.The inner stage is a multiple-choice knapsack problem over the 3E layers and candidate quantizers.Source: §2.2 · p. 4

3 Method

Q-STRATA solves MoE mixed-precision allocation in two stages: a proxy-based inner stage builds per-block candidates, while a model-level outer stage selects budgets across blocks. Lazy greedy descent makes the outer search tractable by exploiting empirically supported diminishing returns.

3.1 Inner stage

  • The inner stage precomputes Pareto-optimal assignments for each MoE block across a shared grid of budgets using a cheap proxy.The proxy allocation is solved as a multiple-choice knapsack problem over the block’s linear layers and candidate quantizers.Source: §3 · p. 4, §3.1 · p. 4
  • Caching one candidate per budget level reduces the global search from Q^3LE assignments to one budget choice per block.The cached candidates form the per-block sets used by the outer stage.Source: §3.1 · p. 4

3.2 Outer stage

  • The outer stage assembles cached candidates and directly minimizes the non-additive model-level objective under a global budget constraint.Because the objective does not decompose across blocks, the outer selection cannot be solved as a multiple-choice knapsack problem.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3 Solving the outer problem

  • Lazy greedy descent starts from the most expensive allocation and repeatedly lowers the block with the smallest marginal loss increase until the target average bitwidth is reached.Each one-level move saves the same budget, and every candidate move is evaluated on the assembled model.Source: §3.3 · p. 5
  • The lazy implementation caches marginal losses in a min-heap, recomputing stale values before committing a move.It reproduces eager descent when stale losses underestimate current values, a condition justified empirically but not guaranteed for the model objective.Source: §3.3 · p. 5
  • O(RLK) model evaluations are required in lazy descent versus O(L^2K) for eager descent, with R near 2.5 in practice.The method obtains assignments for every target budget along a single top-to-bottom sweep.Source: §3.3 · pp. 5–6

4 Experiments

Experiments show that Q-STRATA improves low-bit MoE quantization, that direct outer-stage optimization captures inter-block coupling, and that its search remains effective for dense models.

4.2 Main results

  • Q-STRATA achieves the lowest WikiText2 perplexity for every evaluated model and target bitwidth, with the best zero-shot accuracy except one case.At DeepSeek-V2-Lite and 2.25 bits, MxMoE leads accuracy by 0.3 points.Source: §4.2 · p. 7
  • At 1.75 bits on Mixtral, Q-STRATA reduces perplexity from MxMoE’s 25.20 to 12.14 and improves average accuracy by almost eight points.Source: §4.2 · p. 7
  • Q-STRATA’s gains over MxMoE widen as the budget tightens because the outer stage directly evaluates the model-level objective across blocks.MxMoE shares Q-STRATA’s inner stage but uses uniform block budgets.Source: §4.1 · p. 6, §4.2 · p. 7

4.3 Outer-stage ablation

  • The outer-stage lazy greedy method attains the lowest JSD and perplexity at every tested bitwidth, with a 74-JSD advantage over one-shot ILP at 1.75 bits.One-shot ILP uses a separable proxy and therefore ignores inter-block coupling.Source: §4.3 · p. 7
  • The pruning-free direct cell-level ILP requires more than eight times Q-STRATA’s model-level evaluations, while descent reaches JSD 364 versus 410–419 for both one-shot solvers.Source: Does the inner-stage pruning lose… · pp. 7–8

4.4 Robustness checks

  • With router fine-tuning restored and applied equally, Q-STRATA has lower perplexity than GEMQ in all 18 evaluated pairs.Rotation is not consistently better at low budgets; the strongest Mixtral results there are rotation-free.Source: §4.4 · p. 8

4.5 The outer search on dense models

  • On dense Llama-2-7B, lazy greedy matches or beats AMQ while using 6.9× fewer objective queries at 8K calibration tokens.At 2K tokens, it stays within 0.001 JSD of eager greedy using 51× fewer queries.Source: §4.5 · pp. 8–9, Figure 3 · p. 9

5 Related Work

Prior MoE quantization methods use local or additive proxies, while Q-STRATA instead exploits a hierarchical search space and evaluates objective marginals directly at the outer level.

  • MC-MoE and MxMoE score within-block assignments with proxies but keep every MoE block’s budget uniform.Source: §5 · p. 9
  • GEMQ allocates at expert granularity across blocks using an additive proxy for model quality.Source: §5 · p. 9
  • ScaleBits also uses greedy allocation and diminishing returns, but searches a much finer tile-level space with sensitivity estimates rather than measured objective marginals.Source: §5 · p. 9
  • Q-STRATA’s bi-level hierarchy reduces the outer search to per-block budgets, enabling greedy optimization of the model-level objective directly.Lazy evaluation removes redundant objective queries.Source: §5 · p. 9

6 Conclusion

Q-STRATA separates within-block proxy ranking from across-block model-level allocation, achieving strong low-bit MoE results and efficient dense-model search.

  • Q-STRATA caches one Pareto-optimal within-block candidate per budget level, leaving the outer stage to choose one budget per block.Source: §6 · p. 9
  • Direct model-level optimization in the outer stage captures inter-block coupling that additive proxies miss.Source: §6 · p. 9
  • Across three MoE LLMs, Q-STRATA achieves lower WikiText2 perplexity than uniform GPTQ, MxMoE, and GEMQ in the low-bit regime.The gains over MxMoE persist on the larger Qwen3-30B-A3B.Source: §6 · p. 9
  • The outer search also works as a standalone dense-model allocator, matching AMQ with far fewer objective queries.Source: §6 · p. 9

Limitations

Q-Strata directly evaluates a model-level objective during outer allocation, increasing search-time cost relative to proxy-only methods.

Limitations

  • The outer stage evaluates the model-level objective with forward passes of the assembled quantized model.This direct evaluation captures the objective after combining block assignments.Source: Limitations · p. 10
  • Outer-search evaluation count grows with the number of MoE blocks L rather than the total number of expert linear layers 3LE.Evaluations per move stay near 2.5 and below L.Source: Limitations · p. 10
  • Although incurred once before deployment, Q-Strata remains heavier at search time than proxy-only allocation.Reducing this search cost is left to future work.Source: Limitations · p. 10

A Empirical diminishing returns

The lazy descent relies on diminishing returns: marginal loss increases should grow as the model is compressed, an empirical trend observed across five trials.

A Empirical diminishing returns

  • The exactness condition requires per-block marginal loss increases δ_l(β) not to shrink as compression increases.Equivalently, the marginal gain from restoring one level must diminish as the model becomes more precise.Source: A Empirical diminishing returns · p. 11
  • Five sampled descent chains measured marginal loss increases by raising one block level and comparing calibration-set objectives.Configurations were minmax-normalized within each chain and plotted against effective bitwidth.Source: A Empirical diminishing returns · pp. 11–12
  • All five curves rose near-monotonically as bitwidth fell, so cached marginal losses increase along the descent.This supports using earlier cached values as underestimates for the lazy heap.Source: A Empirical diminishing returns · p. 12
  • The model-level objective is not strictly DR-submodular, and a few small reversals occur despite the broadly monotone trend.The cached values nevertheless serve as the lazy heap’s underestimates at the reported per-move cost.Source: A Empirical diminishing returns · p. 12

B Experimental details

Experiments use GPTQ-based group-128 asymmetric weight-only quantization, allocate bits only across expert-FFN linears, and evaluate several budgeted baselines and search variants.

B Experimental details

  • Experiments cover Mixtral-8×7B-Instruct, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B for Table 13.Quantizer settings include 1, 2, 3, and 4 bits, with an optional offline random Hadamard rotation.Source: B Experimental details · p. 12
  • Only expert-FFN linear layers receive mixed-precision allocation; routing gates, attention projections, and other linear layers remain at 16 bits.This scope is shared by MxMoE, GEMQ, and Q-Strata in Table 1.Source: B Experimental details · p. 12, Table 7 · p. 12
  • WikiText2 calibration uses sequence length 4096, with 128 sequences for final GPTQ, 64 for inner searches, and 32 for Q-Strata’s outer objective.MxMoE and GEMQ use their corresponding stated calibration procedures.Source: B Experimental details · p. 12
  • Evaluation reports WikiText2 perplexity and the average of six zero-shot accuracies at context length 4096.The JSD objective uses top-1000 token logits for memory efficiency in MoE experiments.Source: B Experimental details · p. 12

D.1 Fidelity of the calibration objective

The calibration objective closely tracks final perplexity, Q-Strata remains stable across calibration subsets, and its non-uniform allocations preserve downstream capabilities better than uniform GPTQ at equal budget.

D.1 Fidelity of the calibration objective

  • ρ = 0.98 without rotation and 0.99 with rotation between model-level JSD rankings and final WikiText2 perplexity rankings.The comparison uses 32 random same-budget allocations on DeepSeek-V2-Lite at 2.0 average bits.Source: §D.1 · p. 13, Table 8 · p. 13
  • Q-Strata gains 15.7 to 17.9 MMLU points over uniform GPTQ at the same 2.0-bit budget across all three models.Uniform GPTQ remains at chance level, while Q-Strata also raises GSM8K from at most 0.1 to 3.7–8.1.Source: §D.2 · pp. 13–14, Table 9 · p. 13
  • Q-Strata achieves the best MMLU among compared quantized methods on all three models and the best GSM8K on two of three.DeepSeek-V2-Lite is statistically tied with MxMoE on GSM8K, 8.1 versus 8.4 within one standard error.Source: §D.2 · p. 14

D.4 Stability across calibration subsets

  • Across fresh Qwen1.5-MoE calibration subsets, perplexity agrees within 0.08 at every budget.This indicates the full search outcome is not sensitive to the particular calibration subset in that experiment.Source: §D.4 · p. 14, Table 11 · p. 14

D.5 Rotation at extreme bitwidths

  • At 1.75 bits, rotation-free Q-Strata wins on Mixtral, the settings tie on DeepSeek-V2-Lite, and rotation wins on Qwen1.5-MoE.The Mixtral comparison is 10.22 versus 12.14 perplexity; 1-bit assignments cover 25–59% of expert linears at these budgets.Source: §D.5 · p. 14, Table 12 · p. 14

D.6 Scalability and search cost

Q-Strata’s outer-stage gains persist on Qwen3-30B-A3B, while its search is a one-time cost that produces allocations across the full budget range. Deployment requires one GPTQ pass, and allocation changes quality rather than decode speed at fixed budget.

Scalability

  • Q-STRATA achieves lower WikiText2 perplexity than the inner-only MxMoE baseline at all three budgets on Qwen3-30B-A3B.The model contains 18,432 expert linear layers across 48 blocks.Source: §D.6 · p. 14

Search cost

  • One Stage-2 descent sweeps the entire budget range and emits an allocation at every target bitwidth.Thus, the reported row-level search cost covers all budgets for that model.Source: Table 14 · p. 15
  • The search is a one-time offline cost, while producing the deployed model requires a single GPTQ pass taking less than half an hour on one H100.Final evaluation cost is identical across compared methods, and search cost scales linearly with calibration-set size.Source: §D.6 · pp. 14–15

Deployment footprint and throughput

  • At fixed budget, the two allocations differ in decode speed by about 1%, so allocation primarily affects quality in these measurements.Quantized models reduce model size by 5.2 to 6.3× and speed decoding by about 1.46× over BF16.Source: §D.7 · p. 15

E Budgeted descent for dense models

For dense models, Q-Strata’s budgeted descent accounts for unequal layer sizes by ranking bitwidth reductions according to loss increase per budget saved. The dense evaluation uses HQQ quantizers and compares methods under a shared JSD objective and calibration data.

Budgeted descent

  • Because dense layers have different parameter counts, lowering a large layer by one level frees more budget than lowering a small layer.This motivates normalizing the loss increase by the budget released.Source: E Budgeted descent for dense models · p. 15
  • Dense descent ranks each one-level reduction by loss increase per unit of budget freed, rather than loss increase alone.The ranking uses ΔL_i/(w_iΔ), where w_i is the layer’s parameter count and Δ is grid spacing.Source: E Budgeted descent for dense models · p. 15
  • The algorithm changes its min-heap key to the loss-increase-per-budget ratio and stops when average bitwidth reaches the target.Lazy refresh and staleness checking remain unchanged.Source: E Budgeted descent for dense models · p. 15

Dense evaluation

  • The dense protocol uses asymmetric HQQ quantizers at 2, 3, and 4 bits with group size 128 and evaluates WikiText2 perplexity at context length 2048.Methods optimize the same JSD objective on the same calibration data while differing in evaluation count.Source: E Budgeted descent for dense models · p. 15

Search protocol

  • A single greedy descent from 4.25 to the 2.5-bit target passes through every intermediate budget and yields the whole frontier.One-shot ILP uses (K − 1)L + 1 measurements and re-solves per budget without further query cost.Source: E Budgeted descent for dense models · p. 15

Related papers

Loading 2608.30564v1…