PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization

Renjie Xie, Juncheng Yang, Aoting Hu, Yan Liang, Mingxi Zhang, Zheheng Hong, Wei Xu, Yanzhi Chen

arXiv:2610.20353v1cs.LGstat.ML

TL;DR

Sparse routing reduces per-token computation, but MoE models still require storage for all expert weights, and rankings alone do not specify how strongly their errors should count. NatScale calibrates gradient-based expert weights to a model-specific contrast target, combines them with joint projection errors and a structural prior, and allocates precision under a storage budget. Across nine settings, it achieved the lowest native-to-quantized KL divergence and highest top-token agreement, and the lowest perplexity in seven.

  • Problem

    MoE models must store all expert weights despite sparse routing, and expert rankings alone do not determine how strongly to weight their errors in budgeted precision allocation.

    Source: §1 · p. 1

  • Method

    NatScale applies a rank-preserving, model-specific power transformation to calibrate gradient-based expert sensitivities to an information-theoretic contrast target.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    At the fixed one-nat target, NatScale achieved the lowest KLD and highest top-token agreement in all nine comparisons with five adapted baselines, and the lowest PPL in seven.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Contrast sweeps show that native-output fidelity and perplexity can favor different targets, highlighting the importance of objective-aware precision allocation.

    Source: §6 · p. 9

    NatScale adds computation to estimate sensitivities and select precision, depends on calibration data reflecting deployment inputs, and uses approximate error estimates that may miss component interactions or downstream-task changes.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.

1 Introduction

MoE quantization must reduce stored expert weights under a memory budget, but expert rankings alone do not say how strongly their errors should affect allocation. NatScale calibrates that contrast while preserving rankings, combines calibrated sensitivities with joint output errors and structural priors, and achieves the best native-output fidelity across nine tested settings.

Motivation

  • All expert parameters must be stored despite sparse token routing, motivating mixed precision that assigns more bits to sensitive weights under a memory budget.The cited prior work estimates quantization sensitivity using gradients or curvature and ranks experts using routing, reconstruction, activation, or weight-spectrum signals.Source: §1 · p. 1

The allocation gap

  • Expert rankings alone do not determine memory allocation: sensitivity contrast controls how strongly experts compete for storage.The score vectors (2, 1) and (100, 1) rank experts identically but imply different relative error weights.Source: §1 · p. 1

NatScale

  • NatScale power-transforms gradient-based sensitivities to a model-specific Jeffreys-divergence target, setting contrast while preserving expert rankings.The power transformation is order-preserving, and the target provides a common information-theoretic scale.Source: §1 · pp. 1–2
  • NatScale scores configurations using jointly measured gate, up, and down output errors multiplied by calibrated sensitivities and a structural prior.The prior incorporates weight spectra, variance, and precision decay; joint measurement captures interactions between gate and up outputs before the down projection.Source: §1 · p. 2

Evaluation

  • NatScale achieved the best native-output fidelity among tested methods in all nine WikiText-2 settings across three models and three storage targets.FlashNext was also evaluated on AIME, GPQA, TruthfulQA, and LiveCodeBench.Source: §1 · p. 2

2 Related Work

Prior work addresses quantization sensitivity, expert-level precision allocation, and calibration robustness through different signals and objectives. NatScale targets the strength of gradient-based expert weights and combines calibrated weights with joint configuration errors and a structural prior.

Post-Training Quantization

  • Dense-model post-training methods use calibration activations, codebooks, gradients, or curvature, while dense precision-allocation methods do not directly specify expert-level choices under sparse routing.Examples include GPTQ, AWQ, QuIP#, and AQLM.Source: §2 · p. 2

MoE Quantization

  • MoE allocation methods use routing and error signals, reconstruction costs, dynamic sensitivity, or weight spectra to estimate expert or block importance.NatScale instead calibrates gradient-based expert weights to a prescribed Jeffreys divergence, jointly evaluates configurations, and combines output errors with calibrated weights and AlphaQ’s structural factor.Source: §2 · p. 2

Calibration and Robustness

  • Calibration performance can depend on dataset selection, sequence length, language diversity, and expert-balanced sampling; NatScale remains dependent on calibration coverage and distribution shifts.The cited work also addresses group differences through worst-group loss or group-conditioned importance.Source: §2 · pp. 2–3

3 Methodology

NatScale formulates mixed-precision MoE quantization as a storage-constrained allocation, calibrates gradient-based expert sensitivities to a contrast target, and estimates errors by jointly quantizing projections. It combines these signals with structural priors and uses a relaxed optimization followed by feasible rounding to select configurations.

3.1 Overall Framework

  • For each expert, NatScale selects a gate, up, and down configuration to minimize a proxy for total model perturbation under a global byte budget.Non-routed weights retain native precision.Source: §3.1 · p. 3

3.2 Step 1: Sensitivity Calibration

  • NatScale measures prediction-loss response through expert-output gradients, then power-transforms positive sensitivities to a target symmetric KL contrast while preserving rank order.The default evaluation target is τ = 1 nat; zero-gradient experts retain zero weight.Source: §3.2 · p. 3

3.3 Step 2: Joint Error Estimation

  • Candidate distortion is measured by jointly quantizing gate, up, and down projections on replay inputs, then augmenting output error with a structural prior.The prior uses weight spectra, variance, and effective precision decay.Source: Figure 1 · p. 4, §3.3 · p. 4

3.4 Step 3: Budget Allocation

  • NatScale combines calibrated sensitivities, structural priors, and joint distortions into expert costs, then solves the budgeted allocation by LP relaxation and feasible rounding.The method constructs each expert’s lower convex hull over cost-payload trade-offs; changing the contrast target or budget can reuse cached distortion terms.Source: §3.4 · p. 4

4 Theoretical Analysis

NatScale maps gradient scores to a prescribed information-theoretic contrast while preserving expert rankings. The analysis gives this mapping a unique exponent, motivates one nat as a reference, and explains how contrast affects storage competition without determining which target best preserves outputs.

Well-posed contrast calibration

  • For positive, non-constant gradient scores, every target τ > 0 uniquely determines an exponent that changes weight contrast without changing expert rankings.The symmetric KL divergence varies continuously and monotonically with the exponent, from zero to infinity.Source: §4 · p. 5

Reference target

  • A Weibull reference fit yields exactly 1 nat of calibrated contrast for any positive, nonconstant amplitude sample, without assuming Weibull-distributed data.The derivation uses the unweighted two-parameter Weibull maximum-likelihood fit.Source: §4 · p. 5

Memory allocation

  • Wider contrast separates experts’ effective byte prices and can concentrate higher-precision storage on more heavily weighted experts.The allocation interpretation follows from dividing the expert selection objective by its calibrated weight.Source: §4 · p. 5

Scope and empirical assessment

  • The relation between contrast and storage competition does not determine which target best preserves model outputs, so the paper assesses the one-nat reference empirically.The gap between the returned feasible allocation and the LP optimum bounds optimization error for the fixed empirical proxy.Source: §4 · p. 5

5 Experiments

Across three MoE models and three routed-weight budgets, the experiments compare NatScale with adapted allocation baselines, test contrast and score components, and assess downstream and whole-model performance. NatScale leads on native-output fidelity in all nine routed settings, while the experiments also show metric- and task-dependent trade-offs.

5.1 Experimental Setup

  • The evaluation covers three MoE models at 2.0, 2.5, and 3.0 routed bpw, using shared candidate weights, calibration data, and native-model WikiText-2 references.It compares five adapted allocation baselines; full-model evaluation allows quantization changes to propagate through hidden states and routing.Source: §5.1 · pp. 5–6, §5 · p. 5

5.2 Quantization Performance

  • NatScale achieves the lowest KLD and highest Top1 among adapted baselines in all nine model-budget settings, and the lowest PPL in seven.At the middle Qwen3.8-Flash-Next budget, KLD falls from 0.2487 to 0.1941 versus AlphaQ while Top1 rises from 82.82% to 84.86%.Source: §5.2 · p. 6, Table 1 · p. 6
  • AlphaQ has lower PPL in two settings despite NatScale’s better KLD and Top1 there; on downstream tasks, NatScale leads on AIME, GPQA, and TruthfulQA, while MODE leads on LiveCodeBench.The downstream comparison uses Qwen3.8-Flash-Next at 2.5 routed bpw with thinking disabled.Source: §5.2 · pp. 6–7, Table 3 · p. 8

5.3 Ablation Studies

  • At DeepSeek-V4-Flash-0731 and 2.0 bpw, raising τ from 1 to 2 reduces KLD from 0.7293 to 0.6780 (7.03%) and raises Top1 by 1.02 percentage points.Across nine settings, τ = 2 has the smallest KLD in five, while τ = 1 has lower PPL than τ = 2 in eight; Figure 2 plots changes relative to τ = 1 against AlphaQ references.Source: §5.3 · p. 7, Figure 2 · p. 7
  • Removing gradient weights or precision-dependent bit decay worsens KLD and Top1 in every model-budget setting; removing bit decay also worsens PPL in all nine.At 2.5 bpw, each of the five tested component removals degrades both KLD and Top1.Source: §5.3 · p. 7
  • Joint projection-error scoring beats separate scoring on three of four downstream benchmarks, although separate scoring performs better on LiveCodeBench and neither approach consistently dominates across budgets.Joint scoring measures gate, up, and down projections together to capture combined perturbation effects that separate measurements may overlook.Source: §5.3 · pp. 7–8
  • At 2.5 bpw, NatScale assigns different effective precisions to experts within a layer, while visualized distributions describe how allocation rules divide storage across experts and layers.Layer-averaged budgets can obscure this within-layer variation.Source: §5.3 · p. 8

5.4 Whole-Model Storage and Quality

  • Under whole-model storage constraints on Qwen3.8-Flash-Next, NatScale has lower KLD and higher Top1 than available comparators across all three tiers.Compared with GSQ-RCO IQ3_XXS, NatScale reduces model size by 1.31 GB while lowering KLD by 11.04%; comparisons use different inference backends, and UD scores are externally reported.Source: §5.4 · p. 8, Table 2 · p. 8

6 Conclusion

NatScale calibrates expert-sensitivity contrast rather than relying on rankings alone, combining rank-preserving power-transformed weights with joint projection errors and a structural precision prior. At a one-nat target, it leads five adapted baselines on native-output fidelity in all nine comparisons and has the lowest PPL in seven.

Framework

  • NatScale explicitly calibrates the contrast between expert sensitivities instead of relying on expert rankings alone.Its model-specific power transformation preserves rankings while setting contrast to an information-theoretic target.Source: §6 · p. 8

Allocation design

  • NatScale combines calibrated weights with joint gate, up, and down output errors and a structural precision prior to allocate formats under a storage budget.The prior summarizes weight spectra, variance, and precision decay.Source: §6 · p. 9

Results and scope

  • At the fixed one-nat target, NatScale has the lowest KLD and highest Top1 in all nine comparisons with five adapted baselines, and the lowest PPL in seven.Downstream and whole-model evaluations provide additional evidence of utility; contrast sweeps show native-output fidelity and perplexity can favor different targets.Source: §6 · p. 9

Limitations

The appendix analyzes contrast calibration, its energy reference and weight properties, and how calibrated sensitivity and residual estimates enter allocation costs. NatScale also incurs calibration cost, depends on representative calibration inputs, and uses approximate error estimates.

Limitations

  • Calibration adds computation during model preparation, depends on calibration data reflecting deployment inputs, and uses error estimates that may miss component interactions or downstream task changes.Input-distribution or expert-usage shifts may affect allocation quality.Source: Limitations · p. 9

A Contrast Calibration

  • The calibration analysis assumes nonconstant log sensitivities on positive support and defines contrast through a Jeffreys-divergence target.The power transformation preserves expert rankings while calibrating their relative separation.Source: A Contrast Calibration · p. 12, §A.2 · p. 12

A.1 The Energy Reference

  • At β = 1/2, calibrated squared weights distribute energy in proportion to gradient scores; the exponential density maximizes entropy for nonnegative energy with fixed mean one.The reference is defined in the energy coordinate, not the amplitude coordinate.Source: §A.1 · p. 12

A.2 Entropy-Regularized Reweighting

  • The power family can be expressed as entropy-regularized reweighting, where increasing t strengthens preference for larger log sensitivities and t is set from the realized contrast.The unique maximizer is the stated distribution.Source: §A.2 · p. 12

A.3 Solving for the Exponent

  • For nonconstant positive gradients, every positive contrast target has a unique finite exponent, and that exponent increases with the target.Monotonicity supports solving for the exponent with a bracketed root finder.Source: §A.3 · p. 13

A.4 The Weibull Case

  • A full unweighted Weibull maximum-likelihood fit yields β = bk/4, corresponding to exactly one nat of contrast, without requiring Weibull-distributed data.The fitted shape determines the exponent; the common scale does not change allocation.Source: §A.4 · p. 13

A.5 Weight Entropy and Power Invariance

  • The linear sensitivity weights used in allocation have entropy bounded below and normalized values invariant to a common positive-power reparameterization of gradient scores.With other costs fixed, the resulting allocation is preserved.Source: §A.5 · pp. 13–14

B Residual Modeling and Sensitivity

  • The allocation cost combines calibrated sensitivity weights with candidate residuals; larger weights lower an expert’s effective byte price and strengthen its competition for storage.The bound motivating exposure, gradient amplitude, and residual amplitude does not determine the empirical choices of pooling, calibration, or precision factors.Source: B Residual Modeling and Sensitivity · pp. 14–15

C Implementation Details

Implementation prepares reusable native statistics and candidate distortion tables, then constructs normalized costs and solves a budgeted allocation problem. Joint projection errors are cached using reusable moments, and the returned discrete allocation is feasible and accompanied by an optimization-gap certificate.

Algorithm

  • The implementation collects native gradient moments and exposure, calibrates sensitivity weights, caches joint errors on fixed native inputs, computes structural factors and payloads, then solves and verifies a feasible allocation.Preparation statistics and candidate tables can be reused when changing contrast targets or payload ceilings.Source: C Implementation Details · pp. 15–16

Cost normalization

  • Subtracting complete cost-row minima and dividing by a positive global scale preserve the minimizing allocation.The normalization does not center local MSE before taking its square root.Source: C Implementation Details · p. 15

Structural factor

  • The structural factor combines spectral information, population weight variance, and precision decay, while budget accounting uses actual candidate payload.The adaptation fits the largest 10% of complete 128 × 128 block spectra and uses spectral exponent one.Source: C Implementation Details · p. 16

Moment evaluation

  • Joint gate/up and down candidate errors are computed from reusable activation and weight moments, retaining off-diagonal moments and the cross term without replaying every token for every configuration.The identity is exact for the specified local operator; six retained estimates agreed with direct evaluation within 4.40 × 10^-6 relative discrepancy.Source: C Implementation Details · p. 16

LP relaxation and rounding

  • The solver relaxes discrete choices to an LP for a lower bound, rounds the boundary solution through feasible upgrades, and returns a feasible map with its measured gap to the LP optimum.A feasible parent configuration is retained if its objective is strictly smaller.Source: C Implementation Details · pp. 16–17

D Evaluation Setup

The evaluation compares NatScale and adapted baselines on shared candidate formats and a common native-reference protocol across three MoE models. Calibration statistics are reused across settings, while reported storage targets and model parameter counts have defined scope boundaries.

Shared evaluation protocol

  • The evaluation covers Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, and DeepSeek-V4-Flash-0731, using a common token stream, scored positions, and native reference within each model.KLD measures native-to-quantized distribution divergence, Top1 measures native argmax agreement, and PPL pools true next-token negative log-likelihood.Source: D Evaluation Setup · pp. 17–18

Calibration

  • Calibration uses a frozen 1,048,576-token pool with equal weight across six folds, and reuses native moments, routing frequencies, candidate weights, and joint errors across NatScale variants.Changing threshold or budget requires no new collection of statistics.Source: D Evaluation Setup · pp. 17–18

Shared candidates and baselines

  • NatScale and the five external baselines use shared MFQ candidate formats, while retaining each baseline’s own scoring rule and allocation constraints.NatScale evaluates mixed gate/up/down configurations and keeps non-routed weights native; DeepSeek has no integer candidates above its native MXFP4 endpoint.Source: D Evaluation Setup · p. 17

Baseline adaptations

  • Baseline adaptations differ: MxMoE ties gate and up formats, FRI-MxMoE interpolates measured expert anchors, AlphaQ combines structural factors, and MODE substitutes text sources for modalities.The GEMQ allocation-only adaptation also shares one candidate format across gate, up, and down projections.Source: D Evaluation Setup · pp. 17–18

Storage and model scope

  • Storage budgets cover routed-expert weights and use nominal routed-bpw targets, but NatScale constrains payload and reports headers separately while adapted baselines reserve header bytes within their targets.For Qwen3.8-Flash-Next, the reported 125B excludes 51B ngram embedding and 4B MTP parameters; the 0731 checkpoint also contains DSpark.Source: D Evaluation Setup · pp. 17–18

E Precision Allocation

The allocation figures show how NatScale distributes precision across experts and layers, while the contrast-target ablation compares fidelity and perplexity under fixed candidate weights and payload ceilings. The supplied passages describe the figure encodings and comparison scope but do not report the table’s specific values.

E Precision Allocation

  • Figures 3–5 compare NatScale with MxMoE, FRI-MxMoE, AlphaQ, and MODE at 2.5 nominal routed bpw across three models.The figures use a common method order and color scale while preserving each model’s original expert order.Source: E Precision Allocation · pp. 18–19
  • NatScale’s expert precision varies within layers, while cumulative distributions summarize the spread of storage across experts.Heatmaps show stored bits per combined gate/up/down parameter count, including overhead; layer curves show expert storage per routed parameter count.Source: E Precision Allocation · p. 19

F.1 Contrast Targets

  • The contrast-target ablation reports KLD, Top1, and PPL to show how preferences for contrast differ between likelihood and native-output fidelity.The comparison varies the gradient exponent while keeping raw joint RMS, structural factors, fold frequencies, candidate weights, and payload ceilings fixed.Source: §F.1 · p. 19, F Design Ablations · p. 19

F.2 Component Ablations

The ablations show that bit decay and gradient weighting contribute consistently, while the effects of local RMS error, gradient smoothing, exposure magnitude, and spectral factors vary by setting. Separate-error scoring changes the metric trade-offs relative to the joint-error default.

F.2 Component Ablations

  • Removing bit decay consistently worsens all three metrics across the nine model–budget settings.Removing gradient weights worsens KLD and Top1 in every setting, though it improves PPL in two.Source: §F.2 · p. 19
  • Removing exposure magnitude improves KLD once and PPL twice, but never improves Top1; on Qwen3.8-Flash-Next, removing the spectral factor improves all three metrics at 3.0 bpw.Source: §F.2 · p. 19
  • For Qwen3.8-Flash-Next, the MSE variant also removes bit decay and differs from the default in two factors; comparing it with RMS without bit decay isolates error power under that condition.The other three controls retain bit decay while removing the spectral factor, the variance factor, or both.Source: §F.2 · p. 19

F.3 Joint and Separate Errors

  • The separate-error control sums single-projection errors, changing one projection at a time while the other two remain native; both scoring variants retain the same deployment configurations.The joint-error default retains interactions among gate, up, and down perturbations.Source: §F.3 · p. 20
  • Separate-error scoring raises Top1 from 84.8573% to 84.9175% at 2.5 bpw but increases KLD from 0.194086 to 0.195639.At 3.0 bpw, separate scoring has lower KLD while joint scoring has higher Top1; separate scoring also lowers KLD at 2.0 bpw and raises Top1 at 2.0 and 2.5 bpw.Source: §F.3 · p. 20

G Downstream Evaluation

The downstream comparison evaluates joint and separate-error NatScale on Qwen3.8-Flash-Next without task-specific refitting. The accompanying whole-model extension defines allocation units and candidate scores, then applies them under a storage constraint that includes fixed tensors and metadata.

G Downstream Evaluation

  • Joint scoring achieves higher scores on AIME, GPQA, and TruthfulQA, while separate scoring performs better on LiveCodeBench.Both comparisons use fixed allocations and candidate weights from WikiText-2 evaluation without task-specific refitting.Source: G Downstream Evaluation · pp. 21–22

H.1 Allocation Units and Scores

  • The whole-model extension allocates formats across 24,768 units and 74,304 projections, including routed and shared experts, attention groups, and MHC groups.Within each unit, projections may receive different encodings from the 14-format catalog.Source: §H.1 · pp. 23, 25
  • Routed and shared experts are scored using joint gate/up/down MSE including cross terms, while attention and MHC scores sum single-projection MSEs without cross terms.The score also incorporates structural factors, exposure and sensitivity weights calibrated from native output gradients.Source: §H.1 · pp. 25–26

H.2 Storage Budget

  • Whole-model allocation counts fixed-encoding tensors and a separate reserve for headers and codebooks in its global byte constraint.Allocation uses lower-hull selection and feasible discrete rounding without per-layer or projection-type ceilings.Source: §H.2 · p. 26

H.3 Evaluation and Results

The whole-model evaluation compares NatScale and released GSQ-RCO checkpoints on shared WikiText-2 contexts using native-reference KLD and Top1 agreement. The paper also states that generative AI assisted several writing and review tasks, with authors verifying the work and taking responsibility.

H.3 Evaluation and Results

  • NatScale and four released GSQ-RCO checkpoints are compared using 580 frozen WikiText-2 contexts and 147,900 predictions, with full-vocabulary KLD and native-argmax Top1 agreement.The comparison uses shared native reference hidden states and output head; each method runs on its corresponding backend.Source: §H.3 · p. 26

AI use statement

  • The authors report generative-AI assistance with language editing, literature discovery, LaTeX formatting, and analysis-code review, while taking responsibility for the content and artifacts.They state that they verified sources, code, and numerical claims and reviewed the manuscript.Source: AI use statement · p. 26

Related papers

Loading 2610.20353v1…