Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

Motivation

• MoE deployment remains memory- and bandwidth-constrained because the full expert pool must be hosted despite activating only a fraction of weights per token.These storage and memory-bound computation demands limit practical throughput and cost efficiency.Source: §1 · p. 1

Challenge

• Semi-structured sparsity on Sparse Tensor Cores offers a path to compress weight storage and accelerate computation, but large-scale MoE fidelity and execution remain difficult.Rigid sparsity patterns combined with low-precision quantization can degrade model quality.Source: §1 · pp. 1–2

Approach

• MoESQ relaxes discrete semi-structured mask choices for differentiable joint optimization of sparse supports and weights under a quantization-aware reconstruction objective.It also weights reconstruction by routing significance and distributes expert compression in parallel.Source: §1 · p. 2

Results

• MoESQ improves average task accuracy by up to 4.35 percentage points over the strongest joint sparse-quantization baselines, retaining 96.09% of uncompressed accuracy at trillion-parameter scale.Evaluations span MoE architectures from 30B to one trillion parameters.Source: §1 · p. 2
• On NVIDIA B200, the grouped sparse GEMM kernel achieves up to 1.65× speedup, supporting 1.18× higher serving throughput and up to 4.03× lower end-to-end latency.The throughput comparison is against the fastest dense low-precision backend; the latency comparison is against INT4 Marlin.Source: §1 · p. 2

MoE deployment

• Sparse routing limits per-token computation, but deployment must store the full expert pool, so expert-weight memory scales with total rather than active parameters.Kimi-K2.5 has 1T total parameters, activates 32B per token, and its packed INT4 checkpoint occupies approximately 595 GB.Source: §2 · p. 2

Hardware opportunity

• Blackwell Sparse Tensor Cores support paired 4:8 NVFP4, but routed MoE inference also needs weights matching the hardware representation and an execution path across dynamic experts.The sparse NVFP4 path offers up to 18 PFLOP/s, twice the dense NVFP4 throughput.Source: §2 · p. 3

Compression gap

• Prior joint sparsification and quantization methods were developed mainly for dense LLMs, leaving their hardware-native quality and trillion-parameter MoE scalability unclear.Examples include JSQ and OBR.Source: §2 · p. 3

Execution gap

• Available grouped GEMM kernels support low precision, but no publicly available sparse grouped GEMM path directly supports paired 4:8 NVFP4 weights.Grouped GEMM is needed to execute routing-dependent expert products together despite different token dimensions.Source: §2 · p. 3

Combined gap

• The background identifies a combined gap: no established path simultaneously preserves trillion-scale MoE fidelity and translates sparse low-precision hardware support into reduced storage and faster routed execution.Source: §2 · p. 3

3 METHOD

• MoESQ combines an algorithm for learning hardware-native sparse-quantized expert weights with a grouped sparse GEMM that executes the same representation on Sparse Tensor Cores.Source: §3 · p. 3

3.1 PROBLEM FORMULATION

• The formulation minimizes a targeted module’s reconstruction error subject to weights belonging to admissible sparsity-pattern and quantized-value sets.Support selection, quantized representation, and shared-scale estimation make the optimization discrete and combinatorial.Source: §3.1 · p. 3

3.2 LEARNING HARDWARE-NATIVE SPARSE-QUANTIZED REPRESENTATIONS

• Gumbel–Softmax relaxes discrete support choices into differentiable distributions; annealing drives them toward one-hot patterns, while soft-masked weights are projected into the quantized set.The loss updates support logits and auxiliary weights through a straight-through estimator for the non-differentiable projection.Source: §3.2 · p. 4

3.3 SCALABLE BLOCKWISE EXPERT COMPRESSION

• Blockwise compression reconstructs gate, up, and down expert projections using activation-quantized execution and a router-weighted reconstruction objective.Experts are optimized independently across devices, and each finalized block supplies calibration inputs for the next block.Source: §3.3 · p. 4

3.4 IMPLEMENTATION DETAILS

• Compression keeps router and non-expert parameters fixed, reuses routed calibration targets, shards experts across devices, and passes compressed-block states forward.For NVFP4, the procedure recomputes scales during refinement and packs E2M1 values, sparse metadata, and a UE4M3 scale per 32 logical weights.Source: §3.4 · p. 5
• The kernel extends grouped execution to paired 4:8 NVFP4 by combining Blackwell sparse MMA with grouped scheduling, and packs weights and quantized activations in corresponding scale blocks.Grouped GEMM executes routing-dependent expert problems within one launch; the sparse matrix is placed as operand A for sparse MMA.Source: §3.4 · pp. 5–6

4.1 SETUP

• Experts from Qwen3-30B-A3B, Qwen3.5-397B-A17B, and Kimi-K2.5 are compressed to paired 4:8 NVFP4 W4A4 and evaluated on general-knowledge and reasoning tasks.Joint sparse-quantization baselines include JSQ, SGPTQ, and OBR; kernel comparisons include NVFP4 CuTeDSL, INT4 Marlin, and INT2 Humming.Source: §4.1 · pp. 6–7

4.2 ACCURACY EVALUATION

• 91.44% overall performance recovery on Kimi-K2.5 reasoning tasks; MoESQ also maintains minimal truncation rates comparable to the uncompressed baseline.JSQ has an 88.3% truncation rate on AIME25 and 17.24% overall performance recovery.Source: Table 2 · p. 7, §4.2 · p. 7
• Across general-knowledge evaluations, MoESQ retains 93.70% of reference performance on Qwen3.5-397B-A17B and 96.09% on trillion-parameter Kimi-K2.5.It leads OBR by 5.93 recovery points on Qwen3-30B-A3B and 5.02 points on Kimi-K2.5.Source: §4.2 · p. 7

4.3 EFFICIENCY EVALUATION

• 1.35–1.65× speedup over dense grouped GEMMs across tested configurations, with energy to solution falling from 1,098 to 727 mJ at peak speedup.The kernel also outpaces dense FP4 and weight-only backends from decode-like to prefill-like token counts.Source: Figure 1 · p. 8, §4.3 · p. 8
• 1.18× decode-throughput increase over dense CuTeDSL, with mean TPOT reduced by 5.9 ms and end-to-end latency by 1.1 s under saturated serving.For prefill at concurrency 128, MoESQ reaches 55,654 tokens/s and beats the fastest baseline by 1.18×–1.25×.Source: §4.3 · p. 8, Figure 2 · p. 9

4.4 EXPERT COMPRESSION COMPARISON

• At comparable 2–3-bit-per-expert budgets, MoESQ recovers 96.09% of reference accuracy, beating dense 2-bit GSQ by 1.87 percentage points but trailing REAP by 1.90 points.It also achieves 8,983 tok/s output throughput and 6.2 s end-to-end latency in the matched decode comparison.Source: §4.4 · pp. 8–9, Table 3 · p. 9

• MoESQ combines differentiable joint sparsification and quantization with a dedicated grouped sparse GEMM kernel for hardware-native sparse low-precision execution.The conclusion reports that code, including the kernel, is available on GitHub and compressed models are released as a Hugging Face collection.Source: §5 · p. 9

Feasible set

• Each sparsity group must select an admissible pattern, while each scaling block must use a scale and values in the target sparse-quantized set.Pruned positions satisfy the quantization constraint because zero belongs to the value grid.Source: A SPARSE-QUANTIZED REPRESENTATION · p. 14

Blackwell paired 4:8 NVFP4

• For paired 4:8 NVFP4, each group of eight retains two adjacent pairs, and 16 retained values per 32-weight block share one UE4M3 scale.The six permitted patterns retain two of the four adjacent pairs; retained values use the E2M1 grid.Source: A SPARSE-QUANTIZED REPRESENTATION · p. 14, Figure 3 · p. 15

Storage layout

• The packed layout stores four retained values and a pattern index per eight-weight group, plus one UE4M3 scale per 32-weight block, for 2.75 bits per weight.Source: A SPARSE-QUANTIZED REPRESENTATION · p. 15

Compression scope

• Compression targets routed expert FFN weights in paired 4:8 NVFP4 W4A4 with 32-element scale blocks; attention, shared experts, routers, embeddings, and the LM head remain at reference precision.Source: B EXPERIMENTAL SETUP · p. 15

Baselines

• JSQ, SGPTQ, and OBR are adapted to the same sparse W4A4 target, but SGPTQ uses 512 calibration sequences while MoESQ adds 10 refinement epochs over 8,192 sequences.The comparison is of solution quality, not equal compression cost; OBR omits rotation because rotated experts cannot use the evaluated grouped sparse GEMM path.Source: B EXPERIMENTAL SETUP · p. 15

Calibration data

• All compression stages use the same 10:45:45 mixture of compression-calibration, OpenThoughts-114k, and FineWeb-Edu documents packed into 4,096-token sequences.SGPTQ initialization uses 512 sequences; refinement uses 8,192, with 128 held out for validation.Source: B EXPERIMENTAL SETUP · p. 15

Compression hyperparameters

• Table 4 reports per-model optimization hyperparameters for the router-weighted objective, including support-logit and latent-weight learning rates and the annealed Gumbel temperature schedule.Source: B EXPERIMENTAL SETUP · p. 16, Table 4 · p. 16

Evaluation protocol

• Accuracy is evaluated through vLLM on OpenLLM Leaderboard v1 for all three models, with Kimi-K2.5 also tested on AIME25, GPQA Diamond, and MATH500.Reasoning Pass@1 averages 10, 5, and 5 runs, respectively, using temperature 1.0, top-p 0.95, and a 65,536-token generation budget.Source: B EXPERIMENTAL SETUP · p. 16

Ablation design

• All ablation arms share the SGPTQ initialization, calibration setup, and 10-epoch refinement budget; the full configuration corresponds to the Qwen3-30B-A3B setup in Table 4.The refinement stage permits additional optimization after the one-shot SGPTQ procedure, at higher compression cost.Source: §C.1 · p. 16

Router-weighted reconstruction objective

• 65.44 versus 64.68 OpenLLM average accuracy, and 89.30% versus 88.26% recovery, favor router-weighted over uniform reconstruction, with gains on all six tasks.Learning rates were tuned separately for the two objectives, rather than matched despite their different gradient scales.Source: §C.1 · p. 16

Support–value adaptation

• 65.44 versus 64.44 average accuracy, a 1.00-point gain, results from jointly adapting support logits and latent weights rather than adapting latent weights with the initial support fixed.Source: §C.1 · p. 16

Comparison scope

• Unlike Tables 1 and 2, which compare methods under the same sparse W4A4 target, this comparison spans different compression mechanisms and deployment formats.Source: §C.2 · p. 17

Table 6 comparison

• Table 6 compares OpenLLM Leaderboard v1 results on Kimi-K2.5 for GSQ, REAP, and MoESQ, with recovery measured relative to the reference row.Source: Table 6 · p. 17

Compared methods

• The methods use distinct formats: GSQ uses dense INT2 W2A16, REAP prunes 50% of routed experts and retains released INT4 weights, and MoESQ uses paired 4:8 NVFP4 W4A4.Source: Table 6 · p. 17

C.3 NON-TERMINATING GENERATIONS UNDER JSQ

• Accuracy on terminated generations measures answer quality after successful termination, while the matched-subset comparison controls for JSQ terminating on fewer problems.The table caption defines coverage as the number of problems on which JSQ terminates at least once, out of the benchmark total.Source: §C.3 · p. 17, Table 7 · p. 17
• JSQ is heavily truncated on all three benchmarks, and its accuracy remains substantially below OBR and MoESQ even among terminated generations.Restricting evaluation to the matched subset does not close the gap; the authors attribute much of the degradation to non-terminating generations rather than ordinary answer errors.Source: §C.3 · p. 17
• Representative trajectories show JSQ continuing repetitively after reaching a correct answer, while MoESQ terminates; on a difficult prompt, JSQ repeats while MoESQ and OBR remain coherent.These examples use the same paired 4:8 NVFP4 W4A4 target.Source: Figure 4 · p. 18
• The authors restrict the non-termination observation to the evaluated JSQ adaptation and do not claim it is universal to JSQ or sparse quantization.The cited discussion characterizes the degradation as rambling or repetitive trajectories that fail to terminate rather than merely more conventional wrong answers.Source: §C.3 · p. 18

D.1 IMPLEMENTATION DETAILS

• The serving integration fuses dispatch and scatter with activation quantization for the first expert GEMM, and SwiGLU output preparation for the second.The sparse backend uses the same fused-MoE interface as the dense NVFP4 backend; its activation quantization forms 32-element scale blocks.Source: §D.1 · p. 19

D.2 KERNEL BENCHMARK

• The sparse kernel is compared with CUTLASS dense NVFP4 in native and transposed operand layouts, with the transposed control differing from the sparse kernel only in sparsity.The native dense layout retains the activation-stationary orientation used by vLLM; kernel variants are autotuned independently, and median on-device replay latency is reported.Source: §D.2 · p. 19
• Geometric-mean speedups over Dense (native) are 1.52× for Kimi-K2.5 gate/up and 1.58× for down, and 1.33× and 1.20×, respectively, for Qwen3-30B-A3B.The sparse kernel is faster in every tested G × N configuration; Figure 5 shows speedup by expert-group count G and maximum per-expert token count N.Source: §D.2 · p. 19, Figure 5 · p. 20
• The speedup is smallest for short, memory-dominated problems and approaches the sparse-MMA advantage as arithmetic intensity increases.The full sweep varies expert-group count and maximum per-expert token count.Source: §D.2 · p. 19, Figure 5 · p. 20

D.3 END-TO-END SERVING BENCHMARK

• The benchmark serves Kimi-K2.5 on 8×B200 with shared DP=8 × EP topology and matched concurrency, varying only the expert representation and its associated compute path.Routing, dispatch planning, batching, memory controls, token combination, and output layout are held fixed across backends.Source: §D.3 · p. 20
• Prefill uses 4096 input tokens and one output token per request at concurrency 32, 64, or 128; decode uses 384 input and 128 output tokens per request with 500 requests in flight.Each workload issues 500 requests; decode throughput is averaged across three repeats, whose throughput varies by up to 22%.Source: §D.3 · p. 21
• The full results are presented in Table 12 for end-to-end serving on Kimi-K2.5 at matched concurrency.The comparison includes paired 4:8 NVFP4 W4A4, dense NVFP4 W4A4, INT4 W4A16, and INT2 W2A16 expert kernels.Source: §D.3 · p. 21, Table 12 · p. 22