Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

1 INTRODUCTION

• MoE decoding at small batch sizes is bottlenecked by routed-expert weight reads, although most intermediate channels contribute little to any single token.Channel selectors predict which channels matter and read only those weights.Source: §1 · p. 2
• 40 of 43 comparisons favor or tie top-k over calibrated thresholds at the calibrated budget, while thresholded Gemma 4 can collapse into repetition loops.The comparisons use the same selector scores at 13% and 18% budgets.Source: §1 · p. 2
• A calibrated threshold does not ensure its target read budget because the number of channels passing it varies with the input.Prox thresholds calibrated for an 18% budget read 12.9% on average across the benchmarks.Source: §1 · p. 2
• The proposed reader screens channels using a rotated ternary sketch of the weights, computes top-k exactly, and recovers skipped channels with a low-rank map.The sketch is about 7% of the BF16 checkpoint.Source: §1 · p. 2
• At 17.8% of routed-expert reads, the reader keeps 95.0% of full-model accuracy in its worst model-benchmark pair and makes whole-model decoding up to 1.88× faster than optimized vLLM.It is the most accurate tested selector on average and in its worst model-benchmark pair at both 12.8% and 17.8% reads.Source: §1 · p. 2

2 BACKGROUND

• A routed expert is a gated MLP, and each intermediate dimension is a channel owning one gate/up-projection row and one down-projection column.For Qwen3-Thinking, a token routed to eight experts with 768 channels each faces 6,144 channels per layer.Source: §2 · p. 2
• A calibrated threshold keeps however many channels exceed its cutoff, whereas top-k fixes the channel count in advance.On the same scores, Figure 2 illustrates a threshold keeping six channels for one token and two for another, versus four for each under top-k.Source: §2 · p. 2, Figure 2 · p. 3
• Read cost counts every touched 32-byte sector, including kept-channel, sketch or proxy weights, and auxiliary parameters.At batch one on Qwen3-Thinking, the profiler measures cold-cache DRAM reads of 18.3% versus the counted 17.8%.Source: §2 · p. 3
• The keep-rule comparison runs each selector on the same score with either its calibrated threshold or top-k at the threshold's target budget.A check also runs top-k at the threshold version's realized reads, which favors the threshold version.Source: §2 · p. 3

3 RELATED WORK

• Prior selectors include trained predictors, sign-bit methods, full-projection methods, input-sparsity methods, and low-bit-copy methods.Full-projection selectors read at least a third of each routed expert, while the paper's reader belongs to the low-bit-copy family.Source: §3 · pp. 3–4
• Six earlier keep-rule studies disagree: three favor top-k, two favor thresholds, and one finds a hybrid best.Five of the six test dense models, mostly on short tasks, and two change the score along with the keep rule.Source: §3 · p. 4
• The paper's conclusions concern calibrated thresholds on frozen MoE expert channels, not sparse attention or trained MoE routers.The passage also notes that task-dependent sparsity motivates data-free selection.Source: §3 · p. 4

4.1 A CALIBRATED THRESHOLD DOES NOT FIX HOW MUCH A SELECTOR READS

• At around 6,000 answer tokens, kept counts fall to 71% of calibrated for Qwen3-Thinking, 92% for Gemma 4, and 88% for Qwen3.5.On benchmark answers, the median kept share falls to 10.0% for Qwen3-Thinking.Source: §4.1 · p. 4
• In all 72 benchmark runs, calibrated thresholds read below budget; Prox calibrated for 18% reads 12.9% on average and as little as 9.9%.Calibrating on the models' own reasoning does not stop the drift.Source: §4.1 · p. 4, Figure 4 · p. 5
• Removing earlier answer context raises the kept count for identical tokens, while equalizing per-token counts lowers reasoning KL on Qwen3-Thinking and Gemma 4 but raises it on Qwen3.5.Equalized counts change KL by −3.3 ± 1.0%, −12.5 ± 1.5%, and +10.5 ± 1.1%, respectively.Source: §4.1 · pp. 4–5

4.2 CALIBRATION GIVES NO CONSISTENT ACCURACY GAIN OVER TOP-k

• In 43 comparisons at 13% and 18% budgets, top-k ties thresholds in 30, leads in 10, and trails in 3 by at most 2.8 points.No model favors thresholds when comparisons are pooled by model.Source: §4.2 · p. 5, Figure 5 · p. 6
• At realized reads, top-k still matches or beats thresholds in 37 of 41 comparisons, and calibrated layer budgets or recovery maps improve no benchmark beyond two standard errors.A calibrated recovery map beats the weight-only map in 0 of 18 comparisons.Source: §4.2 · p. 6

4.3 LONG REASONING CAN COLLAPSE UNDER CALIBRATED THRESHOLDS

• On Gemma 4, calibrated thresholds cause repetition to the 32,768-token limit, while replacing only Prox's channel threshold with top-k avoids the collapse.At nearly the same reads, Prox scores 11.7 with its 18% calibrated threshold versus 56.3 with top-k.Source: §4.3 · p. 6
• The authors suspect collapse behaves like a cliff, with accuracy dropping sharply once sparsity-related error crosses a model-dependent level.They state that thresholded reads are not known to the deployer in advance, and equal-read accuracy can be far below top-k.Source: §4.3 · pp. 6–7

5 A READER BUILT FROM THE WEIGHTS ALONE

• On OLMoE, scaling Prox’s score to output units with down-projection norms and router weights divides KL by 1.3 to 1.5 on the same 4-bit weight copy.Across four scores and four budgets, KL divergence falls as kept channels carry a larger share of routed-output energy (Spearman −0.96).Source: §5 · p. 7
• The reader screens candidate channels, computes the top-k exactly, and recovers the rest using only the full model’s weights.For each token and layer, the process screens candidates, computes the kept channels exactly, and recovers skipped channels.Source: Figure 6 · p. 7, §5 · p. 7

5.1 SCREENING CHANNELS WITH A ROTATED TERNARY SKETCH

• At decode time, a rotated ternary sketch estimates gate activations; storing ternary gate and value blocks cuts each block from 256 BF16 bytes to about 30 bytes, adding about 7% to resident memory.At the two lowest budgets, 12.8% and 17.8% of reads, a partial screen reads codes only at the 80 coordinates with largest |Rx|, buying 20 to 50% more exact channels.Source: §5.1 · p. 8

5.2 OUTPUT-SCALED SCORING AND EXACT TOP-k COMPUTATION

• The score estimates each candidate channel’s output contribution as its router weight times down-projection norm times activation magnitude.Because scores are in output units, channels across experts can be compared and the k best selected across a token’s experts.Source: §5.2 · p. 8

5.3 WEIGHT-ONLY LOW-RANK RECOVERY OF SKIPPED CHANNELS

• A rank-32 map approximates skipped channels’ down columns, adding their estimated output while contributing about 0.5% resident memory and 0.5 to 0.8 percentage points of reads.Kept-channel contributions are exact; the map recovers estimated output for skipped channels.Source: §5.3 · p. 8
• At equal reads on OLMoE, spending 0.6 to 0.7 percentage points on recovery lowers KL divergence more than spending them on more exact channels.On Gemma 4, recovery raises IFEval accuracy by 11.6 ± 1.6 points at 18% reads and reduces repetition loops; the partial screen beats the full screen by 13.7 ± 1.3 points on MMLU-Pro at 13%.Source: §5.3 · p. 8

6.1 ACCURACY

• The reader has the highest mean accuracy at both low budgets, and its weakest model-benchmark pair beats every other selector’s weakest pair.Table 1 reports mean and worst accuracy over nine model-benchmark pairs as a percentage of full-model accuracy.Source: Table 1 · p. 9, §6.1 · p. 9
• Prox calibrated for an 18% budget reads 12.9% and reaches 68.5% of full-model accuracy on average, versus 94.5% for the reader.Most of the gap comes from Gemma 4, where Prox’s threshold version falls into repetition loops; on the two Qwen models, it is about as accurate as the reader at similar reads.Source: §6.1 · p. 9

6.2 SPEED

• 1.90×, 1.71×, and 1.60×: MoE blocks run faster at batch one on Qwen3-Thinking, Gemma 4, and Qwen3.5, respectively, than optimized vLLM.The reader’s Triton kernels select the same channels evaluated for accuracy, at 17.8% of reads.Source: §6.2 · p. 9, Figure 7 · p. 10
• 1.15× to 1.36×: whole models decode faster at batch one; at 100 tokens per second per user, one GPU serves 3.3 to 4.3 times as many tokens.Attention runs as before, so whole-model gains are smaller than MoE-block gains.Source: §6.2 · p. 9
• On Qwen3-Thinking, the MoE block moves 18.3% of full-model memory traffic but runs 1.90× faster, as fixed per-step costs do not shrink with fewer reads.Larger batches share these fixed costs, so block speedup grows up to batch four or eight.Source: §6.2 · p. 9
• At batch 128, whole-model speedup falls to 1.00× to 1.24×, while at batch 16 the reader is 1.72× to 1.88× faster.The passage attributes the remaining batch-16 opportunity to MoE models with many small experts, where each expert serves about one token or fewer on average.Source: §6.2 · p. 10

7 LIMITATIONS AND CONCLUSION

• The kernels convert only part of saved reads into speed, were tuned for one budget and GPU type, and were tested on models up to 35B parameters.The reader changes only routed experts during decoding; its sketch and recovery map add about 7% and 0.5% of a BF16 checkpoint, respectively.Source: §7 · p. 10
• The authors suggest that short likelihood evaluations can miss long-generation failures, and that lower KL divergence need not mean higher benchmark accuracy.They report that no threshold version beats top-k on generation tasks at 13% or 18% budgets, and Gemma 4 develops repetition loops under thresholds.Source: §7 · p. 10
• At 17.8% of routed-expert reads, the weight-only reader retains 98.1% of full-model accuracy and runs MoE blocks up to 2.69× faster than optimized vLLM.The conclusion also reports whole-model speedups up to 1.88×.Source: §7 · p. 10

AI USE STATEMENT / REPRODUCIBILITY STATEMENT

• The authors report using LLM coding assistants for experiments and drafting, while retaining responsibility for the study and reported numbers.The paper states that models, benchmarks, calibration data, code, kernels, patches, and evaluation scripts are publicly available.Source: AI USE STATEMENT · p. 11, REPRODUCIBILITY STATEMENT · p. 11

A EXTENDED RELATED WORK

• Prior keep-rule comparisons disagree, and most focus on short tasks on dense models rather than long chain-of-thought MoE evaluation.The paper also distinguishes its comparisons by holding the channel score fixed and testing top-k at both target and realized reads.Source: Table 2 · p. 18, A EXTENDED RELATED WORK · p. 18
• Prior skipped-channel methods fit recovery estimates on calibration data, whereas this reader reuses ternary-screen activation estimates through low-rank down-projection vectors.Related work also covers batching strategies and sparsity at expert, weight, and channel levels.Source: A EXTENDED RELATED WORK · p. 18

B EXPERIMENTAL SETUP

• Evaluation uses Qwen3-Thinking, Gemma 4, and Qwen3.5 in vLLM, with the full model processing prompts and selectors running on later decoded tokens.Shared experts and parallel dense MLPs always run in full and are excluded from routed-expert reads.Source: B EXPERIMENTAL SETUP · p. 19
• The post-trained benchmarks include GPQA-Diamond, IFEval, and MMLU-Pro, while OLMoE uses GSM8K and BBH with task-specific decoding settings.GPQA-Diamond uses four sampled answers per question, IFEval reports prompt-level strict accuracy, and MMLU-Pro uses a fixed 1,000-question subset.Source: B EXPERIMENTAL SETUP · p. 19
• Answers reaching the token limit count as wrong; this avoids crediting collapsed outputs that format checkers might otherwise pass.Of 10,872 IFEval answers reaching the limit in the main runs, 202 ended in repetition loops.Source: B EXPERIMENTAL SETUP · pp. 19–20
• Prox, TEAL, and WINA use UltraChat calibration for post-trained models, while OLMoE thresholds are fitted on Pile text with disjoint development and held-out sets.Prox’s sector-counted budget split plans below target, and every test run reads even less than that plan.Source: B EXPERIMENTAL SETUP · p. 20
• Benchmark layouts were checked for numerical consistency before pooling, and baseline adaptations specify the projection scores, thresholds, and input fractions used.The evaluation includes multiple GPU layouts; Gemma 4 ran only on the RTX PRO 6000 and A6000 setups.Source: B EXPERIMENTAL SETUP · pp. 20–21

C READ ACCOUNTING AND PROFILER CHECK

• The reader’s read accounting charges weight and metadata transfers in whole 32-byte sectors and is checked against GPU-profiler measurements.The accounting includes sketch codes, kept-channel weights, recovery-map parameters, norms, and rotation signs, without modeling cache reuse across tokens.Source: C READ ACCOUNTING AND PROFILER CHECK · p. 21
• At batch one, the reader’s measured DRAM traffic is 18.3% of the full model’s, close to its 17.8% sector-count estimate.The profiler measured 12 of Qwen3-Thinking’s 48 MoE layers; at batch four and eight, the measured shares rise to 21.8% and 25.7%.Source: C READ ACCOUNTING AND PROFILER CHECK · p. 21, Table 7 · p. 22

D CALIBRATED THRESHOLDS READ LESS THAN THEIR BUDGET

• All 216 calibrated-threshold runs read less than their calibrated budgets across nine post-trained model-benchmark pairs.Table 8 reports the lowest, mean, and highest realized routed-expert reads for each selector and budget.Source: Table 8 · p. 22, D CALIBRATED THRESHOLDS READ LESS THAN… · p. 22

D.1 CALIBRATING ON THE MODEL’S OWN GENERATIONS

• On Gemma 4, trace-calibrated Prox still reads 10.5%, 11.6%, 14.0%, and 15.2% against its targets, while GPQA-Diamond scores are 0.0, 0.3, 23.9, and 45.1.At those four settings, 100%, 100%, 65%, and 35% of answers hit the 32,768-token limit; chat-calibrated Prox at 18% scores 11.7 at 13.3% of reads.Source: §D.1 · p. 23
• On Qwen3-Thinking GPQA-Diamond, trace-calibrated Prox scores 57.1 at 9.6% of reads and 62.8 at 11.2%, while chat calibration scores 59.7 at 9.9%.On IFEval, the trace-calibrated version scores 82.1 versus 82.8 with chat calibration.Source: §D.1 · p. 23
• Trace calibration neither stops Prox from reading below its targets nor prevents Gemma 4’s repetition-loop collapse.On Gemma 4 IFEval, trace-calibrated Prox scores 31.2 at 14.9% of reads, against 11.3 with chat calibration.Source: §D.1 · p. 23

E WHAT A CALIBRATED THRESHOLD’S CHANNEL COUNT FOLLOWS

• On Qwen3-Thinking, threshold counts correlate with answer position at −0.57 but with token need at only −0.13 and −0.19 at the two budgets.Token need itself correlates with full-model next-token loss at +0.58 and +0.68, rather than with answer position.Source: E WHAT A CALIBRATED THRESHOLD’S… · p. 23
• At equal reads, fixed per-layer counts lower KL divergence on Qwen3-Thinking and Gemma 4 reasoning but raise it on Qwen3.5.The reported changes relative to chat-calibrated thresholds are −3.3 ± 1.0% and −3.4 ± 1.2% pooled on Qwen3-Thinking, −12.5 ± 1.5% on Gemma 4 at 13%, and +10.5 ± 1.1% and +14.8 ± 1.2% on Qwen3.5.Source: E WHAT A CALIBRATED THRESHOLD’S… · p. 24
• Making each layer’s count constant across answer windows barely changes KL at equal pooled reads, while reversing the schedule raises it on Qwen3-Thinking and Qwen3.5.Constant counts change KL by at most +1.0 ± 0.5%; the reversed schedule raises it by 3.5% to 7.1% across the reported budgets and models.Source: E WHAT A CALIBRATED THRESHOLD’S… · p. 24
• On Qwen3.5, fixed counts tie the threshold on IFEval but score lower on GPQA-Diamond, with longer answers and more length-limit hits.On IFEval, scores are 89.5 at 9.7% of reads versus 88.5 at 9.7%; on GPQA-Diamond they are 79.8 at 9.4% versus 72.7 at 9.1%.Source: E WHAT A CALIBRATED THRESHOLD’S… · p. 24
• On Gemma 4 IFEval, fixed per-layer counts score 0.0 at 10.1% of reads, while Prox top-k with the same count in every layer scores 16.5 at 11.7%.The threshold version also scores 0.0 at 10.1%; the authors say this test cannot distinguish whether collapse comes with calibrated per-layer budgets or the low reads they settle at.Source: E WHAT A CALIBRATED THRESHOLD’S… · pp. 24–25
• At matched realized reads, top-k matches or beats thresholds in 37 of 41 comparisons at 13% and 18% budgets.The four excluded TEAL comparisons are at heavy sparsity, where both versions collapse.Source: Figure 8 · p. 25

F EVERY KEEP-RULE COMPARISON

• In 23 generation-task comparisons, top-k leads beyond two standard errors in 8 and thresholds in none.Two heavy-sparsity TEAL comparisons where both versions collapse are excluded from the count.Source: F EVERY KEEP-RULE COMPARISON · p. 25

F.1 APPLYING PROX TO OLMOE

• For OLMoE comparisons, Prox uses Pile calibration and sector-read budgets; at realized reads, threshold and top-k versions lead in 1 and 2 of 10 whole-sequence comparisons.With the prompt computed by the full model, the threshold version leads in 2 comparisons and top-k in none.Source: §F.1 · pp. 25–26

F.2 REPETITION LOOPS ON GEMMA 4

• Replacing Prox’s channel threshold with top-k raises GPQA-Diamond accuracy by 43.9 to 44.9 points and IFEval accuracy by 42.3 to 43.1 points, avoiding repetition-loop collapse.With both thresholds, Prox scores 0.0 on GPQA-Diamond and IFEval; top-k at both stages scores 51.0 and 45.8, respectively.Source: §F.2 · pp. 26–27

F.3 OUR READER WITH A CALIBRATED THRESHOLD, AND CATS

• Across 36 reader comparisons, calibrated thresholds never beat top-k; across 18 CATS comparisons, thresholds lead only twice, both on Gemma 4 at 53% reads.For the reader, the two remaining comparisons favor top-k, both on Gemma 4 at 12.8% reads; the threshold version reads less than its budget on benchmarks.Source: §F.3 · pp. 27–28

F.4 CALIBRATED LAYER BUDGETS AND CALIBRATED RECOVERY

• At equal total reads, calibrated layer budgets lower KL on reasoning text but do not yield a detected benchmark gain; calibrated recovery maps do not beat the weight-only map beyond two standard errors.On Qwen3-Thinking, chat-calibrated layer budgets lower reasoning-text KL by 15.7 ± 1.9%, while weight-derived budgets change it by +2.4 ± 1.7%.Source: §F.4 · pp. 28–30

• The reader estimates screened pre-activations from ternary-sketch error, then scores channels by expected squared output error if skipped after low-rank recovery.At 28.7% of reads and above, the full sketch supplies block quantization errors used to model pre-activation uncertainty.Source: G WHAT EACH PART OF THE READER… · p. 30
• Across 16 OLMoE settings, kept-channel output energy and held-out next-token KL divergence have rank correlation −0.96.KL divergence scales roughly as (1 − E)^1.2–1.9, where E measures the routed-output energy preserved by kept channels.Source: G WHAT EACH PART OF THE READER… · p. 30
• At equal reads on held-out OLMoE documents, partial screening helps at small budgets and recovery helps at every budget.The ablation comparison measures next-token KL divergence from the full model.Source: Table 16 · p. 30
• A 4-bit weight-copy variant matches the reader at the two smallest budgets and has lower KL from 28.5% of reads upward, with about twice the resident auxiliary memory.It keeps the reader’s output-scaled top-k and weight-only recovery and needs no calibration data.Source: G WHAT EACH PART OF THE READER… · pp. 30–31
• On OLMoE held-out documents, the reader on a GPTQ 4-bit checkpoint reaches next-token KL divergence 0.045 at 18.9% of reads.At 21.5% of reads, its KL is 0.033, versus 0.019 for GPTQ 4-bit weights alone at 26.0%.Source: G WHAT EACH PART OF THE READER… · p. 31
• Removing Hadamard rotation or fixing each expert’s channel share raises OLMoE KL; reading 80 of 128 rotated coordinates is lowest or near-lowest across tested budgets.On Gemma 4 development data, 80 coordinates is also lowest at two budgets and within 0.0003 of the lowest at the third.Source: G WHAT EACH PART OF THE READER… · p. 31

H EVERY MODEL AND TASK

• At 17.8% of reads, the reader trails the full model by 3.9 ± 1.0 points on Gemma 4 and 4.6 ± 1.4 points on Qwen3-Thinking in an eight-sample GPQA-Diamond rerun.The section’s figures and Table 18 report accuracy against reads for each model and benchmark.Source: H EVERY MODEL AND TASK · p. 31, Table 18 · p. 34
• At 17.8% of reads, answer length stays within about 15% of the full model’s on every model-benchmark pair, and at most 1.3% of answers hit the length limit.The same length-limit bound holds for the full model.Source: H EVERY MODEL AND TASK · p. 31

I BASELINE COMPARISON ON OLMOE

• On OLMoE, the reader has lower next-token KL divergence than every baseline evaluated at each budget and reads less than Prox’s nearest point at 13% and 18%.Among baselines evaluated at 13% to 18% of reads, every one except Prox has at least four times the reader’s KL divergence.Source: I BASELINE COMPARISON ON OLMOE · p. 35, Table 19 · p. 35

Same outputs

• The timed Triton kernels keep the evaluated reader’s channel selections on all three models: 0 of 25,606 checked decisions differ.On Qwen3-Thinking, the two implementations’ next-token KL divergences from the full model differ by −0.0011 ± 0.0013 (90% confidence interval).Source: J SPEED DETAILS · p. 36

Long contexts

• At 8,192 to 32,768 context tokens, whole-model decoding is 1.08 to 1.54× faster across the three models, with gains shrinking at longer contexts.Neither version fits in memory for batch four at 32,768 tokens on Gemma 4 and Qwen3.5.Source: J SPEED DETAILS · p. 36

Where the time goes

• At batch one, a Qwen3-Thinking MoE layer takes 64.7 µs in the full model and 34.4 µs with the reader; attention takes 48 µs per layer in both.In the reader, screening takes 9.5 µs, top-k selection 5.0 µs, kept-channel computation 16.4 µs, and recovery projection 8.0 µs.Source: J SPEED DETAILS · pp. 36–37

• At 17.8% of routed-expert reads, our reader raises per-GPU throughput at the same per-user speed for whole-model decoding of three post-trained models against optimized vLLM.Figure 12 uses 128-token prompts on one RTX PRO 6000 Blackwell GPU; per-user speed is inverse time per output token, and per-GPU throughput is the batch’s total output rate.Source: Figure 12 · p. 37

Larger batches

• At batch sizes 8 or 16, whole-model speedup peaks at 1.75× on Qwen3-Thinking, 1.74× on Gemma 4, and 1.88× on Qwen3.5, then falls at batch 128.At batch 128, the respective speedups are 1.00×, 1.07×, and 1.24×.Source: Table 20 · p. 36, J SPEED DETAILS · p. 37

Serving trade-off

• At the same per-user speed of 80 to 120 tokens per second, one GPU produces 3.3 to 4.4 times as many tokens per second in total with the reader.At longer contexts, attention takes a larger share of each step and this advantage shrinks.Source: J SPEED DETAILS · p. 37, Figure 12 · p. 37