PaperlayerView exact revision on arXiv

Source-linked AI summary

Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

Shriniwas Ramesh Suram

arXiv:2608.18261v1cs.AIcs.LG

TL;DR

Serving large MoE models on commodity hardware is constrained by memory bandwidth, motivating the question of whether router cacheability can be trained. A pre-registered multi-domain study finds that locality training reduces cache misses but no configuration meets the joint miss-reduction and perplexity criteria at 137M scale.

  • Problem

    The paper asks whether the accidental routing locality that enables caching can be trained without an accuracy cost.

  • Method

    The authors pre-register locality and domain-partitioning router losses and evaluate them under joint cache-miss and perplexity criteria.

  • Results

    No configuration meets both gates: locality training reduces misses by up to 60% but incurs a perplexity tax at 137M parameters.

  • Takeaways & Limitations

    The negative result makes router cacheability a precise, falsifiable scaling claim rather than a demonstrated accuracy-free optimization.

  • Takeaways & Limitations

    The study uses two sub-compute-optimal model sizes and a Qwen3-30B routing proxy rather than a same-artifact 235B trace.

Abstract

from arXiv · show

Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.

1 Introduction · 2 Background

MoE sparsity makes frontier models runnable beyond datacenters, but decode remains constrained by memory-bandwidth hierarchies. The paper measures this bandwidth wall, profiles routing locality, evaluates placement and training-time locality, and finds strict quality limits.

  • 1 Introduction: MoE routers activate only a small top-k subset of many experts, decoupling total parameter count from per-token active parameters.Qwen3-235B-A22B stores 235 B parameters but activates ∼22 B per token.
  • 1 Introduction: Within 6%, Qwen3-235B-A22B measurements match a first-principles bandwidth model, while request batching collapses at batch 32 in llama.cpp.The batching scheme should amortize disk sweeps but instead produces a measured negative result.
  • 1 Introduction: 2.0× temporal locality, a 52.5% working set, and near-orthogonal per-domain expert sets emerge from llama-moe-trace profiling on Qwen3-30B-A3B.The instrument captures router decisions without changing model weights and supports cache-hit-rate simulation.
  • 1 Introduction: 2–7× decode improvement at full accuracy is achieved by Path-Mapped Serving, but the policy cannot reach interactive speed on this hardware.The result establishes a serving ceiling despite improved expert placement.
  • 1 Introduction: At 137M scale, the pre-registered training-time locality test fails the strict ≤1% perplexity bar under joint miss-and-quality criteria.The evaluation covers a multi-domain corpus and includes adjacent-token routing consistency plus a new domain-partitioning loss.
  • 1 Introduction: Accuracy is held fixed: the study neither quantizes below the model’s shipped 4-bit weights nor distills, defining “same accuracy” as bit-exact with the served model.This preserves the served model’s original accuracy during comparisons.
  • 2 Background: During transformer decode, each step reads active weights once and performs an O(d2) matrix–vector product per layer, making weight-read bandwidth—not FLOPs—the bottleneck.The low arithmetic intensity follows from one-token-at-a-time decoding and per-token FFN computation.
  • 2 Background: MoE storage scales with E while per-token cost scales with k; a learned router selects the top-k experts, and load balancing prevents expert collapse.Fine-grained variants increase E and lower the activation ratio.

3 The Bandwidth Wall

Serving Qwen3-235B-A22B on an 8 GB GPU is limited by SSD bandwidth: each token requires roughly 12 GB of expert reads, yielding 0.44 tok/s warm. A batching strategy that should amortize a sequential sweep instead collapses at batch 32 because paging thrashes the memory manager.

  • The Bandwidth Wall: 12 GB/token of 4-bit expert reads produces 0.44 tok/s warm, implying approximately 2.4 GB/s and matching the SSD’s cold sequential ceiling.The first-principles model predicts 0.20 tok/s from cold disk.
  • The Bandwidth Wall: 50× is the shortfall between the 120 GB/s required for 10 tok/s at 12 GB/token and the SSD’s 2.4 GB/s.The paper frames acceleration as reducing bytes-per-token or increasing bandwidth-to-bytes.
  • The Bandwidth Wall: Batched sequential sweeping follows the union-of-experts model through B=8 but collapses at B=32, where llama.cpp cannot express the strategy reliably.Reading all 134 GB once takes ∼56 s; realizing the approach above B ≈8 requires a purpose-built sequential-sweep engine.
  • The Bandwidth Wall: 0.087 tok/s is the aggregate throughput at batch 32, after 32 concurrent streams trigger page-fault storms and reduce effective IO to ∼0.35 GB/s.The collapse reflects OS memory-manager thrash rather than a flaw in the union-of-experts idea.

4 Measuring Routing Structure · 5 Path-Mapped Serving and Its Ceiling

Routing telemetry reveals substantial temporal and domain structure, enabling Path-Mapped Serving to exploit hot experts without changing model files or accuracy. However, placement alone cannot meet interactive-speed requirements on the evaluated fixed hardware because the required resident-expert footprint exceeds fast memory.

  • 4 Measuring Routing Structure: llama-moe-trace records each layer’s and token’s router top-k expert selections through a roughly 120-line llama.cpp eval-callback addition, without modifying weights.A non-contiguous top-k view required careful handling; flat copying instead read complete argsort rows and produced uniform garbage.
  • 4 Measuring Routing Structure: 2.0× chance adjacent-token expert reuse demonstrates temporal locality in Qwen3-30B-A3B routing.The trace covers 8,000 tokens each from prose, code, math, and medical text.
  • 4 Measuring Routing Structure: 95% of traffic flows through 52.5% of experts, indicating a concentrated routing distribution.This profile is summarized for Qwen3-30B-A3B across four domains.
  • 4 Measuring Routing Structure: Code expert sets overlap other domains by only 0.11–0.16, versus 0.33–0.42 overlap among prose, math, and medical text.The domain structure is consistent with measured Qwen3 specialization.
  • 4 Measuring Routing Structure: 65.9% of requests are served by temporal reuse, exceeding static pinning at 59.2% and approaching the clairvoyant Belady oracle’s 79.1%.The result is reported for the 235B model at the evaluated fast-memory budget.
  • 5 Path-Mapped Serving and Its Ceiling: Path-Mapped Serving keeps currently hot experts in fast memory and streams cold misses from disk, preserving bit-exact accuracy while using measured heatmaps for placement.Rare inputs incur waiting rather than incorrect outputs, and the model file remains untouched.
  • 5 Path-Mapped Serving and Its Ceiling: ∼95% hit rate is required for single-stream 10 tok/s, but achieving it requires ∼52% of experts, or ∼70 GB, resident.That footprint is 4× the machine’s fast memory, so placement alone cannot reach interactive speed and motivates reducing bytes per token architecturally.

6 Training Routers for Cacheability

This section pre-registers whether router training can make expert routing cacheable without sacrificing accuracy. It evaluates locality and domain-confinement losses on a 137M multi-domain MoE under strict joint cacheability and perplexity criteria.

  • Motivation: The study pre-registers hypotheses, experimental arms, metrics, and pass/fail thresholds before running experiments, making a null result meaningful.It tests whether cacheability can be trained at no accuracy cost.
  • Related work: The study explicitly contrasts its harder three-domain corpus and strict joint miss/quality bar with concurrent StickyMoE work on small and medium single-domain models.The supplied passage identifies adjacent-token routing-consistency loss as the concurrent work’s technique.
  • Method: All arms retain Switch load balancing while adding Lloc for adjacent-token expert reuse or Ldom for keeping each domain within an expert slice.Ldom allows a single slice to be pinned and served at full speed; each slice contains 4 exclusive and 4 shared experts out of 16.
  • Setup: The experiment trains a 137M-parameter, 16-expert, top-2 decoder MoE on 300M tokens spanning prose, code, and math.The amended experimental budget is 200M tokens per arm.
  • Pre-registered criteria: RQ1 requires at least a 30% LRU miss/token reduction at 25% capacity with validation perplexity within +1% of arm A.RQ2 separately requires at least 90% static-pin hit@50% within-domain with perplexity within +2%; both criteria must satisfy cacheability and quality jointly.

7 Results

Results refute both research questions: locality training cannot meet the joint miss-reduction and perplexity gates, while domain confinement exceeds the static-hit target but violates its quality gate. Training-free rerouting is complementary to trained locality, reaching roughly 80% miss reduction at modest perplexity cost, whereas domain-primed prefetching provides no measurable benefit.

  • Baseline: 31.9 baseline perplexity accompanies 8.21 LRU miss/token@25% and 0.741 static hit@50%.These values are reported for baseline arm A using two seeds.
  • RQ1: Locality training: Larger locality-loss weights monotonically increase miss reduction and perplexity cost, and no configuration satisfies both pre-registered gates.λ=0.02 gives +0.3% PPL and 15% miss reduction, while λ=0.05 gives 59–60% miss reduction at +2.1–3.1% PPL.
  • RQ2: Domain confinement: 0.991 static-pin hit rate clears the 0.90 target, but domain confinement incurs +3.1% perplexity, exceeding the +2% gate.A single domain slice can therefore be pinned and served almost miss-free, but the quality criterion is still violated.
  • Inference-time rerouting: ∼30% miss reduction costs +0.8% perplexity on one baseline seed and +2.0% on the other, with zero training cost.Pushing rerouting to 50% reduction costs +4.0–5.8%, while 74% costs +25%, because the untrained router selects worse experts.
  • Complementarity: 80–82% miss reduction at +1.9–3.1% PPL results from adding τ=0.5 rerouting to locality training, versus 58–61% at +1.4–2.2% PPL from training alone.Across both seeds, τ=0.7 reaches 86–88% reduction, and the same τ=0→0.5 intervention costs only +0.5–0.9% PPL on the trained model.
  • Domain-primed prefetching: Domain-primed prefetching provides no measurable benefit because the LRU cache re-warms within O(cap) tokens and boundary bursts amortize over realistic domain runs.Mixed-stream miss/token equals the single-domain value to within noise for runs of at least 1k tokens.

8 Discussion

Training reliably improves routing locality, reducing misses by up to 60% and achieving a 99% static hit rate, but incurs a genuine perplexity tax that violates every pre-registered quality bar. The 340M scale rung shows that this tax does not yet shrink with scale, while stacking training-free rerouting retains an advantage.

  • Training locality: ≤60% fewer misses and a 99% static hit rate show that training changes routing as intended, with monotonic and seed-stable effects.The mechanism is real but not free at 137M parameters.
  • Training locality: At 137M parameters, every configuration incurs a perplexity tax that breaks the pre-registered quality bars.The joint criterion prevents reporting a large but costly locality effect as a win through post hoc metric selection.
  • Scaling: At 137M, the 16-expert model has little spare capacity, so forcing expert reuse or confinement removes representational freedom used for language modeling.The paper frames scale-dependent tax reduction as a sharp, pre-registered hypothesis motivated by sparsity scaling.
  • Scaling: The 340M rung shows that the locality tax does not shrink with scale, while the stacked training-free rerouting advantage persists.The comparison uses the same 200M-token budget, a 25% expert cap, LRU, and seed 1.

9 Limitations

The study is limited by sub-compute-optimal scale and training, reliance on a Qwen3-30B proxy for routing, and the absence of a same-artifact 235B trace.

  • Scale and training: Two model rungs, 137M and 340M, are below compute-optimal; the 340M locality tax did not shrink, leaving ≥1B and compute-optimal runs as definitive tests.The study used only 200M tokens, far below compute-optimal, so absolute perplexities are high despite fair identical-budget comparisons.
  • Proxy measurement: Routing profiles come from Qwen3-30B as a same-family proxy for the 235B, while a same-artifact trace on the 235B GGUF remains future work.

10 Related Work · 11 Conclusion · A Per-Domain Results

The paper situates its locality objective among prior MoE routing and inference-optimization work, then concludes that cacheability is trainable but fails the pre-registered joint criteria at small scale. Domain results show a consistent cacheability gain with a perplexity tax concentrated in prose and negligible in code.

  • 10 Related Work: Prior MoE systems establish sparse total/active expert routing, load balancing, and expert specialization that this work seeks to exploit.The cited foundations include sparsely-gated MoE, Switch, GShard, and DeepSeekMoE, alongside routing analyses documenting specialization.
  • 10 Related Work: StickyMoE provides the closest prior for the adjacent-token routing-consistency loss evaluated in this paper.The passage describes StickyMoE as introducing the loss and reporting it favorably.
  • 10 Related Work: Offloaded inference work characterizes memory-bound generation, while quantization, KV-cache compression, and speculative decoding address orthogonal bandwidth or bytes-per-token tradeoffs.These complementary axes are held fixed relative to training-time routing.
  • 11 Conclusion: Commodity-hardware serving of a 235B MoE is memory-bandwidth limited, and cacheability can be trained but not for free at small scale.The conclusion emphasizes that no configuration meets the pre-registered joint criteria and presents the result as a complete negative finding.
  • 11 Conclusion: The study releases its pre-registration, code, llama-moe-trace, router traces, per-run configurations, seeds, and data hashes.The passage provides the project repository URL for these artifacts.
  • 11 Conclusion: The author used AI assistance for the experimental harness and manuscript while retaining responsibility for design, preregistration, runs, interpretation, and verification.The passage states that every reported number was checked against the released artifacts.
  • A Per-Domain Results: ∼0.30 reuse gain appears in every domain, while the perplexity tax reaches +3.0% in prose and is essentially zero in code.Code has PPL ≈4.7 for both the baseline and λ=0.05 locality arm; the prose tax reflects limited perplexity headroom.

B All Runs · C Hyperparameters · D Router Losses (exact form)

The merged material documents the complete run inventory, per-domain baseline-versus-locality evaluation, and the exact router-loss formulation. It distinguishes preliminary sanity/triage runs from the main study and specifies the domain-slice structure used in training.

  • B All Runs: Table 8 lists every training run, including 50M-token sanity and λ-triage runs conducted before the 200M main arms.The sanity and triage rows are not used in the verdict.
  • C Hyperparameters: Table 7 compares baseline (A) with locality (λ=0.05) using per-domain metrics averaged over 2 seeds.
  • B All Runs: In Table 7, “hit@25%” denotes LRU and “static@50%” denotes static-pin.
  • B All Runs: The complete run log reports metrics averaged over prose, code, and math.
  • D Router Losses (exact form): The total training loss is L = LLM + αLbalance + λLloc + µLdom.
  • D Router Losses (exact form): For each of 3 domains, the domain slice S(d) contains 4 exclusive experts plus 4 shared experts within E=16.

E Cache Simulation · F Corrections Log

The cache simulator replays per-layer router traces under multiple policies and reports normalized demand misses, while the corrections log documents measurement, implementation, and analysis fixes. Key corrections include the cold NVMe baseline, batching thrash, dispatch slowdown, initialization error, and normalization bug.

  • E Cache Simulation: Cache simulation replays each layer’s (T, k) expert trace through an independent cache of capacity c = ⌈cap · E⌉, reporting misses per token normalized by L · k expert-loads.Evaluated policies are LRU, LFU, static top-c pinning after warmup, and Belady’s clairvoyant upper bound.
  • E Cache Simulation: The same simulator scores both frontier-model and trained-model traces, spanning the analyses in §4 and §7.Belady is an unachievable upper bound because it evicts the resident expert whose next use is farthest in the future.
  • F Corrections Log: 2.4 GB/s was the cold sequential NVMe rate used throughout, replacing the inflated 5.3 GB/s warm page-cache measurement.The correction distinguishes cold storage bandwidth from warm page-cache behavior.
  • F Corrections Log: B=32 batching throughput collapsed from paging thrash after predictions held through B=8, and the result was retained as a negative finding rather than tuned away.The failure was reported as part of the study’s negative results.
  • F Corrections Log: A non-contiguous argsort view caused garbage reads and sub-chance reuse until a stride-honoring copy fixed the issue; the initial profiling run was discarded.The faulty output made every expert appear exactly k times.
  • F Corrections Log: 373 initial tied-logit loss from N(0, 1) embedding initialization was corrected to std 0.02, restoring the expected starting scale of ∼10.8.The initialization correction was logged before results were interpreted.
  • F Corrections Log: 2.8× slower padded-batched-GEMM dispatch under early-training imbalance was reverted to a per-expert loop, trimming budgets from 300M→200M per arm.The parity test was retained, and the change was logged before results were unblinded.
  • F Corrections Log: The analysis’s negative hit rates were corrected by normalizing summed misses over all layers by L · k rather than one layer’s slots.The original denominator error produced negative rates.

G Reproducibility Artifacts

The paper releases its pre-registration, code, tracing tool, tests, router traces, and per-run configurations, enabling every reported number to be regenerated.

  • The release includes the frozen pre-registration, training and analysis code, orchestration code, and the llama-moe-trace tool.
  • 7 unit tests cover loss shapes, gradient flow, routing limits, domain-mask layout, determinism, and dispatch parity.
  • All router traces and per-run config.json files include seeds, loss weights, token budgets, and datashard SHA-256 hashes.The authors state that every number in the paper is regenerable from these artifacts.
Loading 2608.18261v1…