PaperlayerView exact revision on arXiv

Source-linked AI summary

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Minsang Kim, Seungjun Baek

arXiv:2610.30206v1cs.LGcs.AI

TL;DR

Long-context inference is memory-bound as KV caches grow, and existing dense-transformer compression methods do not account for how eviction can alter MoE expert routing. MoEZip estimates routing fragility from instruction-token router states and attention to protect context positions. Across benchmarks it is especially robust under aggressive compression, outperforming KVZip at 10% retention on LongBench and RULER.

  • Problem

    Long-context KV caches grow linearly with sequence length, while dense-transformer compression methods do not account for MoE-specific routing changes caused by eviction.

    Source: §1 · pp. 1–2

  • Method

    MoEZip estimates routing fragility with a Fisher-information-based metric, combines it with instruction-to-context attention, and reserves highly scored context positions as anchors.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    At 10% KV retention, MoEZip outperforms KVZip [19] by +24 points on average on LongBench and +20 points on average on RULER.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across LongBench, NIAH, and RULER, MoEZip consistently outperforms strong baselines, especially in the high-compression regime.

    Source: §6 · p. 10

    MoEZip requires router outputs and instruction-to-context attention statistics, which may not be exposed in black-box inference settings.

    Source: §6 · p. 10

Abstract

from arXiv · show

Mixture-of-Experts (MoE) language models scale model capacity efficiently by activating a small subset of parameters per token. However, in long-context inference, memory remains a major bottleneck because the KV cache grows linearly with sequence length. Existing KV cache compression methods reduce this cost for dense transformers, but overlook a problem specific to MoE: KV eviction can change which experts are selected during answer generation, degrading the generation quality. We propose MoEZip, a routing-aware KV cache compression for sparse MoE language models. To estimate the sensitivity of expert routing to perturbation, we propose a novel metric based on the Fisher information matrix of routing distributions. MoEZip combines this metric with attention weights for routing-aware scoring of contexts. Finally, we compose a complementary scoring system that combines eviction methods for dense and sparse-MoE architectures. Experiments show that MoEZip achieves robust performance under aggressive cache compression on various benchmarks. At a KV retention ratio of 10\%, MoEZip improves over the SoTA baseline by +24 and +20 points on average on LongBench and RULER, corresponding to 2.3$\times$ and 4.9$\times$ higher scores, respectively.

1 Introduction

MoE models scale capacity by activating only a small subset of parameters per token, but long-context inference remains memory-bound as KV caches grow with sequence length. Existing dense-model compression methods miss MoE routing fragility; MoEZip addresses it with routing-aware scoring and reports gains under aggressive compression.

1 Introduction

  • MoE models scale total capacity without paying the full computation cost of dense models by activating only a small subset of parameters per token.This property has made MoE a promising architecture for large-scale deployment.Source: §1 · p. 1
  • Long-context applications make memory a major inference bottleneck because KV cache size grows linearly with sequence length.The cited settings include interaction histories, retrieved documents, tool outputs, and multimodal observations.Source: §1 · p. 1
  • KV eviction can perturb MoE router decisions and change which experts are activated, a failure mode dense-transformer eviction methods do not capture.MoEZip estimates routing fragility with a Fisher-information-based metric and combines it with instruction-token attention to score context positions.Source: §1 · p. 2
  • At 10% KV retention, MoEZip beats KVZip [19] by +24 points on average on LongBench and +20 points on average on RULER.These correspond to 2.3× and 4.9× higher scores, respectively.Source: §1 · p. 2

2 Preliminaries: Mixture-of-Experts Models and Prefix KV Compression

Sparse MoE models route tokens to a small subset of experts, while autoregressive inference reuses prefix key-value entries through a KV cache. Evicting entries can perturb router decisions, and observations under compression link changed expert selection with degraded quality.

2 Preliminaries: Mixture-of-Experts Models and Prefix KV Compression

  • In sparse MoE layers, a router selects a small subset of experts, with softmax- and sigmoid-type routing both considered.The router logits depend on the post-attention hidden state, which is derived from attention over cached prefix keys and values.Source: §2 · pp. 2–3
  • Prefix KV eviction retains a subset of cache entries to reduce memory and decoding overhead while keeping pruned-cache generation close to full-cache generation.The full cache cost grows linearly with prefix length.Source: §2 · p. 2
  • KV eviction perturbs attention outputs and router logits, potentially changing expert selection and degrading generation quality.Dense transformers commonly retain entries with large attention mass, but sparse MoE models have this additional routing-related failure mode.Source: §2 · pp. 2–3
  • As compression becomes more aggressive, answer-token expert flips rise with quality loss, while instruction-token flip rates track answer-token flips and converge under strong compression.The experiment compares matched token sequences with and without prefix KV eviction.Source: §2 · p. 3, Figure 1 · p. 3
  • Instruction-token flip rates are available before answer generation and therefore can serve as a predictor of routing failure.Answer-token flips can only be measured after generation under the evicted cache.Source: §2 · p. 3

3 Method: Routing-Aware Prefix KV Compression

MoEZip estimates router fragility using the Fisher trace and propagates instruction-token fragility to context positions through attention. It reserves high-scoring positions as anchors, then uses a base compressor to fill the remaining cache budget.

3 Method: Routing-Aware Prefix KV Compression

  • MoEZip uses instruction-token router states as a proxy for unavailable future answer-token routing fragility, combining Fisher-trace fragility with instruction-to-context attention to rank context positions.The top-scoring positions are reserved as anchors within the existing per-layer/head retention budget.Source: §3 · pp. 4–5
  • The Fisher trace provides a tractable measure of routing-distribution sensitivity averaged over perturbation directions.The paper derives the metric for softmax- and sigmoid-type routing; computing the closed-form expressions incurs O(E) complexity.Source: §3 · pp. 4–5
  • MoEZip adaptively allocates part of each layer/head budget to routing-aware anchors, with more fragile samples receiving more anchor protection.The base scorer selects the remaining positions, so the total cache budget stays unchanged.Source: §3 · pp. 5–6
  • MoEZip-Recon computes routing-aware scores during KVZip’s reconstruction pass, enabling reuse of the compressed cache across downstream requests without an extra forward pass.Original MoEZip conditions anchor selection on the downstream instruction.Source: §3 · p. 6
  • MoEZip adds lightweight selection-time computation while preserving retained-cache size and the decoding procedure after eviction.Reported wall-clock overhead is 0.88s on Qasper and 0.86s on average across 4K–32K NIAH contexts.Source: §3 · p. 6

4 Experiment

MoEZip is evaluated against KVZip, SnapKV, and PyramidKV across long-context benchmarks, cache budgets, models, and routing types. It is strongest under aggressive compression, and analyses test its components and the role of routing changes.

4.1 Experimental Settings

  • Experiments cover Qwen3-30B-A3B-Instruct and GLM-4.7-Flash, 12 LongBench tasks, RULER, and NIAH at 4K–32K contexts, with keep ratios from 0.1 to 0.9.The models use softmax and sigmoid routers, respectively; the comparison matches retained-KV budgets and other evaluation settings.Source: §4.1 · pp. 6–7

4.2 Main Results

  • MoEZip consistently outperforms strong baselines on LongBench, NIAH, and RULER, with its clearest advantages under aggressive KV compression.On NIAH, it remains robust across context lengths and target positions; on RULER, it reaches high performance with a smaller KV budget.Source: §4.2 · pp. 7–8, Figure 5 · p. 7
  • MoEZip remains superior on average across 12 LongBench tasks on sigmoid-routed GLM-4.7-Flash and improves the 10-task LongBench macro from 27.6 with KVZip to 44.7 at keep ratio 0.1 on Qwen3-235B-A22B-Instruct-2507.Figure 6 reports the GLM-4.7-Flash comparison; the Qwen3 result extends evaluation across model scales.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 Analyses

  • Removing Fisher-trace fragility scoring or adaptive budget allocation reduces performance, with larger gaps under aggressive compression.The Fisher-trace comparisons also report that higher Fisher-trace bins correspond to higher expert-flip rates under compressed KV.Source: §4.3 · p. 8
  • Replacing max pooling with mean pooling hurts performance across nearly all cache ratios, while adding MoEZip scoring to SnapKV improves average performance across a wide range of ratios.Both differences are particularly notable under aggressive compression.Source: §4.3 · pp. 8–9
  • At keep ratio 0.1, MoEZip-Recon improves over KVZip by 22.44 points and retains 88% of the instruction-conditioned variant’s improvement while enabling cross-request cache reuse.KVZip and MoEZip-Recon use the same fixed reconstruction query at matched cache budgets.Source: Table 1 · p. 8, §4.3 · p. 9

4.4 Routing Intervention with a Fixed Retained Cache

  • With the retained cache fixed, restoring instruction routing reduces answer NLL by 0.70 for KVZip and 0.27 for MoEZip, but does not recover all effects of KV eviction.The corresponding freedecoding LongBench gains are 10.2 and 1.3 points, respectively.Source: §4.4 · p. 9
  • On the fixed KVZip-retained cache, correcting 25%, 50%, and 75% of routing disagreements recovers 23.8%, 56.7%, and 80.5% of the improvement from complete correction.Full correction reduces NLL from 5.670 to 4.970, compared with 4.220 with full KV.Source: §4.4 · p. 10, Table 2 · p. 10

5 Related Work

Prior KV cache compression methods use attention-based selection, reconstruction, or other retention strategies, but were primarily developed for dense LLMs. MoEZip instead preserves routing behavior during compression and complements architecture-level efficiency methods.

5 Related Work

  • Prior compression approaches include attention sinks and heavy hitters, observation-window selection, layer- or head-wise budget allocation, retrieval/streaming-head distinctions, and context reconstruction.KVZip’s query-agnostic reconstruction supports cache reuse across multiple queries.Source: §5 · p. 10
  • These methods were primarily developed for dense LLMs, whereas MoEZip addresses eviction-induced router changes and unstable expert assignments in MoE models.The related-work discussion frames preserving routing behavior as MoEZip’s distinguishing focus.Source: §5 · p. 10
  • Unlike methods that modify sequence modeling or attention architecture, MoEZip focuses on inference-time KV cache compression and is presented as complementary to architecture redesign.Examples of architecture-level approaches include linear and sparse attention methods.Source: §5 · p. 10

6 Conclusion, Limitations, and Broader Impacts

The conclusion presents routing fragility as an underexplored principle for KV compression in MoE models and reports that MoEZip performs well across several benchmarks, especially under high compression. Its use may lower deployment costs, but access to internal routing and attention statistics limits black-box use.

6 Conclusion, Limitations, and Broader Impacts

  • Across LongBench, NIAH, and RULER, MoEZip consistently outperforms strong baselines, especially in the high-compression regime.The conclusion frames routing fragility as an underexplored principle for KV compression in MoE models.Source: §6 · p. 10
  • MoEZip requires router outputs and instruction-to-context attention statistics, which may not be exposed in black-box inference settings.This access requirement is the stated limitation on its use in such settings.Source: §6 · p. 10
  • The authors state that MoEZip may lower deployment costs for both beneficial applications and potential misuse.They also identify broader opportunities for routing-aware compression in sparse architectures.Source: §6 · p. 10

A Local KL Expansion and Router-Specific Fisher Forms

The appendix tests whether Fisher trace tracks routing instability and whether instruction-token router states can serve as a pre-generation surrogate for answer-token fragility. It also compares instruction-conditioned context selection with an answer-oracle diagnostic.

A.2 Token-Level Validation of the Fisher-Trace Proxy

  • Across HotpotQA and Qasper, higher Fisher-trace bins show higher expert-flip rates, with greater separation under more aggressive compression.The token-level validation bins observations by full-KV Fisher trace and measures mean top-1 expert-flip rate under compressed KV.Source: §A.2 · p. 16, Figure 8 · p. 17

A.2.1 Validation under Aggressive Compression

  • At keep ratio 0.1, the highest Fisher-trace quintile has 2.29–4.17 times the expert-flip rate of the lowest across 12 LongBench tasks.Flip rates increase monotonically across quintiles, supporting Fisher trace as a relative ranking signal under aggressive compression.Source: §A.2.1 · p. 17

A.3 Sample-Level Validation of the Fisher-Trace Proxy

  • Sample-level correlations between Fisher trace and compressed-KV flip rate are positive across HotpotQA and Qasper and strengthen as keep ratios decrease.This supports Fisher trace as a signal of sample-level routing fragility, beyond a local uncertainty heuristic.Source: §A.3 · pp. 17–18

A.4 Instruction-token surrogate validation.

  • Instruction-token Fisher scores significantly correlate with answer-token Fisher scores on every evaluated task, including QMSUM and MultiFieldQA.The reported correlations are ρ = 0.351 on QMSUM and ρ = 0.475 on MultiFieldQA; both have p < 0.05.Source: §A.4 · p. 18
  • The instruction-to-answer surrogate remains significant on QMSUM even at keep ratio 0.1, while the Fisher-to-flip-rate link varies with task structure.The weaker QMSUM link is attributed to diffuse routing-critical positions averaging out the per-position signal.Source: §A.4 · p. 19

A.4.1 Instruction-to-Answer Oracle Diagnostic

  • Across four LongBench tasks at keep ratio 0.1, instruction-conditioned selection recovers approximately 82% of the oracle’s aggregate improvement over KVZip.Recovery is 97% on passage retrieval and 87% on HotpotQA, versus 43% on Qasper and 20% on QMSUM.Source: §A.4.1 · p. 19
  • Across five RULER/NIAH settings with 4K-token contexts, instruction-conditioned selection recovers approximately 89% of the oracle’s aggregate improvement.Recovery is 97% at keep ratio 0.2 and 75% at keep ratio 0.1.Source: §A.4.1 · p. 20
  • On RULER QA-1 (SQuAD), instruction- and answer-conditioned scores correlate at 0.99, with anchor-set Jaccard similarity 0.94 and answer-anchor recall 0.97.Recall is the fraction of oracle anchors also selected by instruction-based selection.Source: §A.4.1 · p. 20

A.1 Proof of Proposition 1

The proof connects local changes in routing distributions to Fisher information and derives router-specific forms for softmax- and sigmoid-type routing. The Fisher trace supplies a direction-agnostic measure of local routing sensitivity.

A.1 Proof of Proposition 1

  • For softmax routing, the negative log-likelihood Hessian with respect to logits is the Jacobian diag(g) − gg^T.The derivation uses the score g − x for g = softmax(z).Source: §A.1 · p. 16
  • For sigmoid routing, the Fisher matrix is diagonal, with entries g_e(z)(1 − g_e(z)); the router-specific trace expressions require O(E) computation.The sigmoid gate is g_e(z) = σ(z_e).Source: §A.1 · p. 16
  • The Fisher trace is the isotropic expectation of the direction-dependent quadratic KL expansion and equals the expected squared norm of the score function.This motivates a direction-agnostic local sensitivity measure without assuming that actual KV-eviction shifts follow the isotropic model.Source: §A.1 · p. 16

B Worst-case Analysis

The worst-case analysis considers the routing perturbation direction that maximizes Fisher curvature and derives bounds or exact forms for its largest eigenvalue. Empirically, these alternatives do not consistently improve on Fisher trace, which remains the default proxy.

B Worst-case Analysis

  • The worst-case perturbation aligns with the eigenvector for the largest eigenvalue of the Fisher information matrix.For softmax routing, the paper gives an O(E) upper bound; for sigmoid routing, the maximum eigenvalue is available exactly in O(E).Source: B Worst-case Analysis · pp. 20–21
  • Worst-case curvature scores are competitive with Fisher trace in several moderate-compression settings but can be less stable under aggressive compression.The bound-based score sometimes drops earlier or more sharply, and tighter worst-case curvature does not necessarily improve context-position ranking.Source: Figure 9 · p. 22, Figure 10 · p. 22, B Worst-case Analysis · p. 23
  • MoEZip uses Fisher trace by default because it gives a smoother direction-agnostic summary and more stable performance across tasks and compression ratios.The paper reports that worst-case scores do not consistently outperform Fisher trace.Source: B Worst-case Analysis · p. 23

C Implementation Details

MoEZip adds routing-aware anchors to a base KV compressor using prefill-stage router and attention signals, then allocates anchors according to sample-level fragility. The implementation reuses scoring statistics, with modest reported overhead and stable performance across tested anchor-ratio settings.

C Implementation Details

  • MoEZip uses one prefill pass to collect instruction-token router gates and instruction-to-context attention, then scores and selects routing-aware anchors alongside the base compressor.It leaves the base method’s masking and allocation rule unchanged.Source: C Implementation Details · p. 23
  • The anchor budget is a clipped fraction of the base retention budget, with bounds intended to prevent empty or overly dominant anchor sets.The default bounds are ρmin = 0.05, ρmax = 0.25, Bmin = 32, and Bmax = 128.Source: C Implementation Details · pp. 23–24

D Complexity

  • On Qwen3-30B-A3B at KV cache ratio 0.5, Fisher-based scoring adds 0.27–1.79 seconds across context lengths.The overhead is reported for NIAH scoring time.Source: Figure 11 · p. 24

C Implementation Details

  • MoEZip-Recon uses KVZip’s fixed reconstruction query and can produce a compressed cache reusable across downstream requests.It reuses statistics from KVZip’s reconstruction pass and needs no additional model forward pass beyond that pass.Source: C Implementation Details · p. 25

D Complexity

  • Additional scoring costs O(|M||I|E) for router fragility and O(|M||I|L) for attention aggregation, with the latter dominating when L ≫ E.The method reuses router outputs and attention statistics and adds no transformer forward pass.Source: D Complexity · p. 25

E Additional Analyses

  • Across QMSum, Qasper, MultiFieldQA, and 2WikiMQA, performance is relatively stable as ρmax varies from 0.1 to 0.3 with ρmin fixed at 0.05.The default ρmax = 0.25 provides strong or near-best performance; overly small values can slightly weaken high-compression robustness.Source: Figure 12 · p. 25

E.1 Sensitivity Analysis of ρmin and ρmax.

MoEZip’s performance is robust to its adaptive anchor-budget settings: the default (ρmin, ρmax) = (0.05, 0.25) performs strongly without careful per-task tuning.

E.1 Sensitivity Analysis of ρmin and ρmax.

  • Changing ρmax around the default leaves performance generally robust; values above 0.25 bring only marginal changes, while values that are too small can reduce robustness under aggressive compression.The default ρmax = 0.25 achieves strong or near-best performance.Source: §E.1 · pp. 25–26
  • Varying ρmin produces only minor performance changes across tasks and cache ratios when ρmax is fixed at 0.25.The figure reports results for ρmin ∈ {0.0, 0.05, 0.1}; the default 0.05 performs consistently strongly.Source: §E.1 · p. 26, Figure 13 · p. 26
  • The default (ρmin, ρmax) = (0.05, 0.25) balances performance across tasks without requiring careful per-task tuning.ρmin sets minimum route-aware protection, while ρmax sets the maximum anchor fraction for highly fragile samples.Source: §E.1 · p. 26

E.2 Answer-Token Flip Rate per Compression Ratio

Across compression ratios, MoEZip has lower answer-token flip rates than KVZip and greater task robustness. Router instability increases with compression, varies across layers and inputs, and informs MoEZip’s anchor selection and budget allocation.

E.2 Answer-Token Flip Rate per Compression Ratio

  • MoEZip consistently has lower answer-token flip rates than KVZip across tasks, with improved task robustness as compression becomes aggressive.KVZip can undergo a sharp quality collapse after its flip rate rises; MoEZip delays this transition.Source: §E.2 · pp. 26–27

E.3 Layer-Wise Router Instability under KV Compression

  • Router flips increase with compression, becoming more pronounced in deeper middle-to-late MoE layers for both KVZip and SnapKV.MoEZip uses a small shared set of routing-aware anchors across layers, while the base scorer allocates the remaining budget per layer and head.Source: §E.3 · p. 28

E.4 Router Flip Rate across Different Inputs for the Same Context

  • Inputs sharing the same context can have substantially different router flip rates, so context-side saliency alone cannot explain routing instability.MoEZip uses instruction-token fragility and instruction-to-context attention to select input-specific anchors.Source: §E.4 · p. 28

E.5 Paired Bootstrap Confidence Intervals

  • Paired task-level bootstrap intervals exclude zero for MoEZip’s improvements over all baselines at every reported keep ratio on LongBench and RULER.Tables 9 and 10 report 95% percentile intervals from 10,000 task-resampling samples.Source: Table 9 · p. 28, Table 10 · p. 28, §E.5 · p. 29

E.6 Query-Agnostic Compression and Cache Reuse

  • At keep ratio 0.1, MoEZip-Recon improves over KVZip by 22.44 points while preserving cross-request cache reuse, retaining 88% of instruction-conditioned MoEZip’s improvement over KVZip.MoEZip-Recon uses a fixed reconstruction query; the reuse comparison concerns the final compressed context cache, not identical end-to-end costs.Source: §E.6 · p. 29, Table 11 · p. 29

E.7 Global versus Fine-Grained Anchor Selection

  • Global anchor selection exceeds per-layer and per-GQA selection by 8.6 and 5.6 macro points, respectively, winning on all 12 tasks versus per-layer and 11 of 12 versus per-GQA.The remaining KV budget is filled independently by the base compressor at each layer and head.Source: §E.7 · p. 29, Table 12 · p. 29

E.8 Complementarity of Routing-Aware Anchors

  • KVZip already retains only 15–24% of MoEZip’s anchors in an average cache, but low overlap alone does not establish utility; a matched query-agnostic comparison improves the macro from 19.51 to 41.95.At Bf = 128, approximately 76–85% of anchor slots protect positions not already retained by KVZip in that cache.Source: §E.8 · p. 30, Table 13 · p. 30

E.9 Evaluation on a 235B MoE Model

On the evaluated Qwen3-235B model at keep ratio 0.1, MoEZip improves LongBench scores over KVZip and attention-only selection. Its advantage also persists under the evaluated INT8 cache configuration.

E.9 Evaluation on a 235B MoE Model

  • On a four-task subset, MoEZip raises the macro from 16.7 with KVZip to 35.5 and exceeds attention-only selection by 11.4 points.The attention-only comparison supports a contribution beyond instruction-conditioned attention alone.Source: §E.9 · p. 30, Table 14 · p. 31
  • Across 10 LongBench tasks, MoEZip raises the macro from 27.6 to 44.7 and outperforms KVZip on every task at keep ratio 0.1.Gains span retrieval, classification, code completion, and question answering.Source: §E.9 · p. 30, Table 15 · p. 31
  • The highest Fisher quintile has 2.8× and 2.5× the lowest-quintile flip rate on Qwen3-30B and Qwen3-235B, respectively, supporting transfer of routing signals across these scales.The diagnostic uses 50 sampled examples per task.Source: §E.9 · pp. 30–31

E.10 Composition with INT8 KV Quantization

  • INT8 reduces MoEZip’s macro by 0.6 points, while its advantage over KVZip changes from 13.6 to 13.3 points, preserving approximately 97.8% of the BF16 gain.This supports compatibility with the evaluated INT8 configuration; lower-bit quantization is not evaluated.Source: §E.10 · p. 31

E.11 Cross-Layer Routing Associations and Anchor Examples

At a 10% KV keep ratio, answer-token routing flips at adjacent MoE layers are associated, while selected anchors include both answer-relevant details and structural markers. The checklist also records the method’s access requirements and the authors’ reproducibility and uncertainty-reporting practices.

E.11 Cross-Layer Routing Associations and Anchor Examples

  • At a KV keep ratio of 0.1, a flip at one MoE layer is associated with a higher probability of a flip at the next, but this does not establish causal propagation.The conditional probabilities pool answer-token observations across evaluated tasks and adjacent-layer pairs.Source: §E.11 · p. 31, Table 18 · p. 33
  • Selected global anchors include answer-relevant entities and factual details, as well as schema markers and evidence delimiters.These are qualitative examples, not a quantitative classification or causal explanation of anchor utility.Source: §E.11 · pp. 31–32

NeurIPS Paper Checklist

  • The authors identify access to router outputs and token-to-context attention statistics as requirements, and describe Fisher trace as a practical proxy rather than an exact predictor.The stated limitation concerns prediction of eviction-induced perturbations.Source: NeurIPS Paper Checklist · p. 34
  • The paper specifies models, datasets, baselines, keep ratios, compression and decoding setups, and implementation details, and states that anonymized code and scripts will be provided.The experiments use public benchmarks and open checkpoints; no model training or optimizer is involved.Source: NeurIPS Paper Checklist · pp. 35–36
  • The authors report paired task-level bootstrap confidence intervals for representative LongBench and RULER aggregates, using 10,000 resamples, but not repeated full inference runs.The stated reason for omitting repeated inference runs is the computational cost of evaluating multiple large MoE models across tasks, baselines, and cache ratios.Source: NeurIPS Paper Checklist · p. 37
  • The authors report using NVIDIA A100 80GB GPUs and providing per-run GPU counts and approximate GPU-hour estimates.The compute-environment information is stated to be in Appendix C.Source: NeurIPS Paper Checklist · p. 37

Related papers

Loading 2610.30206v1…