PaperlayerView exact revision on arXiv

Source-linked AI summary

CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training

Jing Li, Jian Meng, Yingmeng Gao, Suming Qiu, Linyuan Qiu, Dongfang Li, Baotian Hu, Binfan Zheng, Rongqian Zhao, Weijian Sun, Xin Chen

arXiv:2610.05744v1cs.LGcs.AI

TL;DR

MoE routing can create workload skew that degrades training efficiency and stability, with overloaded experts risking out-of-memory failures at large scale. CIPHER-MoE constrains expert capacity through token selection, and reports 1.10×–1.94× training speedups across large MoE models while preserving training quality.

  • Problem

    Non-uniform token routing creates imbalanced expert workloads that degrade communication efficiency, hardware utilization, and training stability, and can trigger out-of-memory failures at scale.

    Source: §1 · pp. 1–2

  • Method

    CIPHER-MoE constrains expert learning capacity through capacity-aware token selection and dropping, including affinity-based filtering of candidate tokens.

    Source: §1 · p. 2, §3.2.1 · p. 5

  • Results

    Across large MoE models, CIPHER-MoE achieves 1.10×–1.94× measured training-time speedups while preserving training quality.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    On DeepSeek-V4-Pro, lower memory footprint enables successful training under limited resources without fixed routing.

    Source: §4.4 · p. 9

    The theoretical capacity analysis assumes independent candidate draws and a spherical reference model, with bounded signal-survival and ranking risks.

    Source: §3.2.1 · p. 5

Abstract

from arXiv · show

Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router's token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit capacity control to reduce hotspot expert workloads without additional hardware resources or complex runtime design. The proposed method has been evaluated on large-scale MoE models, including DeepSeek-V4-Pro, showing up to 64.9 percentage points Top-1 expert workload reduction and 1.10$\times$-1.94$\times$ training acceleration, while preserving the training quality. The source code will be released soon.

1 INTRODUCTION

MoE training faces workload imbalance as non-uniform routing overloads some experts, harming efficiency and stability. CIPHER-MoE constrains expert capacity to address this imbalance, with reported speedups across large MoE models.

1 INTRODUCTION

  • MoE architectures scale model capacity while activating only a small subset of experts per token, avoiding dense Transformers’ trillion-parameter compute and memory costs.MoE replaces the dense feed-forward network with sparsely activated experts and dynamically routes tokens to them.Source: §1 · p. 1
  • In DeepSeek-V4-Flash, 10% of experts consume over 88% of routing workload; skewed routing degrades training efficiency, communication, utilization, and stability.Overloaded experts can also trigger out-of-memory failures on large-scale clusters.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Existing system-level balancing adds replicas, communication, or runtime scheduling, while routing penalties, heuristic batch balancing, and routing distillation have stated limitations.The cited work says these approaches may incur overhead and complexity, fail to handle instantaneous memory peaks, or scale poorly to trillion-parameter MoE models.Source: §1 · p. 2
  • CIPHER-MoE is a plug-and-play method for stable, efficient trillion-parameter MoE training that investigates bounded learning capacity across experts and demonstrates feasibility of dropping tokens from expert training.The authors validate it through theoretical analysis and empirical experiments across MoE models from 284B to 1.6T parameters.Source: §1 · p. 2
  • Across DeepSeek-V3, DeepSeek-V4-Flash, GLM-5, and DeepSeek-V4-Pro, CIPHER-MoE achieves 1.10×–1.94× measured training-time speedups.The contribution statement also claims preserved LLM training quality.Source: §1 · p. 2

2 RELATED WORK

Related work addresses MoE imbalance through system-level execution changes, algorithmic routing objectives, and capacity-aware token–expert assignment. CIPHER-MoE distinguishes itself by bounding capacity for affinity-aware assignments without replication or complex runtime orchestration.

System-level workload balancing and efficient MoE execution

  • System-level approaches rebalance expert-to-hardware mappings or optimize sparse execution, but rely on mechanisms such as replication, migration, profiling, or specialized scheduling.Examples include Faster-MoE and FlexMoE [Nie et al., 2023], Tutel and PipeMoE [Hwang et al., 2023], and MegaBlocks [Gale et al., 2023].Source: §2 · p. 3

Algorithmic load balancing and expert specialization

Capacity-aware token–expert assignment

  • Capacity-aware alternatives include LocMoE [Li et al., 2024] and Expert Choice Routing [Zhou et al., 2022], while CIPHER-MoE derives a sufficient capacity bound for affinity-preserving bidirectional assignment.Tokens identify preferred experts, which select their highest-affinity tokens within budget; excess assignments are rerouted without expert replication or parameter migration.Source: §2 · p. 3

3 PROPOSED METHOD: CIPHER-MOE

CIPHER-MoE controls expert workloads after conventional token-side Top-K routing, using similarity-aware admission and explicit per-expert capacity. Its design combines capacity analysis, assignment handling, and a locality-aware training objective.

3.1 PRELIMINARIES OF MIXTURE-OF-EXPERT TRAINING

  • Fixed routing balances assignments but removes affinity-based selection, risking expert homogenization; Table 1 reports suboptimal performance versus vanilla MoE training.DeepSeek-V4-Pro with freely explored routing is also reported to fail before its first iteration from OOM under the described setup.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 CIPHER-MOE: BIDIRECTIONAL EXPERT WORKLOAD BALANCING

  • Similarity-based admission filters each expert’s candidate tokens by threshold and retains at most its capacity, prioritizing the highest-cosine-similarity candidates.Capacity may vary by layer, expert, and training stage but is fixed within each micro-batch; rejection by one expert leaves the token’s other selected-expert paths intact.Source: §3.2 · pp. 4–5

3.2.1 THEORETICAL ANALYSIS OF SIMILARITY-BASED TOKEN SELECTION

  • Under A1–A3, sufficient capacity Be for similarity selection can be lower than natural-order selection’s necessary capacity; Ce ≥ Be suffices for signal retention, not final model quality.The analysis compares both policies on the same candidate pool and target, with assumptions governing signal availability, filtering survival, and ranking.Source: §3.2.1 · pp. 5–6

3.2.2 STRICT TOKEN DROP VS. TOKEN–EXPERT REROUTE

  • Strict token drop removes rejected assignments while preserving other accepted paths; rerouting sends them to qualified experts with spare capacity, but can underperform strict dropping.The paper attributes rerouting’s weaker domain-specific performance to shifted expert specialization and gradient interference; shared experts still process every token.Source: §3.2.2 · p. 6

3.2.3 TRAINING OBJECTIVE OF CIPHER-MOE

  • CIPHER-MoE adds a locality loss that encourages assignments toward experts local to the current expert-parallel group, alongside the training objective.The assignment distribution used in the objective depends on whether strict-drop or reroute selection is applied.Source: §3.2.3 · p. 7

4 EXPERIMENTAL RESULTS

Experiments cover 284B–1.6T MoE models and compare CIPHER-MoE with vanilla, fixed-routing, and LocMoE training.

4.1 TRAINING EFFICIENCY AND MEMORY USAGE

  • 1.47× maximum per-step speedup on DeepSeek-V4-Flash; CIPHER-MoE reports 1.42× versus other baselines and approximately 87.5% device-memory allocation.Strict and reroute policy overheads are 1.57% and 3.85% of total training time, respectively.Source: Figure 3 · p. 8, §4.1 · p. 8

4.2 LLM TRAINING QUALITY WITH CIPHER-MOE

  • Both CIPHER-strict and CIPHER-reroute converge smoothly to the same low-loss regime as reference methods, with the best reported quality-efficiency tradeoff across model sizes and architectures.Table 2 is described as showing optimal speedup and quality for CIPHER-MoE.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 EXPERT UPDATE DISTRIBUTIONS AND DEGREE OF HOMOGENIZATION

  • CIPHER-MoE promotes expert differentiation; CIPHER-strict shows the clearest router-adaptation separation, and update dynamics agree with routed-representation geometry.Affinity-aware capacity control suppresses weak or repetitive assignments, while selective rerouting sends overflow tokens to compatible experts with capacity.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

4.4 TRAINING EFFICIENCY AND EVALUATION OF DEEPSEEK-V4-PRO

  • DeepSeek-V4-Pro's baseline fails under limited memory because of OOM errors, while CIPHER-MoE trains stably at global batch size 8192 and outperforms the fixed-router baseline on multiple OR benchmarks.CIPHER-MoE also achieves a higher weighted-average score.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION

CIPHER-MoE addresses expert workload imbalance across hundred-billion- to trillion-parameter models through capacity-aware token selection and rerouting.

5 CONCLUSION

  • Across models from 284B to 1.6T parameters, CIPHER-MoE achieves measured training speedups of 1.10×–1.94× while preserving domain-specific and general capabilities.Evaluations include DeepSeek-V3, DeepSeek-V4-Flash, GLM-5, and DeepSeek-V4-Pro.Source: §5 · p. 9
  • CIPHER-MoE balances workloads by constraining overloaded experts through capacity-aware token selection and rerouting.The work also reveals a bounded relationship between expert capacity and workload distribution.Source: §5 · p. 9
  • The results support CIPHER-MoE as a simple, efficient, stable, and scalable foundation for large-scale MoE training.Source: §5 · p. 9

A.1 GRADIENT DYNAMICS AND TRAINING STABILITY OF MOE MODELS

Across the evaluated model-dataset configurations, CIPHER-MoE maintains well-behaved gradient dynamics and suppresses routing hotspots.

A.1 GRADIENT DYNAMICS AND TRAINING STABILITY OF MOE MODELS

  • Gradient norms remain finite and nondegenerate, generally settling from initial transients into stable ranges without evidence of explosion or vanishing.Figure 6 covers DeepSeek-V4-Flash, DeepSeek-V3, and GLM-5 on MedMCQA, SIQA, and OR.Source: §A.1 · p. 14, Figure 6 · p. 14
  • CIPHER-strict often has smoother, lower gradient norms than the baseline, while CIPHER-reroute remains bounded despite occasional task-dependent fluctuations.Source: §A.1 · p. 14
  • Peak expert traffic falls from as high as 83.6% to 18.7%; strict minimizes routing volume, while reroute yields the most balanced utilization with broader expert coverage.These routing patterns accompany the reported stable optimization dynamics.Source: §A.1 · p. 14

A.2 TRAINING EFFICIENCY AND STABILITY OF QWEN MODELS

The Qwen3.6-35B-A3B evaluation reports limited efficiency gains alongside stable step times and convergent gradient norms.

A.2 TRAINING EFFICIENCY AND STABILITY OF QWEN MODELS

  • Efficiency improvement remains below 5% for Qwen3.6-35B-A3B, whose 2048 hidden-state dimensions do not satisfy the favorable dimension-margin regime described for CIPHER-MoE.Source: §A.2 · p. 14
  • Figure 7 tracks expert traffic across representative layers, activated experts per token, and load entropy during training.Source: Figure 7 · p. 15
  • CIPHER maintains stable step times and convergent gradient norms on Qwen3.6-35B-A3B, with CIPHER-strict showing smoother late-stage optimization.Figure 8 presents step-time trends, post-warm-up step-time distributions, and raw gradient norms.Source: Figure 8 · p. 15
  • General-benchmark evaluation results for OR SFT are reported as percentages.Source: Table 4 · p. 15

A.3 EVALUATION RESULTS OF GENERAL BENCHMARKS

The section evaluates general capabilities across five categories and reports that CIPHER-MoE can strengthen broad transfer rather than merely preserve it. For DeepSeek-V3 after SIQA SFT, CIPHER-strict improves the macro-average across five benchmarks by 4.3% relative to baseline.

A.3 EVALUATION RESULTS OF GENERAL BENCHMARKS

  • The evaluation covers mathematics, general knowledge, logical reasoning, reading comprehension, and instruction following.The listed benchmark sets include AIME-26, AMC, GSM8K, MATH-500, ARC, GPQA-D, MMLU-Pro, HellaSwag, BBH, DROP, and IFEval.Source: §A.3 · pp. 15–16
  • 4.3% relative improvement: CIPHER-strict raises the DeepSeek-V3 SIQA-SFT macro-average across ARC, BBH, GPQA-D, HellaSwag, and MMLU-Pro versus baseline.The reported result is in the general-benchmarks evaluation for SIQA SFT.Source: §A.3 · p. 16, Table 6 · p. 16
  • The authors report that CIPHER-MoE strengthens broad transfer performance rather than merely preserving it.They suggest selective routing suppresses low-value signals and concentrates expert updates on informative token–expert associations.Source: §A.3 · p. 16

A.4.1 COMPARISON TASK AND ASSUMPTIONS

The comparison analyzes whether affinity-based admission retains informative tokens with fewer expert-execution slots than natural-order admission on the same candidate pool. Its assumptions distinguish signal availability, candidate recall, and conditional ranking behavior.

A.4.1 COMPARISON TASK AND ASSUMPTIONS

  • The comparison uses the same candidate pool and retention target, contrasting the first C offered tokens with affinity-ranked candidates admitted into C execution slots.Affinity computation and candidate selection are not counted as expert-execution capacity.Source: §A.4.1 · p. 16
  • A1 assumes independent candidate draws from an expert’s exposure distribution and positive availability of a discriminative token; its geometric reference bounds signal probability using dimension and affinity margin.The exposure-conditioned distribution need not be unconditioned uniform spherical; the bound does not require independence between discrimination and exposure.Source: §A.4.1 · pp. 16–17
  • A2 requires a chosen discriminative witness to reach the ranking pool and survive thresholding, separating candidate-recall failure from capacity truncation.Increasing execution capacity cannot recover a witness already excluded from the ranking pool.Source: §A.4.1 · p. 17
  • A3 bounds, conditionally, how many competitors rank ahead of the surviving witness; dependence is allowed if the bounds hold after witness selection and prior observations.A marginal ranking-error rate alone does not establish this conditional guarantee.Source: §A.4.1 · p. 17

A.4.2 CAPACITY SEPARATION FOR AFFINITY-BASED ADMISSION

The analysis bounds the capacity needed for affinity-based admission to retain a discriminative witness, then sharpens the bound under ranking-error controls. It also distinguishes signal retention from output fidelity and states the assumptions limiting these guarantees.

A.4.2 CAPACITY SEPARATION FOR AFFINITY-BASED ADMISSION

  • Under the stated assumptions, capacity at least B_e is sufficient for affinity admission to retain a discriminative witness, while natural-order admission must budget for encountering a scarce signal.B_e is a sufficient budget, not an exact minimum or an optimal training-time configuration; the finite-pool constraint still applies.Source: §A.4.2 · pp. 17–19

A.4.3 SHARPER CAPACITY BOUNDS AND STABILITY CONDITIONS

  • The geometric bound holds at finite dimension, and increasing dimension alone does not yield growing separation when δ = c/√d for fixed c.For bounded μ and fixed exposure and availability parameters, dδ(d)^2 → ∞ is sufficient for growing separation; availability must remain valid.Source: §A.4.2 · p. 18, §A.4.3 · p. 20

A.4.2 CAPACITY SEPARATION FOR AFFINITY-BASED ADMISSION

  • The capacity theorem does not prescribe processing more tokens than are available; when π_e = 1, natural-order admission needs one slot and this separation is nontrivial no longer.The assumed scarcity and availability cannot both hold for a pool if the claimed natural-order lower bound exceeds N_e.Source: §A.4.2 · p. 19
  • If rerouting preserves strict assignments and adds only threshold-qualified tokens within remaining capacity, every discriminative witness retained by strict is retained by reroute.The guarantee does not apply if rerouting evicts a strictly accepted token, and additional assignments can change expert mixtures and training updates.Source: §A.4.2 · p. 19

A.4.3 SHARPER CAPACITY BOUNDS AND STABILITY CONDITIONS

  • A bounded conditional ranking budget can yield B_e = O(1), but a larger candidate pool requires correspondingly stronger control of ranking errors.The stated scaling conditions are N_e(ϱ + ν) = O(1) and N_eχe^-G(d,δ_eff) = O(1), with ε − η bounded away from zero.Source: §A.4.3 · p. 20

A.4.4 SIGNAL RETENTION AND OUTPUT FIDELITY

  • The output-fidelity bounds apply at fixed expert outputs and downstream computation, not to accumulated parameter changes during optimization.The section keeps capacity analysis distinct from empirical training stability, specialization, and final capability.Source: §A.4.4 · p. 21
Loading 2610.05744v1…