LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts

1. Introduction

• Existing MoE designs are largely guided by high-level sparsity arguments and offline throughput, with limited consideration of online latency, memory-bandwidth, and communication constraints.The paper frames the design objective as maximizing achievable accuracy under fixed inference-cost constraints.Source: §1 · p. 1
• The paper evaluates MoE design across accuracy per FLOP and accuracy per parameter, with the latter reflecting memory footprint, bandwidth, routing communication, and sharding overhead.These factors are often dominant bottlenecks in interactive, low-latency inference.Source: §1 · p. 1
• LatentMoE projects activations into a smaller latent space, reducing routed parameter loads and all-to-all traffic while enabling proportionally more experts and active experts at approximately constant inference cost.The projection and proportional scaling use the factor d/ℓ.Source: Figure 1 · p. 2, §1 · p. 2
• The study combines hardware–software co-design analysis across the throughput–latency frontier with accuracy measurements and theoretical analysis to identify structural inefficiencies in existing MoEs.The identified bottlenecks arise from expert parameterization, routing-induced all-to-all communication, and memory access patterns.Source: §1 · p. 2
• Across evaluated regimes, LatentMoE achieves higher accuracy at fixed inference cost or substantially lower inference cost at fixed accuracy than standard MoE architectures.The reported validation includes pretraining up to 95B parameters and over 1T tokens; the architecture was adopted by Nemotron-3 Super and Ultra and scaled to larger regimes (NVIDIA et al., 2025).Source: §1 · pp. 2–3

2. LatentMoE Core Design Principles

• The section frames its design analysis around a Qwen3-235B-A22B example deployed on 64 GB200 GPUs, with expert parallelism and data parallelism over the same group.It specifies 128 experts, 8 active experts per token, hidden dimension 4096, and intermediate dimension 1536.Source: §2 · p. 3

2.1. Memory Bandwidth Bottleneck

• At latency-critical batch sizes, MoE experts are bandwidth-bound when per-expert token counts are below the compute-bound threshold of 1418 in the Qwen3-235B-A22B example.The analysis requires arithmetic intensity I ≥ 1250 for compute-bound operation; typical latency-critical batches have a few hundred tokens per expert.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2. Communication Bottleneck

• In throughput-oriented deployments, all-to-all communication can dominate MoE execution; the GB200 NVL72 and Qwen3-235B-A22B example gives a communication-to-compute time ratio of approximately 9.The section identifies reducing routed hidden dimension d or active experts K as ways to mitigate communication overhead.Source: §2.2 · p. 5

2.3. Model Quality

• The effective nonlinear budget per token is tied to selected-expert width, so reducing active experts K or intermediate dimension m risks degrading model quality; preserving K·m is advised.Reducing hidden dimension below the task-specific feature rank r_eff discards task-relevant information and degrades accuracy.Source: §2.3 · p. 5
• Reducing hidden dimension d targets both memory-bandwidth and communication costs, while scaling active experts K and hidden width down risks model quality; scaling N and K up can improve expressivity if d remains above r_eff.The proposed transformation reduces d by α while increasing K by α, preserving memory-bandwidth and communication costs and improving expressivity and combinatorial sparsity.Source: §2.3 · p. 6

3. LatentMoE Architecture

• Each token is down-projected into Rℓ, processed by routed experts there, then aggregated and up-projected; routing and shared experts remain in R^d.Source: §3 · pp. 6–7
• ℓ-MoEeff compresses only the input dimension while preserving the intermediate dimension, then scales N by α to expand specialization without additional inference overhead.Source: §3 · pp. 6–7
• The default ℓ-MoEacc variant scales both expert count N and active experts K by α, exponentially increasing expert diversity.Source: §3 · p. 7
• Scaling K by α = d/ℓ keeps communication cost and memory bandwidth constant versus standard MoE while improving accuracy at iso-inference cost.Source: §3 · p. 7

4. Evaluation

• The design-space exploration uses 16B-total, 2B-active Transformer models for ablations and 95B-total, 8B-active models as a scaling test, and also trains hybrid Mamba-Attention MoEs.The evaluation compares model architectures and training configurations summarized in Tables 1 and 2.Source: §4 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.1. LatentMoE Ablations

• Validation quality is preserved at compression ratios α≤4, and compressing the hidden dimension 4× without increasing expert count causes significant quality degradation.The study adopts α=4 for subsequent experiments and reports that setting remains effective at the larger scale.Source: §4.1 · pp. 8–10, Figure 3 · p. 9, Figure 4 · p. 9
• For the 16B-total, 2B-active model, ℓ-MoEeff matches baseline accuracy and ℓ-MoEacc achieves lower validation loss at ℓ=512 and α=4.The paper recommends ℓ-MoEacc for Pareto-optimal accuracy versus inference cost.Source: §4.1 · p. 10, Figure 5 · p. 10

4.2. LatentMoE Scaling Studies

• At 95B parameters, ℓ-MoEeff matches baseline convergence, while ℓ-MoEacc performs better; at 300B tokens, the comparison reports downstream accuracy results.The LatentMoE variants use the baseline Transformer’s hyperparameters; further tuning might improve accuracy.Source: §4.2 · p. 10, Figure 6 · p. 11, Table 3 · p. 11
• After 1T training tokens, hybrid ℓ-MoEacc achieves higher accuracy than baseline across all tasks, while ℓ-MoEeff is comparable to or better than the standard granular MoE baseline.The compared hybrid models use identical hyperparameters.Source: §4.2 · p. 11, Table 4 · p. 12

4.3. Inference Performance

• At higher concurrencies, measured per-GPU throughput drops by at most 6% for ℓ-MoEacc versus standard MoE.The authors note possible software optimizations, including separate CUDA streams and specialized smaller-matrix GEMM kernels.Source: §4.3 · pp. 11–12, Table 5 · p. 12

4.3.1. Projected Serving Impact at Trillion-Parameter Scale

• The trillion-parameter serving projections simulate over 200K operating points for decode-heavy chunked-piggybacking and prefill-heavy disaggregated traffic.The iso-accuracy standard-MoE baseline is constructed using an estimated Effective Parameter Multiplier of about 1.35×, producing a 1.35T model by scaling depth from 61 to 80 layers.Source: §4.3.1 · pp. 12–13
• 1.24×–3.46× slower: the accuracy-matched Kimi-K2-1.35T standard-MoE model is slower than Kimi-K2-1T-LatentMoE across the projected serving frontier.The projection estimates about 350B additional parameters for the standard-MoE construction; native Kimi-K2-1T remains within up to ∼9% of the LatentMoE variant.Source: Figure 7 · p. 13, §4.3.1 · pp. 13–14

• LatentMoE is presented as an alternative MoE design that achieves higher accuracy under iso-parameter and iso-FLOP constraints.Source: §5 · p. 14
• Quantization, sparsity, expert pruning, and merging are orthogonal to LatentMoE and can be composed with it for further efficiency gains.Source: §5 · p. 14
• Unlike MoLAE, a posttraining low-rank expert-compression method, LatentMoE couples compression with increased expressivity and combinatorial sparsity.MoLAE's grouped latent projections and partial expert compression forgo dispatch communication savings and limit memory-bandwidth reduction.Source: §5 · p. 14
• Manifold-Constrained Hyper-Connections (mHC) (Xie et al., 2026) improves quality under iso-compute through residual-path changes, and the authors consider it complementary to LatentMoE.Combining the approaches is left for future work.Source: §5 · p. 14

• LatentMoE is designed to maximize accuracy per FLOP and per parameter by accounting for inference systems' dominant memory-bandwidth and communication bottlenecks.Source: §6 · p. 14
• Projecting tokens into a lower-dimensional latent space reduces routing all-to-all communication, memory bandwidth, and computation required per expert.The resulting savings are reinvested in scaling expert count and routing diversity without increasing inference cost.Source: §6 · p. 14
• Up to 95B parameters, across hybrid architectures and projected trillion-parameter serving scenarios, LatentMoE consistently outperforms standard MoEs on the accuracy–efficiency Pareto frontier.Source: §6 · p. 15