Source-linked AI summary
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini, Philip Levis
TL;DR
Distributed MoE inference is limited by costly communication when token-selected experts reside on other GPUs. FoE partitions attention heads and experts into localized groups with per-group routing and cross-group synchronization. It reports up to 5.2× lower forward-pass latency, 3.62× lower TTFT, and 1.95× lower TBT while matching baseline quality, although deployment benefits depend on distributed settings and topology.
Problem
Distributed MoE routing requires expensive network communication for tokens assigned to experts on different GPUs, creating a major inference bottleneck.
Method
FoE partitions KV heads and experts into H independent groups, routes k/H experts per group, and synchronizes post-attention residuals across groups.
Results
Up to 5.2× lower end-to-end forward-pass latency, 3.62× lower TTFT, and 1.95× lower TBT are reported while matching baseline MoE quality.
Takeaways & Limitations
FoE structurally localizes expert communication, with benefits that compound in multi-node deployments while preserving comparable generation quality.
Takeaways & Limitations
FoE benefits apply to distributed deployments, while the evaluation assumes homogeneous topology and skewed setups may require placement-aware tuning.
Abstract
from arXiv · showhide
Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing the end-to-end forward-pass latency by up to 5.2x, TTFT by 3.62x, and TBT by 1.95x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.
1 Introduction
LLMs scale computational demands with parameter count, while MoE reduces active computation but introduces distributed routing communication as a major bottleneck. FoE restructures routing into localized expert groups and reports substantially lower communication and inference latency with comparable generation quality.
- LLM computation scales with parameter count, limiting training efficiency and inference deployment.
- MoE sparsely activates experts per token, reducing active computation while preserving large total parameter counts.
- 68%+ of end-to-end latency can come from distributed expert communication when selected experts reside on different GPUs.
- FoE partitions KV heads and experts into H groups, using per-group routing and cross-group residual synchronization to localize dispatch traffic.
- Up to 5.2× lower end-to-end forward-pass latency, 3.62× lower TTFT, and 1.95× lower TBT are reported on LongBench with comparable generation quality.
2 Background
MoE replaces dense feed-forward computation with routed expert subsets, enabling parameter scaling with lower per-token computation. Distributed inference relies on expert parallelism and systems optimizations, but these generally retain global communication limits that FoE addresses structurally.
- MoE replaces dense feed-forward blocks with multiple experts and activates only a token-selected subset.
- Token-choice MoE combines the weighted outputs of the top-k experts selected for each input token.
- Expert parallelism scatters experts across devices and requires token activations to move to the devices holding selected experts.
- Prior systems mitigate MoE costs through load balancing, expert replication, communication-computation overlap, kernel fusion, and routing-aware placement.
- FoE differs by restructuring global routing into disjoint balanced expert groups that natively confine communication topologies.
3 Federation of Experts
Federation of Experts restructures MoE into independent expert groups that partition KV heads, experts, and GPUs, confining routing communication and increasing local expert resolution. Its communication analysis reports exact single-node reductions and favorable multi-node scaling under bandwidth-weighted costs.
- Architecture Formulation: FoE replaces one global expert-parallel group with H independent groups, each owning 1/H of the KV heads, experts, and GPUs.Routing selects k/H experts within each group, while group representations are maintained between intermediate layers.
- Communication Analysis: FoE raises the local activation rate from ρ_moe = 1/G to ρ_foe = min(H/G, 1), reducing effective all-to-all volume.Under balanced routing, effective per-layer A2A volume is 2k(1 −ρ) · S · d.
- Communication Analysis: In single-node deployments with G ≤H, FoE eliminates the A2A term and leaves only a cross-group all-reduce.The remaining all-reduce has volume 2(G −1)/G.
- Communication Analysis: Single-node communication is reduced exactly k× for any G ≤H, including both model-matched and under-provisioned deployments.The result is especially relevant to edge boxes, workstations, and other single-node inference settings.
- Communication Analysis: For multi-node deployments, FoE keeps dispatch and combine all-to-alls within nodes, while bandwidth-weighted costs account for slower inter-node links.The analysis uses r = B_intra/B_inter, typically 15–20 on H100-class hardware, to predict wall-clock latency.
- Communication Analysis: For k = 8, H = 8, and r = 20, the communication-time asymptote is approximately 6.3× and is within about 5% of the actual formula for N ≥2.This quantifies the multi-node communication advantage under the stated deployment parameters.
4 Experiments
Experiments evaluate FoE against standard expert parallelism across single-node and multi-node inference, communication efficiency, load balancing, and generation quality. FoE preserves comparable model quality while improving localization, balancing, and latency-related performance.
- Experimental setup: FoE is evaluated on single-node 8-GPU and multi-node two-server configurations using FlexServe, with standard global-all-to-all expert parallelism as the primary baseline.The multi-node setup uses two identical 8-GPU servers connected via InfiniBand.
- Model quality and convergence: FoE pretraining follows the same convergence trajectory as MoE, with indistinguishable 1B cross-entropy curves at the end of Chinchilla-level training.The models are pretrained from scratch because standard MoE checkpoints cannot be evaluated off-the-shelf.
- Single-node inference: 3.1×: FoE reduces average single-node TTFT by up to 3.1× across Poisson request-arrival scales while improving tail behavior relative to MoE.The evaluation reports TTFT, TBT, and E2E latency using mean, p50, and p99 values.
- Multi-node inference: 3.44×: FoE’s TTFT speedup over standard MoE at arrival scale 0.5 rises from 2.66× single-node to 3.44× multi-node.At the same scale, average FoE p99 TTFT is 2.575s versus 7.066s for standard MoE.
- Locality and load balancing: 1.0: FoE achieves perfect LAR at EP size 8, compared with 0.39 for Sem-MoE, 0.21 for ExFlow, and 0.12 for MoETuner.FoE’s isolated expert groups structurally confine routing locally in the single-node configuration.
- Generation quality: FoE maintains comparable zero-shot accuracy to equivalently sized MoE baselines, indicating that communication savings do not significantly degrade model quality.The comparison uses models pretrained directly from scratch, and the differences remain within typical training seed variance.
5 Limitations
FoE reduces communication overhead but adds architectural and training complexity, may require hyperparameter tuning, and has deployment and evaluation-scope boundaries.
- FoE introduces additional model architecture and training complexity despite reducing communication overhead.
- FoE models reach similar performance to traditional MoE models with identical training hyperparameters, but isolated expert and KV-head groups may require careful tuning.
- FoE benefits distributed deployments, while single-GPU use adds cross-group all-reduce overhead without eliminating all-to-all communication.
- On heterogeneous multi-node networks, expert-group placement affects intra-node traffic, so skewed topologies may require placement-aware tuning.
- The evaluation covers only 1B and 7B models on one 8×H100 node and a two-node InfiniBand cluster.
6 Conclusion
FoE partitions transformer layers into independent expert groups, replacing global all-to-all communication with smaller intra-group exchanges and cross-group reduction. It improves inference efficiency while matching baseline MoE quality on the reported evaluations.
- FoE partitions each transformer layer into independent expert groups.
- FoE replaces global all-to-all communication with smaller intra-group all-to-alls and a cheaper cross-group all-reduce.
- 5.2× end-to-end forward-pass speedup, 3.62× lower TTFT, and 1.95× lower TBT are reported for FoE.
- FoE matches baseline MoE performance on cross-entropy loss and zero-shot reasoning.
- FoE’s multi-node benefit compounds because intra-group all-to-alls remain intra-node while reduce communication uses inter-node networking.
A.1 Model Specification
The reported FoE and size-matched MoE runs use OLMoE-style transformers pretrained from scratch with shared training settings, while FoE uses eight KV heads and eight expert groups.
- The experiments report small ∼1B and full ∼7B pretraining runs for FoE and size-matched MoE models.
- Both models are OLMoE-style transformer MoEs pretrained from scratch on FineWeb-Edu with the OLMoE-1B-7B-0924 tokenizer.
- FoE uses n_kv = 8 KV heads, matching H = 8 expert groups.
- Table 8 marks architecture rows that differ between FoE and its MoE baseline while sharing the remaining settings.
A.2 Detailed Communication Traces
Single-layer communication traces compare standard MoE and FoE on an 8-GPU node, showing substantially lower synchronization overhead for FoE. The measured communication-time reduction is 12.3× on the 8×H100 setup.
- 12.3× lower communication time was measured for FoE on an 8×H100 setup.The measurement concerns stream 31 in an 8,192-token batch run.
- The trace profiles one layer of an 8,192-token batch run on an 8-GPU single node.
- Figure 5 shows the standard MoE baseline communication trace, while Figure 6 shows the proposed FoE trace.
- Stream 7 represents compute and stream 31 represents communication in both flamegraphs.
- The FoE trace has reduced communication overhead compared with the standard MoE trace.
A.3 End-to-End Forward Pass Latency
Table 9 compares end-to-end forward-pass latency as synthetic batch size increases. Standard MoE latency escalates beyond batch sizes of 2048 due to all-to-all communication, whereas FoE remains near-constant up to moderately large sizes.
- Beyond synthetic batch sizes of 2048, standard MoE forward-pass latency escalates substantially because global dispatch is bottlenecked by all-to-all communication.The synchronization required for standard MoE parallelism spans the network topology.
- FoE keeps forward-pass latency near-constant up to moderately large synthetic batch sizes by confining routing and load balancing within localized expert groups.
- Table 9 reports end-to-end forward-pass latency in seconds across synthetic batch sizes.