INC-MoE: Overlapping In-Network Collectives and Expert Computation for Expert-Parallel Mixture-of-Experts Inference

1 Introduction

• Dispatch and combine are critical-path steps in every MoE layer, and their share of layer time grows with expert-parallelism degree.In inter-node deployments, lower inter-node than intra-node bandwidth can make MoE layers communication-bound.Source: §1 · pp. 1–2
• Overlapping in-network AllGather and ReduceScatter cuts bandwidth-bound communication time by 33.3% for four ranks and 42.9% for eight relative to ring collectives.The collectives use opposite link directions, allowing concurrent transfers to use both directions at once.Source: §1 · pp. 2–3
• INC-MoE pipelines token bands through AllGather, expert FFN, and ReduceScatter in one persistent kernel, running the three stages concurrently at steady state.Role-specialized thread blocks gather the next band, compute the current band, and reduce the previous band; tail assistance shortens the final RS-only period.Source: §1 · p. 3
• Up to 33.1% lower TTFT and 49.5% higher prefill throughput than vLLM’s default path on OLMoE prefill with four-way EP in an emulated four-node deployment.INC-MoE is a drop-in vLLM MoE-layer replacement requiring no model-definition changes.Source: §1 · p. 3, Figure 1 · p. 3
• In a standalone concurrent-collective benchmark, in-network collectives achieve 25.6–31.8% higher equivalent bandwidth than concurrently executed ring collectives.The layer’s end-host kernel uses a message interface and is unchanged when the switch is emulated on a dedicated accelerator.Source: §1 · pp. 2, 4

2.1 MoE Layers and Expert Parallelism

• MoE replaces a transformer block’s dense FFN with expert FFNs and a router; under expert parallelism, each rank hosts some experts and owns a batch of tokens.Different ranks may own different numbers of tokens.Source: §2.1 · p. 4

2.2 Dispatch and Combine as AllGather and ReduceScatter

• AllGather makes all ranks’ tokens visible to every rank for local expert computation, then ReduceScatter sums partial outputs and returns each rank its own token rows.The AllGather sends tokens regardless of routing, but under the stated E=64, k=8, N=4 example it sends about 9% more data than a fully deduplicated all-to-all.Source: §2.2 · p. 4

2.3 In-network Collectives

• In-network computing uses switch multicast and reduction to implement collectives, reducing end-host-link data crossings and replacing dependent ring or tree steps with a switch pass.Prior work mostly applied INC to all-reduce; the passage identifies expert-parallel prefill across nodes within a serving engine as the setting INC-MoE targets.Source: §2.3 · pp. 4–5

2.4 Device-initiated Communication

• Device-initiated communication lets a running kernel post transfers, signal peers, and wait for completions without returning control to the host, enabling fine-grained communication-computation pipelining in one persistent kernel.Host-launched per-band collectives would require repeated launches and cross-stream synchronization.Source: §2.4 · p. 5

3 Design of INC-MoE

• INC-MoE targets serialized AG, FFN, and RS stages whose communication remains exposed on the critical path in the default EP path.Ring AG and RS each saturate both link directions, so overlapping them does not shorten their combined time.Source: §3 · p. 5

3.1 In-Network AllGather and ReduceScatter with Message Semantics

• Message-semantic AG uses switch multicast, while RS uses switch-side reduction followed by a root-local addition; end hosts exchange messages with the switch, not peer memory.A client library issues and completes collectives from device code, while a switch-side service replicates or reduces payloads.Source: §3.1 · pp. 6–7

3.2 Communication Cost

• 33.3% for N=4 and 42.9% for N=8: overlapping in-network AG and RS reduces bandwidth-bound communication time versus concurrent ring collectives.The bound assumes full-duplex links, line-rate switch forwarding and reduction, and messages large enough for bandwidth to dominate; it requires AG and RS to be in flight together.Source: §3.2 · pp. 8–9

3.3 Overlap Kernel: Band Pipeline with Role-Specialized Thread Blocks

• At steady state, role-specialized blocks gather band j+1, compute band j, and reduce band j−1, overlapping both collectives with each other and with the FFN.Tail assistance lets gather blocks help with local reductions during pipeline drain and is most effective at 64K or more tokens per accelerator.Source: §3.3 · p. 10

3.4 Pipeline Configuration

• INC-MoE selects band counts, block allocations, and tail assistance by token-count regime because the optimal pipeline configuration shifts with workload size.More bands shorten fill and drain but add per-band collective, synchronization, and smaller-GEMM costs.Source: §3.4 · p. 11

3.5 Fused Expert Computation

• Each band’s local experts run two GEMMs; epilogue fusion applies SwiGLU and router weighting while accumulating outputs directly into the RS send buffer.The implementation prioritizes communication efficiency over individually optimized GEMM throughput, and exposes the final output in place as a tensor view.Source: §3.5 · p. 12

End host and switch

• The end-host side comprises a fused kernel, INC client library, and Python-exposed launcher, while a switch-side service implements the message semantics.The prototype runs the switch service on a dedicated accelerator emulating an INC-capable switch.Source: §4 · p. 12

Serving-engine integration

• INC-MoE replaces vLLM’s MoE layer between routing and expert computation while leaving routing, including top-k weight normalization, to the engine.Per-rank token counts come from engine metadata; model dimensions, expert count, k, data type, and EP degree are runtime parameters.Source: §4 · p. 12
• Any MoE model with SwiGLU FFNs in fp16 or bf16 can use the layer without changes to the kernel or model definition.Source: §4 · p. 13

5.1 Setup

• The testbed emulates four nodes with four EP-rank accelerators and a fifth accelerator for the INC switch; the shared 400 Gbps link provides each rank an effective 100 Gbps.The OLMoE-1B-7B prefill evaluation uses DP=4, EP=4, and varies aggregate prompt tokens per accelerator from 24K to 126K.Source: §5.1 · p. 13, Figure 5 · p. 13

5.2 In-Network Collective Bandwidth

• With four ranks at 100 Gbps per rank, concurrent in-network AG and RS reach 163–172 Gbps equivalent bandwidth, 25.6–31.8% above the ring baseline.The emulated switch’s fixed message costs and sequential rooted reductions affect the measured bandwidth; the authors expect these effects to be largely reduced on switch hardware.Source: §5.2 · pp. 14–15, Figure 6 · p. 14

5.3 End-to-End Performance

• Across prompt sizes, INC-MoE reduces TTFT by 20.2–33.1%, corresponding to a 25.3–49.5% prefill-throughput increase over the baseline in the same runs.Figure 7 reports prefill throughput per accelerator.Source: §5.3 · p. 15, Figure 7 · p. 15
• INC-MoE throughput rises from 28.2K to 34.3K tokens/s as prompt size grows, while baseline throughput stays about 22.5K–23.8K tokens/s.The paper attributes the growing advantage to larger messages being processed more efficiently by the emulator and fill/drain phases taking a smaller share as bands increase from 3 to 10.Source: §5.3 · p. 15, Figure 7 · p. 15

MoE systems and EP communication

• DeepEP, pplx-kernels, and MegaScale-Infer optimize end-host communication, whereas INC-MoE offloads dispatch and combine to the switch using AG/RS.Source: §6 · pp. 15–16

Overlapping communication and computation

• Prior overlap approaches include tile-level fusion, persistent kernels, and micro-batch overlap; INC-MoE pipelines token slices within one MoE-layer kernel using role-specialized blocks.Its switch-executed collectives overlap with each other and with computation; micro-batch overlap is described as orthogonal.Source: §6 · p. 16

In-network computing

• Unlike work proposing new scale-up hardware, INC-MoE targets scale-out networking with static AG/RS collectives and focuses on message semantics, concurrent use, and the end-host pipeline.Source: §6 · p. 16

• INC-MoE executes dispatch as in-network AllGather and combine as in-network ReduceScatter, overlapping both with each other and expert computation in a single band-pipelined kernel.Source: §7 · p. 16
• On OLMoE prefill with four-way EP in an emulated four-node configuration, INC-MoE reduces TTFT by up to 33% and increases throughput by up to 50% versus vLLM’s default collective path.Source: §7 · p. 16
• Because the two in-network collectives load opposite link directions, overlapping them reduces bandwidth-bound communication time by (N−2)/(2(N−1)) relative to ring collectives.Source: §7 · p. 16
• The implementation uses message semantics, a device-side communication library, and an accelerator-based emulator of an INC-capable switch, and integrates into vLLM as a drop-in replacement.Source: §7 · p. 16