Source-linked AI summary
Democratizing MoE inference on commodity GPUs with CoMoE
Ruwen Fan, Yuezhi Zu, Junru Li, Qingda Hu, Xinjun, Yang, Jiwu Shu, Youyou Lu
TL;DR
MoE inference depends on costly inter-GPU communication, while lower-cost consumer GPUs lack the high-bandwidth P2P links used by datacenter systems. CoMoE makes the host an active routing hub for token dispatch and combine; on RTX 5090 GPUs, it improves throughput over SGLang and approaches A800 performance at a fraction of the hardware investment.
Problem
MoE inference relies on costly high-bandwidth P2P links, while consumer GPUs use host-mediated PCIe without P2P and existing collectives add redundant transfers and synchronization stalls.
Source: §1 · pp. 1–2Method
Results
Up to 1.46× throughput over SGLang, CoMoE reaches 85–90% of the strongest A800 configuration’s throughput at no more than 23.4% of equivalent GPU investment.
Source: §1 · p. 2, §6 · p. 13Takeaways & Limitations
CoMoE brings consumer-GPU MoE inference closer to datacenter throughput at substantially lower hardware cost.
Source: §1 · p. 2, §6 · p. 13
Abstract
from arXiv · showhide
Deploying Mixture-of-Experts (MoE) models relies heavily on Expert Parallelism, which generates intense inter-GPU communication. Consequently, state-of-the-art inference systems require high-bandwidth, P2P interconnects (e.g., NVLink) in datacenter GPUs to handle massive token routing, making deployment prohibitively expensive. Consumer GPUs offer comparable compute power at significantly lower cost, promising to democratize MoE inference for individuals and enable privacy-preserving local deployments. However, their bandwidth-limited (only weak PCIe bus bandwidth) and host-mediated interconnects (no P2P support) introduce severe communication bottlenecks. We present CoMoE, a communication-efficient MoE inference system that resolves this mismatch through novel host-centric routing. Our key insight is that the unique communication topology provides the opportunity to elevate the host to an active routing hub, which can fundamentally reduce communication volume and eliminate global synchronization-induced stalls. Specifically, for token dispatch, we introduce host-backed token multicast to write shared tokens to the host exactly once, eliminating outbound transmission redundancy. For token combine, we propose a fine-grained, token-level aggregation mechanism using host staging buffers, which replaces rigid global synchronization and mitigates straggler effects. Evaluation on RTX 5090 GPUs shows that CoMoE improves inference throughput by up to 1.46x, approaching the performance of NVLink-capable A800 GPUs at only 23.4% of the hardware cost.
1 Introduction
MoE inference requires substantial inter-GPU communication, making datacenter systems costly, while consumer GPUs offer comparable compute at lower cost but suffer communication bottlenecks. CoMoE makes the host an active routing hub and reports higher throughput on RTX 5090 GPUs at a fraction of A800 hardware cost.
1 Introduction
- EP requires costly all-to-all token dispatch and output aggregation, and current inference systems rely on high-bandwidth P2P links such as NVLink.On consumer GPUs, communication overhead reaches 81.6% of execution time, while host-mediated PCIe and collective primitives add redundant transfers and global synchronization stalls.Source: §1 · pp. 1–2
- Consumer GPUs offer compute power and memory bandwidth on par with datacenter GPUs at substantially lower cost, supporting local, privacy-preserving deployments.The introduction describes the RTX 5090 as costing one-sixth of its datacenter counterpart.Source: §1 · pp. 1–2
- CoMoE elevates host memory from a passive bounce buffer to a centralized routing hub for dispatch and combine.Dispatch writes each shared token to the host once; fine-grained host staging supports token-level aggregation in a completion-driven pipeline that overlaps transfers with computation.Source: §1 · p. 2
- Up to 1.46× throughput over SGLang, CoMoE reaches 85–90% of the strongest A800 configuration’s throughput at no more than 23.4% of equivalent GPU investment.The evaluation uses representative MoE models and ShareGPT; CoMoE also brings TTFT close to an A800 system with NVLink.Source: §1 · p. 2
2 Background and Motivation
Expert Parallelism routes tokens to experts distributed across GPUs and then gathers their outputs, making MoE inference dependent on inter-GPU communication. Consumer GPUs offer comparable compute at lower cost, but PCIe and host mediation create a communication bottleneck that conventional collectives amplify.
2.1 MoE Inference with Expert Parallelism
- EP distributes experts across GPUs, requiring input-dependent token dispatch to target GPUs and a combine phase to gather expert outputs.Routing is irregular because tokens may target multiple experts, tying MoE inference efficiency to the communication substrate.Source: §2.1 · pp. 2–3
2.2 Consumer-Grade GPUs
- RTX 5090 offers comparable memory bandwidth and same-order tensor throughput to A800 at roughly 17% of its price, but consumer GPUs lack P2P and route transfers through host memory.This mismatch makes fine-grained MoE traffic contend for a narrower PCIe substrate and can shift the bottleneck from computation to interconnect efficiency.Source: §2.2 · p. 3
2.3 Motivation
- Up to 1.97× higher latency on consumer GPUs, with communication consuming 81.6% of RTX 5090 prefill time versus 44.8% on datacenter GPUs.The passage attributes the gap almost entirely to communication while computation remains comparable across platforms.Source: §2.3 · pp. 3–4, Figure 3 · p. 4, Figure 4 · p. 4
- Tokens are routed to an average of 2.98–5.37 distinct GPUs, and standard all-to-all dispatch sends duplicate copies of shared tokens.For Qwen3-30B-A3B, a token is transmitted to an average of 5.37 GPUs, increasing egress contention on PCIe.Source: §2.3 · p. 4
- At prefill length 2,048, coarse-grained combine synchronization can stall for 301µs, or up to 11.9% of the combine phase.Transfers wait for peers to enter the collective and reduction waits for the slowest peer, preventing effective overlap with computation.Source: §2.3 · p. 4
- The communication mismatch motivates a design that accounts for host-mediated transfers, centralized PCIe topology, and scarce inter-GPU bandwidth.Conventional collectives amplify the bandwidth limitation through dispatch redundancy and rigid combine synchronization.Source: §2.3 · p. 4
3 Design and Implementation
CoMoE turns the host-mediated communication topology of consumer GPU servers into an active routing hub. It redesigns dispatch and combine around host memory to reduce redundant data movement and synchronization stalls.
3 Design and Implementation
- CoMoE makes host-mediated communication an active design principle, using the host as a routing hub to reduce redundant transfers and synchronization stalls.The design pairs host-backed token multicast for dispatch with fine-grained host staging for combine.Source: §3 · pp. 4–5
3.1 Zero-Redundancy Dispatch via Host-Backed Token Multicast
- Each remote token is written to host memory once, then destination GPUs pull it in parallel, reducing source-side PCIe traffic from repeated transmissions.Under the stated expert-routing assumptions, conventional All-to-All sends a token to roughly 3–5 of 8 GPUs on average.Source: §3.1 · pp. 5–6
3.2 Decomposed Fine-grained Combine via Host Staging
- Combine decomposes All-to-All into one-sided host-staged deposits and drains, coordinating per token rather than waiting for global collective completion.Receiver blocks update per-token completion state and can launch accumulation when a token’s contributions arrive, overlapping communication tails with combine work.Source: §3.2 · pp. 6–7
- NUMA-aware placement has senders write into staging buffers in the receiver’s local memory domain, enabling efficient remote writes and local reads.This avoids remote-read requests and reduces PCIe transaction footprint.Source: §3.2 · p. 7
- Thread Tile Pipeline partitions blocks to transfer tokens in tiles, while batched commit shares commit state to amortize synchronization across token deposits or drains.Tiling addresses underutilized PCIe bandwidth for varying per-token state sizes; batched commit preserves token-level readiness while reducing host-visible synchronization events.Source: §3.2 · p. 7
3.3 Implementation
- CoMoE is an SGLang backend selected through configuration, with model code and scheduler unchanged and GPU kernels accessing pinned, mapped host buffers directly.For Qwen3-30B-A3B, staging uses 512 MiB per GPU; the RTX 5090 transfer kernel occupies 24 of 170 SMs (14.1%).Source: §3.3 · pp. 7–8
4 Evaluation
The evaluation tests CoMoE on RTX 5090 and A800 systems across MoE models and serving workloads, finding improved throughput, latency, and communication efficiency on consumer GPUs. It also examines the dispatch and combine techniques and estimates the hardware cost of matching A800 serving throughput.
4.1 Experimental Setup
- The evaluation used four BF16 MoE configurations, ShareGPT prompts, and eight-GPU RTX 5090 and A800 systems, with TTFT, TPOT, throughput, and agent-task latency metrics.The A800 system used NVLink; the setup also included DeepEP as a specialized A800 comparison.Source: §4.1 · pp. 8–9, Table 2 · p. 8
4.2 Technique Analysis
- 55.7% average prefill communication-latency reduction; dispatch latency fell 1.68× for Qwen3-30B-A3B and 1.40×–1.96× for three other models.CoMoE reduced the baseline communication-to-compute ratio from above 4.6× to about 2.25×; Figure 11 compares dispatch latency across models.Source: §4.2 · p. 9, Figure 10 · p. 9, Figure 11 · p. 9
- 13.2% and 11.2% throughput gains over CoMoE-Sync on Qwen3-30B-A3B and GLM-4.7-Flash show the benefit of avoiding a global combine barrier.The authors attribute the gains to masking straggler effects with host-mediated one-sided transfers; Figure 12 compares synchronized and unsynchronized combine.Source: Figure 12 · p. 9, §4.2 · p. 10
- 2.46× geometric-mean combine speedup over NCCL All-to-All and 1.59× over CoMoE-raw came from the full combine design.NUMA-aware placement, thread-tile pipelining, and batched commit each reduced latency in the tested token-count ranges; Figure 13 reports ablation latencies.Source: Figure 13 · p. 10, §4.2 · p. 10
4.3 Overall performance
- 18.8% lower TTFT and 26.9% lower TPOT on average across evaluated models and request rates, with smaller relative gains at the highest request rate.CoMoE's TTFT was close to the A800 baseline on DeepSeek-V2-Lite and GLM-4.7-Flash.Source: §4.3 · p. 10, Figure 14 · p. 11, Figure 15 · p. 11
- 1.46× maximum throughput gain over the RTX 5090 baseline; CoMoE reached 85.3%–90.0% of A800 DeepEP throughput across models.On GLM-4.7-Flash, CoMoE reached 98.8% of the A800 baseline throughput.Source: §4.3 · pp. 10–11, Figure 16 · p. 11
- 18.5%–23.4% of A800 DeepEP hardware cost was sufficient for CoMoE to match its serving throughput.The estimate accounts for the 1.11×–1.17× RTX 5090 node over-provisioning needed to match an A800 node.Source: §4.3 · p. 11, Table 3 · p. 12
- 13.1% lower median and 9.6% lower P90 end-to-end latency on BrowseComp-VL tasks with DeepResearch and Qwen3-30B-A3B.External tool calls reduced the gains, but latency improvements held across tasks of varying complexity.Source: Figure 17 · p. 12, §4.3 · p. 12
5 Related Works
Prior work improves MoE communication through scheduling, hierarchical All-to-All, token reordering, expert movement, and specialized communication libraries. CoMoE addresses the comparatively underexplored consumer-GPU inference setting, complementing low-cost approaches focused on offloading or compression.
5.1 MoE Communication Optimization
- FastMoE [11] and FasterMoE target training communication, while DeepSpeed-MoE [27] and Tutel [15] use hierarchical All-to-All to reduce inter-node volume.Aurora reorders token transmission, Janus moves experts between GPUs, and DeepEP and PPLX integrate communication into GPU kernels.Source: §5.1 · p. 12
5.2 Low-cost MoE Inference
- MoE-Infinity, HOBBIT [31], and ExpertFlow use expert offloading and prefetching; FloE [38] uses hybrid expert-matrix compression; KTransformers and Fiddler [16] leverage CPU/GPU heterogeneous resources.CoMoE is orthogonal to these approaches, targeting efficient communication in a consumer GPU-only setting.Source: §5.2 · p. 12
6 Conclusion
CoMoE targets consumer GPUs whose interconnects limit MoE inference, using host-backed token multicast and fine-grained host-staged combine to reduce dispatch redundancy and synchronization bottlenecks. It achieves up to 1.46× throughput gains over baselines and 85–90% of A800 DeepEP throughput at 23.4% of the hardware cost.
6 Conclusion
- CoMoE achieves up to 1.46× throughput gains over baselines and 85–90% of A800 DeepEP throughput at 23.4% of the hardware cost.Its host-backed multicast removes dispatch redundancy, while fine-grained host-staged combine removes synchronization bottlenecks.Source: §6 · pp. 12–13