PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Effects of GPU Server Topology, Parallelism, and Congestion Control on MoE Inference: A Controlled Simulation Study

Kaikai Yuan, Rui Xi, Yu Liu

arXiv:2609.37828v1cs.DC

TL;DR

MoE inference combines TP collectives and EP dispatch/combine, but the joint effects of parallelism, physical topology, and congestion control remain to be quantified systematically. The paper uses ASTRA-sim with NS-3 to compare controlled configurations, finding that exposed communication and completion time vary substantially with these choices. The conclusions apply to the evaluated synthetic traces and simulator semantics, not directly to production request latency.

  • Problem

    It remains to quantify whether locally optimal choices remain optimal as topology, rank mapping, NIC configuration, and congestion control vary, because these interactions shape completion time.

    Source: §1.2 · p. 2

  • Method

    The study uses ASTRA-sim, Chakra ET, and NS-3 to compare server topologies, TP/EP partitions, collective algorithms, and network modes under fixed GPU count and workload.

    Source: §1.4 · p. 3, §1.5 · p. 3

  • Results

    Exposed communication contributes 89.9%–95.8% of mean completion time, while TP16EP2 takes 3.68–4.35× the mean completion time of TP2EP16.

    Source: §8 · p. 33

  • Takeaways & Limitations

    Within the evaluated workload and simulator semantics, topology, parallelism, collective implementation, and congestion control jointly determine exposed communication and completion time.

    Source: §8 · p. 33

    Simulation completion time is not online end-to-end latency, and the synthetic execution traces do not fully encode production serving behaviors.

    Source: §5.5 · p. 32

Abstract

from arXiv · show

Mixture-of-experts (MoE) models expand capacity via sparse activation, but inference across GPUs introduces tensor-parallel (TP) collectives and expert-parallel (EP) dispatch and combine operations. Completion time depends not just on communication volume but on how logical groups map onto intra-server interconnects, GPU--NIC connections, and the inter-node network. Using ASTRA-sim with the NS-3 discrete-event backend, we build a controlled matrix of 32 GPU ranks with data and pipeline parallelism fixed at one. Workloads are fixed-length 4096-token prefill-like synthetic Chakra traces from four MoE configurations. Experiments cover six server topologies, four TP/EP partitions, two TP collective algorithms, and four network/congestion-control modes, yielding 768 deterministic simulations. In the 144-configuration feedback-enabled subset per model, exposed communication accounts for 89.9%--95.8% of mean completion time. TP16EP2 requires 3.68--4.35x the mean completion time of TP2EP16. With fixed rank mapping, ASTRA-sim Double Binary Tree (DBT) incurs 28.3%--83.2% more time than Ring. InfiniBand-like High Precision Congestion Control (HPCC) is ~0.9% lower than HPCC over RDMA over Converged Ethernet (RoCE), whereas RoCE with Data Center Quantized Congestion Notification (DCQCN) is 23.8%--35.7% slower than RoCE HPCC. Topology effects are conditional: Topology~6 leads at low TP degrees but loses its advantage at high TP degrees, and additional GPUs or NICs help only when rank mapping balances traffic across injection paths. Under uniform 32-way sharding, the largest checkpoint-weight shard is ~48.75 GB per rank, so all configurations meet a 64 GB per-accelerator weight-residency criterion. Within the evaluated workload and simulator semantics, server topology, parallelism, collective implementation, and congestion control jointly determine exposed communication and completion time.

1. Introduction

MoE inference makes communication central: TP collectives and EP dispatch/combine must be considered alongside the physical paths they traverse. The introduction motivates a controlled simulation study of how these factors interact, rather than assuming hardware metrics alone predict completion time.

1.1 System and Communication Challenges in MoE Inference

  • MoE routing adds expert dispatch and aggregation to TP collectives, shifting part of the system bottleneck to inter-device communication and possible stalls under expert-load imbalance.Prior work addresses communication behavior, placement, and architecture, but communication also constrains parallelization, expert placement, and system architecture.Source: §1.1 · p. 2

1.2 Related Work and Open Questions

  • Existing work leaves open whether locally optimal choices remain optimal across server topologies, rank mappings, NIC configurations, and congestion-control schemes.Serving, model/parallelism, and networking studies address different levels, while isolated GPU, NIC, or bandwidth increases may not predict completion time.Source: §1.2 · p. 2

1.3 Coupling among Topology, Parallelism, and Network Mechanisms

  • Completion time depends on which physical paths logical communication edges traverse and whether synchronized flows compete at shared egress points.More GPUs or NICs do not have a simple monotonic relationship with performance; rank-to-NIC mapping can leave ports idle or create hot spots.Source: §1.3 · pp. 2–3

1.4 Scope and Simulation Methodology

  • The study uses ASTRA-sim, Chakra ET, and NS-3 to compare fixed model workloads across configurations, targeting relative trends rather than online-service latency.The method models links, switch queues, RDMA flows, and congestion feedback; completion time excludes request queuing, batching, KV-cache lifecycle, and runtime overhead.Source: §1.4 · p. 3

1.5 Research Questions and Contributions

  • A unified simulation matrix varies topology, TP/EP partitioning, collective algorithms, and network/congestion control while holding GPU count and model workload constant.The analysis examines main effects and interactions, relates trends to communication proxies, and uses queue and flow statistics to explain representative differences.Source: §1.5 · p. 3

2. Background

The background explains how MoE execution generates TP and EP communication, how logical groups map to physical interconnects, and how the simulation stack represents execution and network behavior.

2. Background

  • The study's background frames execution, communication mapping, and simulation as foundations for analyzing MoE inference.The section establishes how MoE inference generates TP and EP communication and how hierarchical physical interconnects shape modeled behavior.Source: §2 · p. 4

2.1 Execution and Communication in MoE Inference

  • MoE layers combine TP collectives with EP token dispatch and combine; their traffic depends on token count, hidden width, activated experts, partitioning, and routing skew.The critical path can include local computation, TP and EP communication, and expert computation; production serving dynamics alter message sizes and overlap opportunities.Source: §2.1 · p. 4

2.2 TP/EP and Logical Communication Groups

  • TP shards operator tensors and requires collectives, while EP distributes experts and requires token dispatch and result combination; neither group alone determines physical transfer paths.The experiments fix DP=1 and PP=1, and rank placement can route communication across PCIe, NUMA, or inter-server links.Source: §2.2 · pp. 4–5

2.3 ASTRA-sim–Chakra–NS-3 Simulation Stack

  • ASTRA-sim schedules Chakra ET operation graphs by dependencies, decomposes collectives into transfers, and submits them to a network backend that models routes and completion times.NS-3 models links, queues, RDMA, and congestion feedback; events absent from the trace, including dynamic batching and compute–communication overlap, are not inferred automatically.Source: §2.3 · p. 5

3. Experimental Methodology

The methodology uses a controlled 32-rank simulation matrix to examine how model architecture, TP/EP partitioning, topology, collective algorithms, and congestion-control modes affect communication and completion time, with conclusions bounded by fixed workload, simulator assumptions, and rank mapping. A 4.9% difference from an independent collective microbenchmark supports ASTRA-sim’s use for communication-performance comparisons when physical GPU clusters are unavailable.

3.1 Model Selection

  • Four deliberately diverse MoE configurations separate TP message scale and EP routing intensity rather than treating total parameter count as a proxy for communication.Workload dimension and layer count shape TP collective size and frequency; activated-expert count and top-k shape EP dispatch/combine volume. Kimi K3 uses its reported latent MoE dimension, h = 3584, as the communication input rather than its attention hidden dimension of 7168.Source: §3.1 · p. 6

3.2 Model-Residency Capacity Based on Actual Weight Files

  • All four models fit the 64 GB-per-accelerator weight-residency criterion under uniform sharding across 32 ranks, but this accounts only for checkpoint weights.Kimi K3 requires approximately 48.75 GB per rank, the largest of the four; the criterion does not establish capacity for KV cache, activations, buffers, or runtime overheads.Source: §3.2 · p. 7

3.3 Parallelization Strategies and Rank Mapping

  • The design fixes 32 ranks with DP=1 and PP=1, varies four TP/EP partitions, and holds a contiguous logical rank mapping constant across experiments.Adjacent partitions double TP and halve EP; TP1EP32 and TP32EP1 are omitted because each removes one of the two communication mechanisms being jointly evaluated.Source: §3.3 · p. 8

3.4 Physical Topology Design

  • Six physical topologies vary node density, GPU/NIC ratio, and intra-node interconnects while preserving 32 GPUs; fixed contiguous placement makes comparisons controlled but mapping-dependent.The paired designs distinguish GPU/NIC and server-count effects, dual-socket UPI effects, intra-node consolidation, and an intra-node-only baseline; cost, energy, rack space, and reliability are not modeled.Source: §3.4 · pp. 8–9

3.5 Network and Congestion-Control Settings

  • Network experiments compare a no-rate-feedback control with DCQCN and HPCC feedback, while the simulator’s IB HPCC is only an IB-like mode, not native InfiniBand.DCQCN uses ECN/CNP feedback and HPCC uses in-band telemetry; NS-3 does not fully implement native InfiniBand flow control, service levels, arbitration, or vendor NIC firmware.Source: §3.5 · pp. 9–10

3.6 Collective-Algorithm Combinations

  • The main matrix fixes Direct All-to-All for EP and varies TP between Ring and Double Binary Tree, comparing these specific ASTRA-sim implementations.An exploratory 20-run Kimi K3 screen found mean completion time of 2.388 s for ring_ring and 1.036 s for ring_direct; this was excluded from confirmatory matrix means. ASTRA-sim DBT transfers full data_size on each tree edge, unlike Ring’s message sharding.Source: §3.6 · p. 11

3.7 Workload Construction

  • Each model generates a fixed 4096-token, prefill-like serial workload with attention, TP collectives, EP dispatch/combine, and compute events under uniform expert routing.The traces abstract dynamic batching, decoding, routing skew, capacity drops, and compute–communication overlap; compute time uses a roofline model, and the reported capacity criterion covers weights only.Source: §3.7 · pp. 11–12

3.8–3.10 Simulator, Data Extraction, and Validation

  • The 768-unit matrix uses fixed simulator baselines; completion and component times come from the slowest rank, and communication credibility is assessed against an independent collective microbenchmark.The comparison uses Twall ≈ TGPU + Tcomm and descriptive means over stated subsets; relative validation error is |Tsim − Tmeasured|/Tmeasured × 100%. Local flows bypass congestion feedback while retaining modeled intra-node traffic.Source: §3.8 · pp. 12–13, §3.9 · p. 13, §3.10 · p. 13

3.10 Validation of Communication-Performance Credibility

  • 4.9% was the difference between simulated and measured communication time, supporting ASTRA-sim’s use for evaluating cluster communication performance when physical GPU clusters are unavailable.The validation supports comparisons of topology, parallelism, collective algorithms, and congestion-control schemes.Source: Table 3 · p. 14, §3.10 · p. 14

4. Experimental Results

Across Flash’s 192 configurations, communication dominates completion time, while topology, TP/EP partitioning, collective choice, and congestion control interact to shape performance.

4.1.2 Topology Comparison

  • Topology 6 shifts from fastest at TP2EP16 (0.1056 s) to 2.32× slower than Topology 1 at TP16EP2 (0.6721 versus 0.2896 s).The full matrix is shown in Figure 4-1; configuration variation is substantially greater in communication than GPU time.Source: §4.1.1 · p. 14, Figure 4 · p. 15, §4.1.2 · p. 15
  • Topology 6’s mean completion time is 28.1% below the slowest topology, but only 11.0% faster than Topology 1.Topology 1’s dedicated NIC per GPU avoids shared injection-point contention despite external-network traffic.Source: §4.1.2 · p. 15
  • Topology 2 is approximately 17.0% faster than Topology 3, whose dual-CPU/UPI path adds synchronous-path cost, though the full difference cannot be attributed to UPI.Topology 4 is approximately 19.8% slower than Topology 2; more GPUs and NICs help only when rank-to-NIC mapping uses them evenly.Source: §4.1.2 · p. 15

4.1.3 TP/EP Strategy Comparison

  • TP16EP2 takes 6.36× TP2EP16 completion time with Ring on Topology 6, while mean completion time rises 4.51× across TP2-to-TP16 strategies.Communication fraction rises from 82.1% at TP2 to 97.6% at TP16, overwhelming reduced GPU time.Source: §4.1.3 · p. 15

4.1.4 Network and Congestion-Control Comparison

  • RoCE DCQCN is 35.7% (0.192850 s) slower than RoCE HPCC after excluding Topology 6, entirely due to communication.IB HPCC and RoCE HPCC differ by approximately 0.1% in mean completion time; this does not establish native-protocol equivalence.Source: §4.1.4 · p. 16

4.1.5 Collective-Algorithm Comparison

  • Across the full matrix, dbt_direct is 86.0% (0.331142 s) slower than ring_direct, entirely due to communication.ASTRA-sim DBT sends full data_size over each tree edge while Ring shards by rank count, concentrating tree traffic on fewer egresses.Source: §4.1.5 · p. 16

4.1.6 Minimum-Completion-Time Combination

  • The minimum is Topology 6 + TP2EP16 + ring_direct, while DCQCN raises Topology 3 completion time by 98% versus IB CC0 at TP16EP2.Across interactions, communication fraction ranges from 65.7% for the fastest combination to 99.0% for the slowest.Source: §4.1.6 · p. 16, §4.1.7 · pp. 16–17

4.2 Kimi K3

Kimi K3 and GLM-5.3 remain communication dominated across the evaluated configurations, but their topology, parallelism, and collective rankings differ in degree. High TP, topology-specific congestion, and collective mapping produce substantial completion-time changes in both models.

4.2.1 Overall Statistics

  • Kimi K3 has an 89.4% aggregate communication fraction, the lowest of the four models, yet its maximum completion time is 15.6× its minimum.Its mean GPU time is 0.137182 s, the highest among the models.Source: §4.2.1 · p. 17

4.2.2 Topology Comparison; 4.3.2 Topology Comparison

  • Topology 6 shifts from fastest at TP2EP16 to 2.63× slower than Topology 1 at TP16EP2 for Kimi K3, and 2.50× slower for GLM-5.3.For Kimi, the TP16EP2 times are 2.3814 s on Topology 6 and 0.9072 s on Topology 1; for GLM-5.3 they are 2.1282 s and 0.8511 s.Source: §4.2.2 · p. 18, §4.3.2 · p. 20

4.2.3 TP/EP Strategy Comparison; 4.3.3 TP/EP Strategy Comparison

  • Kimi K3 completion time on Topology 6 with IB CC0 and Ring rises 10.4× from TP2EP16 to TP16EP2; GLM-5.3 rises 6.27× under the same conditions.For GLM-5.3, TP16 communication time is 5.00× TP2, while GPU-time savings are much smaller than the concurrent communication increase.Source: §4.2.3 · p. 18, §4.3.3 · p. 20

4.2.4 Network and Congestion-Control Comparison; 4.3.4 Network and Congestion-Control Comparison

  • RoCE DCQCN is 32.3% (0.416637 s) slower than RoCE HPCC for Kimi K3 and 23.8% (0.381462 s) slower for GLM-5.3, entirely due to communication.The protocol comparisons exclude Topology 6, which has no external network.Source: §4.2.4 · p. 18, §4.3.4 · p. 21

4.2.5 Collective-Algorithm Comparison; 4.3.5 Collective-Algorithm Comparison

  • Kimi’s dbt_direct is 26.2% (0.330850 s) slower than ring_direct, while GLM-5.3’s is 74.1% (0.863681 s) slower.Both reported penalties are entirely due to communication; Kimi’s DBT/Ring ratio rises to 1.32 at TP8, then dips to 1.29 at TP16.Source: §4.2.5 · p. 18, §4.3.5 · p. 21

4.2.6 Minimum-Completion-Time Combination; 4.3.6 Minimum-Completion-Time Combination

  • Topology 6 + TP2EP16 + ring_direct is the minimum for Kimi K3 and GLM-5.3, but its network-mode runs do not compare inter-node protocols.For Kimi, total parameter count alone cannot predict completion time; its modeled workload dimension and TP communication proxy are below GLM-5.3 and V4 Pro.Source: §4.2.6 · p. 19, §4.3.6 · p. 21

4.2.7 Summary of Interaction Effects

  • Kimi’s DBT/Ring ratio rises from 1.05 at TP2 to 1.32 at TP8 before declining to 1.29 at TP16, unlike Flash’s monotonic amplification.A larger EP communication fraction partially dilutes the TP AllReduce difference in total time.Source: §4.2.7 · p. 19

4.3.7 Summary of Interaction Effects

  • GLM-5.3 has the highest completion time despite fewer parameters than Kimi K3 and V4 Pro, consistent with its hidden dimension, layer count, and communication proxy.Its slowest combination reaches 5.1021 s, and its global sensitivity is 15.0×.Source: §4.3.1 · p. 19, §4.3.7 · p. 21

4.4 DeepSeek V4 Pro 0813

DeepSeek V4 Pro 0813 remains communication dominated, with completion time shaped by topology, TP/EP strategy, collective implementation, and congestion-control mode. Topology 6’s advantage at low TP reverses at high TP, and DBT is slower than Ring in the evaluated mapping.

4.4.2 Topology Comparison

  • Topology 6 shifts from fastest at TP2EP16 to 2.37× slower than Topology 1 at TP16EP2 (1.6863 versus 0.7120 s).The full matrix is shown in Figure 4-4; Topology 6 is 26.1% faster than Topology 4 on average.Source: Figure 4 · p. 22, §4.4.2 · p. 22
  • Topology 6 is 26.1% faster than Topology 4, while Topologies 1 and 2 are 8.9% and 13.3% slower than Topology 6, respectively.Topologies 5, 3, and 4 are 22.9%, 29.3%, and 35.2% slower than Topology 6; additional NICs do not guarantee lower time when rank mapping constrains injection.Source: §4.4.2 · p. 22

4.4.3 TP/EP Strategy Comparison

  • TP16 communication time is 5.15× TP2, while TP16EP2 completion time is 4.93× TP2EP16 on Topology 6 with IB CC0 and Ring.GPU time falls by 0.094 s from TP2 to TP8, while communication rises by 1.128 s.Source: §4.4.3 · p. 22

4.4.4 Network and Congestion-Control Comparison

  • RoCE DCQCN is 24.3% (0.330629 s) slower than RoCE HPCC after excluding Topology 6, entirely due to communication.IB HPCC and RoCE HPCC differ by approximately 0.1%; this does not imply equivalence between the native protocols.Source: §4.4.4 · pp. 22–23

4.4.5 Collective-Algorithm Comparison

  • dbt_direct is 77.8% (0.763545 s) slower than ring_direct, exceeding the 0.330629-s RoCE HPCC–DCQCN gap after excluding Topology 6.If logical tree edges remain concentrated on a few egresses, faster feedback cannot remove structural hot spots.Source: §4.4.5 · p. 23

4.4.6 Minimum-Completion-Time Combination

  • The minimum is Topology 6 + TP2EP16 + ring_direct; its four network-mode runs agree because no external network is traversed.Under these synthetic workloads, reducing inter-node paths, high-TP Ring phases, and DBT tree-edge hot spots is more stable than total model parameters.Source: §4.4.6 · p. 23

4.4.7 Summary of Interaction Effects

  • V4 Pro’s global sensitivity is 12.4×, the lowest among the models, while its topology, TP, and algorithm interaction patterns resemble Flash and GLM-5.3.At TP16EP2, DCQCN raises Topology 3 completion time by 44% relative to IB CC0.Source: §4.4.7 · p. 23

4.5 Cross-Model Comparison

Across models, TP/EP strategy is a major source of completion-time variation, while topology, collective algorithm, and congestion-control effects depend on model and configuration. Low-TP/high-EP strategies and Ring are consistently associated with lower completion times in the evaluated matrix.

4.5.1 Comparison Basis and Overall Statistics

  • Across 144 feedback-enabled configurations, completion time ranks Flash < Kimi K3 < V4 Pro < GLM-5.3; communication contributes 89.9%–95.8% of mean time.GLM-5.3 has 26.9% as many parameters as Kimi K3 but 22.4% longer mean completion time, with 28.7% higher communication.Source: Table 4 · p. 24, §4.5.1 · p. 24

4.5.2 Cross-Model Topology Rankings

  • Topologies 1, 2, and 6 rank among the top three for every model, but their relative benefits depend on model message sequences and physical paths.Topology 6 ranks first for Flash, GLM-5.3, and V4 Pro, and second for Kimi K3; lower rankings for Topologies 3–5 vary by model.Source: Table 4 · p. 24, §4.5.2 · p. 24

4.5.3 Cross-Model TP/EP Rankings

  • TP2EP16 < TP4EP8 < TP8EP4 < TP16EP2 for every model, with TP16EP2 taking 3.68–4.35× the completion time of TP2EP16.All models meet the 64 GB weight-residency criterion; lower TP reduces AllReduce participation and serial phases.Source: Table 4 · p. 25, §4.5.3 · p. 25

4.5.4 Cross-Model Network and Congestion-Control Rankings

  • IB HPCC is 0.9% lower than RoCE HPCC after equal aggregation, while RoCE DCQCN is 23.8%–35.7% slower than RoCE HPCC.Network-mode differences correspond to communication because GPU time is unchanged; the comparisons exclude Topology 6.Source: Table 4 · p. 25, §4.5.4 · pp. 25–26

4.5.5 Cross-Model Collective-Algorithm Rankings

  • Ring beats DBT for every model, while DBT adds 28.3%–83.2% completion time depending on the model.The varying penalty is consistent with differences in message sequence, layer count, synchronization phases, and egress mapping.Source: Table 4 · p. 26, §4.5.5 · p. 26

4.5.6 Low-Time Combinations and Overall Ranking

  • TP2EP16 + ring_direct is the common low-time region, with Topologies 6, 1, and 2 leading among topology choices.After removing DBT, V4 Pro is 13.4% faster than Kimi K3, and Flash is 66.5% faster than GLM-5.3.Source: §4.5.6 · p. 26, Table 4 · p. 26

4.5.7 Cross-Model Configuration Sensitivity

  • Configuration sensitivity is nonmonotonic in absolute completion time: Flash has the highest max/min ratio at 20.6×, while V4 Pro has the lowest at 12.4×.A short GPU baseline magnifies relative variation, whereas a high communication baseline reduces relative dispersion.Source: Table 4 · p. 27, §4.5.7 · p. 27

4.6 Matched-Configuration Analysis and Ranking Stability

Matched comparisons show that model rankings are not fully invariant across system configurations. Flash remains fastest in every match, while Kimi K3 and V4 Pro reverse order depending on collective and TP/EP strategy.

4.6.1 Purpose and Statistical Basis

  • The analysis ranks models by completion time within each of 192 identical topology, TP/EP, network, and algorithm configurations.It measures workload completion-time rankings, not model capability, output quality, or per-token throughput.Source: §4.6.1 · p. 27

4.6.2 Rank Distribution

  • GLM-5.3 ranks fourth in 157 configurations but second or third in 35, so its highest overall mean does not make it slowest in every match.At TP16EP2, Kimi K3 and V4 Pro lead in 23 and 25 configurations, respectively.Source: §4.6.2 · p. 27, Table 4 · p. 27, §4.6.3 · p. 28

4.6.3 Pairwise Wins and Conditional Reversals

  • Flash is fastest in all 192 matched configurations, but Kimi K3 and V4 Pro reverse order across Ring and DBT settings.V4 Pro wins 60 of 96 ring_direct configurations, while Kimi K3 wins 85 of 96 dbt_direct configurations.Source: §4.6.2 · p. 27, §4.6.3 · p. 28, Table 4 · p. 28

4.7 Synthesis: Consistent Main Effects and Conditional Interactions

In the evaluated matrix, TP/EP and collective choice have consistent main effects, while topology and congestion-control outcomes interact with parallelism and physical mapping. The figures illustrate these conditional effects across the four models.

Consistent main effects

  • TP/EP is the largest stable source of variation, Ring beats DBT by 28.3%–83.2%, and RoCE DCQCN adds 23.8%–35.7% over RoCE HPCC.IB HPCC versus RoCE HPCC differs by only −2.9% to +0.1%; network comparisons apply only to external-network topologies.Source: §4.7 · p. 28, Table 4 · p. 28

Topology × parallelism

  • At TP16EP2, Topology 6 grows to 4.93–10.44× its Topology 6/TP2EP16 baseline and loses its low-TP consolidation advantage.Topology 4 reaches 5.44–14.61× under the same comparison; Figure 4-5 fixes RoCE HPCC + ring_direct.Source: §4.7 · p. 29, Figure 4 · p. 30

Collective × parallelism

  • DBT/Direct-to-Ring/Direct ratios rise from 1.09–1.10 at TP2EP16 to 2.01–2.15 at TP16EP2 for Flash, GLM-5.3, and V4 Pro.Kimi K3’s ratio peaks at 1.32 at TP8EP4 and then declines to 1.29.Source: §4.7 · p. 29, Figure 4 · p. 30

Congestion control × topology

  • DCQCN/HPCC ratios are highest for Topology 3 at 1.57–1.89, and the penalty varies nonmonotonically with NIC or node count.Figure 4-7 averages across TP/EP and collective settings; congestion control regulates contention but cannot remove structural hot spots.Source: §4.7 · p. 29, Figure 4 · p. 31

Overall synthesis

  • TP2EP16 + ring_direct gives low completion time for every model; Topology 6 helps mainly at low TP, while Topologies 1 and 2 rank well among multi-node designs.With an external network, IB HPCC and RoCE HPCC means are close, and RoCE HPCC outperforms RoCE DCQCN.Source: §4.7 · p. 29

5. Discussion: Coupling between Communication Structure and Physical Mapping

The discussion attributes completion-time differences to how communication groups and collective paths map onto physical links and queues. These conclusions are bounded by the fixed mapping, routing, and simulated workload rather than online request latency.

5.1 TP Collective Scaling Dominates Parallel-Strategy Rankings

  • With Direct EP and uniform routing, enlarging TP adds Ring phases and synchronization points, while larger EP groups add peers whose transfers can progress concurrently.The low-TP/high-EP region is consistently low-time across all four models; the result does not establish that larger EP is universally better.Source: §5.1 · p. 31

5.2 Collective Performance Depends on Implementation Semantics and Mapping

  • Ring’s regular neighbor traffic can outperform DBT’s mapped branches when branches share PCIe uplinks or NIC egresses, despite Ring’s greater phase count.The main comparison fixes EP Direct and varies the TP algorithm.Source: §5.2 · pp. 31–32

5.3 Effective Communication Capacity Depends on Link Use, Not Resource Count

  • More NICs or fewer nodes do not guarantee speedup: links unused by rank mapping or routing cannot shorten the slowest collective path.GPU-to-NIC affinity and contention for PCIe uplinks and switch ports determine local queues and the global critical path.Source: §5.3 · p. 32

5.4 Congestion Control Regulates Queues but Cannot Remove Structural Hot Spots

  • Congestion control can regulate traffic in shared queues but cannot remove mapping-generated edges or enlist unused NIC and PCIe links.The reported HPCC and DCQCN comparisons are conditional on the modeled buffers, ECN/PFC parameters, and bursts.Source: §5.4 · p. 32

5.5 Simulation Completion Time Is Distinct from Request Latency

  • Simulation completion time is event-DAG makespan, not online end-to-end latency; production scheduling, decode, and arrival variation are not fully encoded.Communication above 90% refers to event-DAG critical-path exposure, not raw network activity in a production GPU timeline; runtime overlap could reduce times and gaps.Source: §5.5 · p. 32

6. Practical Implications: Design-Space Pruning and Measurement Calibration

The paper recommends using controlled simulation to prune topology and parallelism choices, then diagnosing and calibrating promising configurations on the target cluster before validating production behavior.

6. Practical Implications: Design-Space Pruning and Measurement Calibration

  • Topology selection should account for GPU–NIC affinity, PCIe-switch crossings, and critical-egress load; Topology 6's low/moderate-TP benefit reverses at high TP.The paper recommends improved intra-node mapping for high TP and balanced NIC/GPU binding for multi-node designs.Source: §6 · p. 32
  • TP2EP16 + ring_direct is a preferred candidate because the weights fit across 32 GPUs and it is consistently fast.Smaller TP reduces AllReduce phases and scope, while larger EP uses Direct communication for concurrent expert dispatch and combine.Source: §6 · p. 32
  • RoCE HPCC is preferred over DCQCN only under the current simulated parameters and requires calibration to target NICs, switch buffers, ECN/PFC thresholds, and background traffic.The paper advises balancing NIC use and removing placement hot spots before tuning feedback algorithms; IB CC0 should not predict absolute latency on a real uncongested network.Source: §6 · pp. 32–33
  • The proposed workflow screens designs in simulation, diagnoses hot spots from port bytes and queue timelines, calibrates with a small request set, then validates throughput and latency on the production runtime.Validation metrics include TTFT, TPOT, and P95/P99.Source: §6 · p. 33

7. Limitations

The evaluation uses synthetic workload and simplified runtime abstractions, while its network and collective comparisons are limited to common ASTRA-sim and NS-3 implementations and parameters.

7.1 Workload and Runtime Abstractions

  • Synthetic Chakra workloads and approximate compute models omit kernel scheduling, dynamic routing, KV-cache traffic, fragmentation, expert imbalance, and routing skew.The Twall ≈ TGPU + Tcomm model also limits overlap, launch, batching, pipeline-parallel, and scheduling behavior; production TTFT, TPOT, and throughput cannot be reported directly.Source: §7.1 · p. 33

7.2 Network and Collective Implementations

  • The single-switch external network isolates endpoint injection, GPU/NIC binding, and congestion control, so comparisons apply to the evaluated backend and parameters.ASTRA-sim decomposes Ring, DBT, and Direct into point-to-point transfers; NS-3 models RDMA headers, PFC, INT, and rate feedback.Source: §7.2 · p. 33
  • Per-flow policy maps local flows to fixed local links and inter-node flows to RDMA control paths, constraining what the network comparisons represent.All algorithms and topologies use the same backend and parameters.Source: §7.2 · p. 33

8. Conclusion

Across a controlled 32-accelerator simulation matrix, exposed communication dominates mean completion time and outcomes vary with parallelism, topology, and collective choice. The authors conclude that these factors jointly shape performance within the evaluated workload and simulator semantics.

8. Conclusion

  • Under uniform 32-way sharding, Kimi K3 requires approximately 48.75 GB of checkpoint weights per rank, and all four models meet the 64 GB-per-accelerator residency criterion.The experiments cover four MoE configurations, six topologies, four TP/EP partitions, two collective algorithms, and four network/control modes.Source: §8 · p. 33
  • Exposed communication contributes 89.9%–95.8% of mean completion time; TP16EP2 takes 3.68–4.35× TP2EP16, and dbt_direct adds 28.3%–83.2% over ring_direct under fixed rank mapping.Excluding Topology 6 and equally aggregating per-model conditional means, IB HPCC time is approximately 0.9% below RoCE HPCC, while RoCE HPCC beats RoCE DCQCN for every model.Source: §8 · p. 33
  • Topology 6 benefits from no external network at low TP but loses its advantage at high TP as intra-node PCIe paths concentrate; Topologies 1 and 2 retain low multi-node times through distributed NIC injection.Every model's minimum is Topology 6 + TP2EP16 + ring_direct.Source: §8 · p. 33
  • Within the evaluated workload, rank mapping, and simulator semantics, performance jointly reflects model communication, parallelism, intra-server interconnect, and congestion control.The authors state that adding GPUs, NICs, or nominal bandwidth alone does not guarantee improvement.Source: §8 · p. 33

A. Experimental Topologies and Combinations

The experimental design compares six server topologies and combinations of TP/EP partitions, network-control modes, and collective algorithms; the accompanying table describes node configurations and key interconnects.

A. Experimental Topologies and Combinations

  • Table A-1 presents node configurations and key interconnects for the six topologies.It accompanies the topology and experiment-combination descriptions.Source: Table A-1 · p. 37
  • The experiment combines six topologies, four TP/EP partitions, four network/congestion modes, and two algorithm combinations; Topologies 1–5 use 400 Gbps external links, while Topology 6 has no external network.Endpoint-to-switch propagation delay is 0.7 µs in IB-like modes and 1 µs in RoCE modes.Source: A. Experimental Topologies and… · p. 37
  • Topologies 1–3 range from a 32-node single-GPU, single-NIC cluster to 16-node dual-GPU servers with either single- or dual-CPU organization.Topology 2 uses single-CPU servers with PCIe switches; Topology 3 uses dual-CPU servers.Source: Figure A-1 · p. 38, Figure A-2 · p. 38, Figure A-3 · p. 38
  • Topologies 4–6 consolidate the 32 GPUs into four-node eight-GPU servers, two-node 16-GPU servers, or one 32-GPU server.Topologies 4 and 5 use dual CPUs and respectively dual NICs or four NICs; Topology 6 has eight fully connected PCIe switches.Source: Figure A-4 · p. 39, Figure A-5 · p. 39, Figure A-6 · p. 39

Related papers

Loading 2609.37828v1…