Source-linked AI summary
RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication
Chutian Wang, Wenhao He, Jingmin Zhu, Qingyu Yin, Heng Xu, Xiuyu Li
TL;DR
Irregular, input-dependent All-to-All communication can bottleneck expert-parallel MoE models, even when routing and placement are fixed. RailWave combines source-local Rail balancing with reusable permutation scheduling and phase-adaptive path selection; it reports up to 5.84× speedup on H800 and 4.36× on H20 over Native.
Problem
Irregular, input-dependent dispatch and combine exchanges in expert-parallel MoE models can make All-to-All communication a major execution bottleneck.
Source: §1 · p. 1Method
RailWave combines source-local RailBalance, reusable topology-derived permutation scheduling, and a calibrated selector to shape spatial and temporal traffic execution.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported results demonstrate benefits from jointly shaping the spatial and temporal execution of expert-parallel traffic.
Source: §5 · p. 9Aggregate Rail equality depends on eligible sets and how integer remainders are distributed across sources.
Source: §3.1 · p. 5
Abstract
from arXiv · showhide
Irregular All-to-All communication is a major bottleneck in expert-parallel Mixture-of-Experts (MoE) models. Even with fixed expert routing and placement, uneven utilization of parallel network Rails and incast can limit communication performance. We present RailWave, a phase-adaptive communication layer built on DeepEP that addresses these bottlenecks below the routing layer through spatial and temporal traffic shaping. RailBalance redistributes source traffic across eligible Rails using source-local information, while a reusable, topology-derived permutation schedule limits concurrent senders per receiver without rebuilding demand-dependent schedules for each communication phase. A lightweight calibrated selector chooses an execution path according to each phase's traffic characteristics and offline profiling results. On training-derived communication workloads from the 106B GLM-4.5-Air model, RailWave delivers up to 5.84x speedup on H800 and 4.36x on H20 over Native. Code is available at https://github.com/CyberSecurityErial/RailWave-EP.
1 INTRODUCTION
Irregular, input-dependent All-to-All exchanges can bottleneck expert-parallel MoE execution, and routing or placement alone does not determine physical traffic behavior. RailWave shapes traffic across Rails and transfer waves, reporting communication speedups over Native on H800 and H20.
1 INTRODUCTION
- MoE expert-parallel dispatch and combine exchanges are irregular and input-dependent, making All-to-All communication a major execution bottleneck.MoE activates only a subset of parameters per token, while expert parallelism distributes experts across GPUs.Source: §1 · p. 1
- Routing, placement, and replication shape logical communication demand but do not fully determine physical execution, particularly in Rail-optimized clusters.Replication-based methods can also incur per-layer memory overhead from redundant expert states.Source: §1 · p. 1
- Balanced expert and destination loads can coexist with hot source Rails, while balanced source Rails can still leave multiple senders converging on one receiver.Analysis of 900 training step–layer pairs found expert-load imbalance has limited linear explanatory power for Rail imbalance.Source: §1 · p. 1, Figure 1 · p. 2
- RailWave combines source-local RailBalance across eligible Rails with reusable topology-derived permutation scheduling that bounds incast without rebuilding demand-dependent schedules for each phase.The layer is built on DeepEP and jointly controls where traffic enters the scale-out network and which node-pair transfers overlap.Source: §1 · pp. 1–2
- Because Rail balancing over a whole phase can still concentrate traffic within a wave, a calibrated selector chooses among four paths using traffic characteristics and offline profiling.The selected path is based on current traffic characteristics and validated offline profiling results.Source: §1 · p. 2
- Up to 5.84× communication speedup on H800 and 4.36× on H20 over Native execution.Evaluated on real GLM-4.5-Air training workloads across two 32-GPU clusters.Source: §1 · p. 2
2 PRELIMINARIES
Expert-parallel dispatch and combine create irregular All-to-All traffic whose physical performance depends on Rail utilization and transfer overlap, not only routing and placement. RailWave addresses these execution-level bottlenecks while leaving the logical demand unchanged.
2.1 EXPERT-PARALLEL COMMUNICATION
- Dispatch sends routed token representations to remote experts, while combine returns their outputs; routing and placement determine exchange participants and volume.These exchanges form irregular All-to-All communication.Source: §2.1 · p. 2
2.2 RAIL-OPTIMIZED NETWORKS AND TRAFFIC DESCRIPTORS
- B captures communication scale, ρg source-Rail imbalance, and κ receiver concentration under actual sending windows.Source-Rail imbalance and receiver concentration describe distinct traffic conditions.Source: §2.2 · p. 3
2.3 COMMUNICATION BOTTLENECKS WITH FIXED ROUTING
- Among 900 training step–layer pairs, expert-load imbalance has limited linear explanatory power for Rail-load variation, with R2 = 0.019 for source-Rail imbalance and 0.296 for destination-Rail imbalance.The six selected cases had similar expert-load ratios (8.2–8.4) but destination-Rail assignment shares ranging from 15.5% to 24.1%.Source: §2.3 · p. 3, Figure 2 · p. 4
- 67.1%: with rank-to-rank demand, payload, backend, and rounds fixed, the sum of per-layer P50 dispatch–combine latencies rises from 1.801 to 3.009 s under stronger incast.Figure 2(c) compares replay orders with identical rank-to-rank demand.Source: §2.3 · p. 3, Figure 2 · p. 4
2.4 MOE COMMUNICATION OPTIMIZATION
- RailWave operates after routing and placement methods: it controls Rail distribution and concurrent transfers, while payload reduction and computation overlap remain complementary optimizations.Compared with demand-dependent Birkhoff scheduling, it reuses topology-derived cyclic peer plans with source-local RailBalance.Source: §2.4 · p. 4
3 SYSTEM DESIGN
RailWave combines source-local Rail redistribution with node-pair scheduling and selects among four execution paths using validated offline decisions. Its design covers preparation, selection, and execution, with reusable peer plans and phase-specific traffic statistics.
3 SYSTEM DESIGN
- RailWave’s three-stage workflow combines reusable peer plans with a selector that chooses among NATIVE, RAIL-ONLY, PERMUTATION-ONLY, and JOINT execution paths.The selected plan guides dispatch and combine, preserving the original token positions.Source: §3 · p. 5
3.1 SOURCE-LOCAL RAILBALANCE
- RailBalance redistributes each node pair’s traffic across eligible Rails using source counts and fixed egress–ingress correspondence, without a destination-side quota exchange.Its guarantee applies to each source’s contribution; aggregate equality also depends on eligible sets and integer-remainder distribution.Source: §3.1 · p. 5
3.2 REUSABLE PERMUTATION SCHEDULING
- Reusable cyclic pairwise scheduling gives each node at most one outgoing and incoming peer per wave, so every remote ordered pair appears once.Disjoint pairs communicate concurrently, while waves execute in order; Joint can additionally redistribute Rails within each wave, with gains depending on Rail skew and relocation cost.Source: §3.2 · p. 6
3.3 CALIBRATED EXECUTION SELECTION
- The selector weighs Rail-utilization gains against redistribution cost and reduced incast against extra rounds, using payload, source-Rail skew, and receiver concentration.Offline profiling retains stable alternatives with conservative advantages over Joint; nearby requests reuse validated decisions, while requests outside calibrated neighborhoods fall back to Joint.Source: §3.3 · pp. 6–7
4 EXPERIMENTS
Across three replay patterns, RailWave outperforms Native on H800 and H20, with gains persisting at higher latency percentiles. Ablations show that the best execution path depends on Rail skew and receiver concentration, motivating calibrated adaptive selection.
4.1 EXPERIMENTAL SETUP
- The evaluation replays routing statistics from actual GLM-4.5-Air training across 45 MoE layers and compares four DeepEP execution paths alongside NCCL and FAST.Experiments use four-node, 32-GPU H800 and H20 clusters; reported latency excludes CPU planning, index construction, and prepacking.Source: §4.1 · p. 7, §4 · p. 7
4.2 OVERALL PERFORMANCE
- RailWave achieves 2.02–5.84× speedups over Native at P50 on H800 and 1.74–4.36× on H20 across Ring, Moderate, and Extreme, with gains persisting at P95 and P99.The communication-performance figures cover the three replay patterns on both clusters.Source: §4.2 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
4.3 ABLATION AND MECHANISM ANALYSIS
- In Ring, Rail-only closely matches Joint while Permutation-only remains close to Native, indicating spatial shaping provides most of the gain when incast is limited.The ablation separates RailBalance from permutation scheduling.Source: §4.3 · p. 8, Figure 7 · p. 8
- Under Moderate and Extreme, Joint improves performance on both clusters, reaching 1.44× and 1.97× speedups over the better single mechanism on H800, respectively.RailBalance improves Rail utilization, while permutation scheduling controls incast.Source: §4.3 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
- Under Moderate on H20, Native sends approximately 54.12%, 24.33%, 14.85%, and 6.69% of traffic across four NIC positions, while Rail-only and Joint produce approximately equal shares in both directions.Permutation-only retains the uneven distribution.Source: §4.3 · p. 8, Figure 8 · p. 9
- When Rail skew and receiver concentration are both high, Joint reduces mean latency to 4.11 ms, versus 10.96 ms for Permutation-only and 73.28 ms for Rail-only.Other, more balanced replay cases can favor a lighter execution path.Source: §4.3 · p. 8, Figure 9 · p. 9
- On 12 unseen requests, all four cases where the calibrated selector switched away from Joint reduced median communication latency by 5.7–16.4%, or 1.061–1.196× versus Joint.Three requests selected Permutation-only and one selected Native; the other eight fell back to Joint.Source: §4.3 · p. 8, Figure 10 · p. 9
5 CONCLUSION
RailWave targets physical-network inefficiencies that remain after expert routing and placement are fixed. Its combined spatial and temporal traffic shaping improves parallel Rail utilization and controls receiver-side concurrency, with communication speedups over Native on both evaluated platforms.
5 CONCLUSION
- RailWave combines source-local RailBalance with reusable permutation scheduling to shape expert-parallel traffic spatially and temporally while preserving expert assignments and logical communication demand.The communication layer is built on DeepEP.Source: §5 · p. 9
- RailWave targets physical-network inefficiencies that remain after expert routing and placement are fixed.Source: §5 · p. 9
- RailWave achieves communication speedups of up to 5.84× on H800 and 4.36× on H20 over Native.Source: §5 · p. 9
A.1 HARDWARE AND SOFTWARE CONFIGURATION
The evaluation uses two four-node clusters with eight GPUs per node. Their H800 and H20 nodes differ in their ConnectX-7 adapter and port configurations; the compared DeepEP paths share a backend and launch-level configuration.
A.1 HARDWARE AND SOFTWARE CONFIGURATION
- Both evaluation clusters contain four nodes with eight GPUs per node.Table 1 summarizes their hardware and software configurations.Source: §A.1 · p. 14, Table 1 · p. 14
- Each H800 node has eight 400-Gb/s ConnectX-7 adapters, while each H20 node has four adapters configured with two 200-Gb/s ports apiece.H800 GPUs have one PIX-proximate adapter each; H20 PCIe-local GPU pairs share an adapter with a separate port per GPU.Source: §A.1 · p. 14
- Native, Rail-only, Permutation-only, and Joint use DeepEP 2.1.0 and the same launch-level communication configuration within each paired replay experiment.The configuration allocates 129 GIN communication contexts and uses 16 communication SMs with automatic QP selection.Source: §A.1 · p. 14
A.2 INTEGRATION WITH ULTRAEP
The UltraEP integration evaluates source-node skew and source-Rail skew using synthetic traffic directed to one destination node. Results show that the useful execution strategy depends on whether source Rails are already balanced, while the reported weight-synchronization measurements do not represent full-size expert weights.
A.2 INTEGRATION WITH ULTRAEP
- The integration evaluates two traffic families in which three source nodes send to one destination: source-node skew with locally balanced Rails, and source-Rail skew with traffic concentrated on some Rails.These patterns expose receiver-side contention, with the second also introducing source-Rail imbalance.Source: §A.2 · p. 14
- The workloads use approximately 28 MiB or 42 MiB per GPU per All-to-All operation, with routing approximately uniform across the destination node’s eight experts.They correspond to 65,536 and 98,304 remote tokens, respectively.Source: §A.2 · pp. 14–15
- For source-node skew, Permutation achieves 9.694× speedup over Native at 28 MiB and 8.670× at 42 MiB; adding RailBalance provides little additional benefit.Source-side Rails are already approximately balanced in this workload.Source: §A.2 · p. 15, Table 2 · p. 15
- For source-Rail skew, Joint achieves 7.862× speedup over Native at 28 MiB and 9.064× at 42 MiB, and exceeds the better single mechanism by 2.125× and 3.295×, respectively.Source: §A.2 · p. 15, Table 2 · p. 15
- The preferred execution strategy depends on traffic: temporal scheduling suffices when Rails are balanced, while joint spatial and temporal shaping gives larger gains when Rail imbalance and receiver contention coexist.Source: §A.2 · p. 15
- Weight-synchronization measurements use FC tensors with only 1,024 BF16 elements and therefore do not represent synchronization costs for full-size MoE expert weights.The measurements verify execution of the UltraEP integration path.Source: §A.2 · p. 16
A.3 ADAPTIVE SELECTOR CALIBRATION AND VALIDATION
The selector uses normalized payload, Rail-imbalance, and contention coordinates, with calibration anchors reused only for nearby requests. Its validation measures 24 H800 requests and quantifies benefit only on actual path switches.
A.3 ADAPTIVE SELECTOR CALIBRATION AND VALIDATION
- The selector normalizes payload as log2(B/56), Rail imbalance as (ρg−1)/7, and contention as (κ−1)/2; these scales are configuration-specific.The 56-MiB reference corresponds to 32,768 tokens per node in the stated BF16, hidden-dimension-7,168 workload; the scales are re-established when topology or workload regime changes.Source: §A.3 · p. 16
- Anchors require repeatable measurements with p90/p10 ≤1.2 and a conservative cross-quantile advantage of at least 5%.Low-contention development cases also require equivalent-path controls to agree within 10%; the filters leave four anchors.Source: §A.3 · p. 16
- At runtime, anchors are reused only within a 1.25× payload range and 0.02 on both normalized shape coordinates; requests outside these neighborhoods fall back to Joint.The payload guard keeps the 7-, 14-, and 28-MiB neighborhoods disjoint, and all thresholds and anchors were frozen before unseen-request evaluation.Source: §A.3 · pp. 16–17
- Across the 24 requests, seven actual path switches were used to quantify selector benefit, while the remaining 17 requests fell back to Joint.Table 5 summarizes measured validation; its speedup ranges include only requests where the selector changes execution path.Source: Table 5 · p. 17, §A.3 · p. 17
- Validation measured 24 H800 requests using five warmups, interleaved measurements, and the median of the first 10 samples.Timing covered the complete dispatch–combine interval and used the maximum across ranks.Source: §A.3 · p. 17
A.4 PACKING OVERHEAD
Pipelining the next wave’s packing with current-wave communication reduces exposed packing overhead and improves end-to-end dispatch–combine latency over sequential packing.
A.4 PACKING OVERHEAD
- Permutation-only and Joint require per-wave packing, compared across PREPACKED, SEQUENTIAL, and PIPELINED approaches.PIPELINED overlaps packing of the next wave with communication of the current wave; Table 6 reports dispatch–combine median latency.Source: §A.4 · p. 17
- 2.87–3.68% end-to-end latency improvement over sequential packing comes from pipelining next-wave packing with current-wave communication.The reported differences are measured at the full communication-path level, not as isolated packing-kernel times.Source: §A.4 · p. 17
A.5 STATIC MEMORY OVERHEAD
RailWave uses communication workspace instead of redundant expert storage, while replay workloads preserve measured training-derived loads and vary sender arrangements to cover incast conditions.
A.5 STATIC MEMORY OVERHEAD
- RailWave trades communication workspace for the persistent storage of redundant experts; training replica gradients add storage depending on their residency duration.Table 7 reports dominant incremental storage in GiB/GPU, with EPLB assuming BF16 expert weights and FP32 replica gradients.Source: §A.5 · pp. 17–18, Table 7 · p. 18
- 4.11 GiB/GPU of additional RailBalance workspace is required under the stated configuration, and sequential waves reuse that workspace.The example uses D=4, G=8, T=50,176, and H=7168, with sd=sc=2H+32.Source: §A.5 · p. 17
A.6 REPLAY WORKLOADS AND EXECUTION PROTOCOL
- Replay uses training-derived layer loads with controlled sending arrangements rather than one incidental training timeline, reflecting variation in communication overlap.The motivation notes that compute-time variation, rank skew, and network jitter can change overlap despite unchanged logical execution order.Source: §A.6 · p. 18
- MODERATE and EXTREME preserve each source’s total volume and destination-local-GPU histogram while using peak incast degrees [2, 1, 2] and [3, 3, 2], respectively.EXTREME is a controlled stress case, not an estimate of how often severe incast occurs in training; Table 8 encodes scheduled senders per receiver.Source: §A.6 · p. 18, Table 8 · p. 18
- In MODERATE and EXTREME, Permutation-only and Joint reorder the same demand into three one-to-one waves, while Native, Rail-only, and NCCL retain the input schedule.Each remote node pair appears once, preserving total volume and wave count; timing covers all waves but excludes CPU planning, index construction, and prepacking.Source: §A.6 · pp. 18–19