Source-linked AI summary
VarioPath: Workload-Aware All-to-All Communication for PCIe GPU Clusters
Yao Fei, Jin Fang, Size Zheng, Gongming Zhao, Hongli Xu, Jiacheng Zhu, Zhijing Xin
TL;DR
AlltoAllv scheduling on PCIe GPU systems must contend with shared links, skewed traffic, and changing demand. VarioPath combines offline topology analysis with online demand decomposition across reusable channels. It outperforms existing systems and reduces Qwen3 prefill and Wan2.1 generation latency.
Problem
PCIe AlltoAllv scheduling must account for shared-link contention and skewed traffic, while demand changes make static schedules unsuitable.
Source: §2.1 · p. 3Method
VarioPath precomputes reusable topology-feasible channels and uses an online scheduler to decompose each invocation’s demand across them.
Source: §1 · p. 2, §7 · p. 12Results
VarioPath outperforms existing systems and reduces Qwen3 prefill and Wan2.1 generation latency by up to 27.2% and 6.1%, respectively.
Source: §7 · p. 13Takeaways & Limitations
Separating topology-dependent channel analysis from demand-dependent scheduling supports lightweight per-invocation AlltoAllv scheduling without repeating topology search.
Source: §7 · p. 12The channel-analysis DFS may visit O(2^|R|) candidate subsets in the worst case, though pruning reduces the explored and stored choices.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
AlltoAllv communication is a critical primitive in distributed large-model inference, particularly for mixture-of-experts (MoE) models. The growing adoption of PCIe GPU systems for cost-efficient inference makes AlltoAllv performance on these systems increasingly important. Without a dedicated scale-up interconnect (e.g., NVLink or Infinity Fabric), PCIe GPU systems carry both intra-node and inter-node traffic through the PCIe hierarchy, where concurrent transfers can contend for PCIe link bandwidth. This link contention, compounded by skewed traffic distributions and dynamic traffic demand, makes efficient AlltoAllv scheduling challenging. Existing approaches are either poorly suited to PCIe GPU systems or incur substantial schedule synthesis overhead that reduces their practicality in real-world deployments. We present VarioPath, an efficient AlltoAllv scheduling framework for PCIe GPU systems. It combines an offline topology-aware analyzer with an online demand-aware scheduler. The analyzer records contention-free transfer patterns as AlltoAllv channels and exploits topology symmetry to build a compact catalog for efficient search. The online scheduler decomposes each AlltoAllv invocation's demand across a sequence of channels, adapting to rapidly changing and skewed traffic while incurring low planning overhead. Evaluation on four platforms (up to 256 GPUs) shows average AlltoAllv speedups of 5.88x over FAST and 1.72x over DeepEP. End-to-end experiments show that VarioPath reduces Qwen3 inference latency by up to 27.2% and Wan2.1 generation latency by 6.1%.
1 Introduction
VarioPath addresses AlltoAllv bottlenecks on PCIe GPU systems by combining offline topology analysis with online demand-aware scheduling. Across four platforms, it improves AlltoAllv performance and reduces inference latency.
1 Introduction
- Skewed and dynamic AlltoAllv traffic complicates many-to-many coordination and can make communication a major distributed-inference bottleneck.Input-dependent expert routing can cause a few destinations to receive substantially more data than others.Source: §1 · pp. 1–2
- PCIe systems carry intra-node and inter-node traffic through shared links, where contention can reduce throughput despite their cost-control advantages for inference.Unlike dedicated scale-up fabrics, PCIe GPU and NIC paths share the PCIe hierarchy; these systems are also widely deployed for inference.Source: §1 · p. 1
- Existing approaches are poorly matched to PCIe AlltoAllv: scale-up-dependent methods can increase contention, static schedules do not adapt to demand, and MILP synthesis can take seconds to hours.PCIe-oriented systems instead tend to target regular collectives rather than AlltoAllv.Source: §1 · p. 2
- VarioPath separates topology-dependent offline analysis from demand-dependent online scheduling to provide topology-aware, demand-adaptive scheduling with low runtime overhead.The online scheduler decomposes each invocation’s demand across a sequence of channels.Source: §1 · p. 2
- Its channels encode contention-free concurrent transfer patterns; topology symmetry organizes them into a compact catalog for efficient search.The bottleneck-aware scheduler allocates each invocation’s demand across channels.Source: §1 · p. 2
- Up to 5.88× AlltoAllv speedup and up to 27.2% lower inference latency are reported for VarioPath.The evaluation spans four PCIe GPU platforms, from single-node servers to 256 GPUs, and includes Qwen3 and Wan2.1 inference.Source: §1 · p. 2
2 Background and Motivation
PCIe AlltoAllv scheduling must handle shared-link contention, skewed transfers, and changing demands. VarioPath addresses these challenges with joint peer–path selection and chunk-level remapping over reusable, topology-derived channels.
2.1 AlltoAllv Scheduling Challenges on PCIe Systems
- PCIe endpoint-disjoint transfers can still contend on shared links, while skewed transfer sizes leave lighter paths idle and create communication tails.Figure 1 models aggregate bandwidths of 128, 228, and 356 GB/s for fastest-path assignment, path-only adaptation, and joint peer–path selection, respectively.Source: §2.1 · pp. 2–3, Figure 1 · p. 3
2.2 Why Existing Designs Fall Short
- Existing designs either depend on an independent scale-up fabric, cannot adapt physical paths to changing demand, or require schedule synthesis that takes minutes to hours.FAST and DeepEP add intra-node transfers that share PCIe resources with NIC traffic; general-purpose libraries use static schedules.Source: §2.2 · pp. 3–4
2.3 Design Observations and Key Insight
- Joint peer–path selection avoids modeled PCIe contention where path-only adaptation still leaves transfers contending.The compared stages reach 128, 228, and 356 GB/s for fastest-path assignment, path-only adaptation, and joint selection, respectively.Source: Figure 1 · p. 3, §2.3 · p. 4
- Chunk-level remapping can balance path loads, while reusable channels separate stable topology constraints from invocation-specific demand.The example splits transfer A and remaps its second chunk to the NIC; the resulting decomposition replaces repeated topology search with lightweight online scheduling.Source: Figure 2 · p. 3, §2.3 · p. 4
3 Scheduling Model
The scheduling model represents an AlltoAllv demand as rank-pair data volumes over available paths and modeled shared resources. It builds valid schedules from feasible channel activations and estimates their completion time.
3 Scheduling Model
- The model represents an invocation with a rank-pair demand matrix and assigns each available path an estimated rate and a set of modeled resources.Self-traffic is omitted, and the modeled resources include directed shared PCIe and NIC-facing resources.Source: §3 · p. 4
- A channel pairs a permutation of ranks with one selected physical path per pair, giving each rank one outgoing and one incoming lane.A channel activation assigns a nonnegative integer volume to each lane, and its nonzero-volume lanes execute concurrently.Source: §3 · p. 5
- A valid schedule uses feasible activations that exactly reconstruct demand, and its modeled duration sums activation overheads and slowest-lane completion times.The objective captures the trade-off between shortening communication tails and limiting the overhead of additional activations.Source: §3 · p. 5
4 VarioPath Design
VarioPath separates topology-dependent analysis from invocation-dependent scheduling: it reuses a catalog of feasible channel families, then instantiates channels and allocates each invocation’s demand across activations.
4.1 Topology-Aware Channel Analysis
- Channel shapes encode lane counts between rank groups, while balanced rows and columns preserve one outgoing and one incoming lane per rank.Larger lane counts can favor group pairs with greater demand.Source: §4.1 · pp. 5–6
- A channel family realizes a shape only when its selected path instances meet group-pair counts and use each modeled resource at most once.Endpoint binding assigns rank pairs one-to-one; balance alone does not ensure realizability, as M1 has no compatible family while B1 realizes M2.Source: §4.1 · p. 6, Figure 4 · p. 6
- The catalog is built by pruned depth-first search, with path- and family-level dominance filters reducing search branching and exposed alternatives.The DFS may visit O(2^|R|) candidate subsets in the worst case, and the catalog is reused per machine configuration.Source: §4.1 · pp. 6–7
4 VarioPath Design
- Each online iteration selects a channel shape, binds candidate channels, chooses the highest-utility activation, and subtracts its served demand until none remains.The catalog is traversed coarse to fine, narrowing topology choices at the group level while retaining demand-aware endpoint and volume assignment.Source: §4.2 · p. 7
4.2 Channel-Guided Demand Decomposition
- For each iteration, the scheduler scores feasible shapes using residual group-pair demand as bottleneck pressure, then recomputes pressure after updating demand.The fixed lane budget means assigning more concurrency to one group pair leaves less for others.Source: §4.2 · p. 7
- For each family under the chosen shape, alternating source and destination assignments seek a high-scoring binding; lane volumes and candidate utility then determine the activation.The Alternating Algorithm solves assignment problems without decreasing the binding score; utility accounts for useful bytes, modeled duration, and per-activation overhead.Source: §4.2 · pp. 7–8
5 Implementation
VarioPath combines topology-aware profiling with device-resident communication kernels. Its implementation targets efficient data movement, low synchronization overhead, sparse expert traffic, and runtime adaptation.
5 Implementation
- Topology profiling reconstructs the GPU/HCA PCIe hierarchy, enumerates direct-P2P and NIC paths, and records shared-resource footprints and bandwidths for reuse across calls.Packing groups data by destination into contiguous symmetric send buffers; receive kernels unpack directly into the output layout, avoiding a separate reordering pass and fine-grained transfers.Source: §5 · p. 8
- Double buffering avoids global barriers when resetting signals, while device-resident counters and slot indices keep state consistent across CUDA Graph replays.The avoided global barrier measured approximately 20 µs on the 32-GPU RTX 5090 testbed.Source: §5 · p. 8
- Sparse expert transfers skip zero-demand source–destination pairs and encode local-expert activity and token counts in a control word sent after payload on the same queue pair.The receiver uses the bitmap to skip inactive experts and obtains active-expert counts from GIN signals, avoiding separate count transfers and output-count atomics for empty groups.Source: §5 · p. 8
- A specialized unsignaled-put protocol polls sentinel-initialized receive buffers; its 1 KiB microbenchmark reaches 7.872 µs P50 latency, versus 12.928 µs for GIN signal and 9.726 µs for prepacked NCCL LL.The runtime sets invocation-wide time quantum from a target round count between N and 3N, trading scheduling overhead against finer-grained adaptation.Source: §5 · p. 9
6 Evaluation
Across communication workloads, VarioPath outperforms the evaluated baselines, and end-to-end experiments report lower Qwen3 prefill and Wan2.1 generation latency. Its scheduler approaches a full MILP optimum, while the catalog supports deployments up to 256 GPUs with reported online planning overhead.
6.1 Collective Communication Performance
- 1.55×–2.00× geometric-mean speedup over DeepEP across 42 random-demand AlltoAllv test points; VarioPath has the highest bandwidth at every point.It also records 3.49×–9.41× speedup over FAST and 3.73×–10.61× over NCCL; joint peer and path selection avoids shared-link concentration.Source: §6.1 · p. 10, Figure 7 · p. 10
- 1.82× geometric-mean speedup over NCCL and 1.99× over MSCCL across 80 AlltoAll configurations, with only one configuration where it does not outperform MSCCL.This result covers uniform traffic; VarioPath adapts paths by stage, unlike the fixed transport paths of NCCL and MSCCL.Source: §6.1 · p. 10, Figure 8 · p. 10
- Across the Zipfian skew sweep from s = 0.4 to 0.8, VarioPath bandwidth drops 9.4–12.3%, versus NCCL’s 15.2–27.0%; FAST declines 2.0–6.0% yet remains slower throughout.Source: §6.1 · p. 11, Figure 9 · p. 11
- 30.91 GB/s at M = 128 on 256 GPUs, versus 28.11 GB/s for DeepEP and 18.85 GB/s for NCCL.VarioPath has the highest algorithmic bandwidth at all eight evaluated token counts.Source: §6.1 · p. 11, Figure 10 · p. 11
6.2 End-to-End Inference Performance
- 19.1–27.2% lower Qwen3–30B–A3B prefill latency across EP16, EP24, and EP32; Wan2.1 speedup ranges from 1.011× to 1.066× across four sequence-parallel layouts.The Wan2.1 benefit depends on workload layout, not sequence-parallel degree alone.Source: §6.2 · p. 11, Figure 11 · p. 11
6.3 Design Validation
- 99.18% of MILP-optimal bandwidth on average across 50 skew AlltoAllv instances, corresponding to a 0.82% average optimality gap.The peer–path ablation finds completion time rises to 2.84× the unrestricted design when both peers and paths are fixed; Table 3 reports normalized completion time.Source: §6.3 · p. 12, Table 3 · p. 12, Figure 12 · p. 12
6.4 Catalog Tree Scale and Planning Overhead
- 3–1,616 shapes and 6–2,140 families occupy 1.7 KiB–168.6 MiB across deployments from 8 to 256 GPUs; construction takes 0.8 s–94.6 min.Online planning takes 3.2–78.12 µs per activation and accounts for 0.0715–4.1241% of communication time; the one-time catalog cost is amortized across demands on the same topology.Source: §6.4 · p. 12, Table 4 · p. 12
7 Conclusion
VarioPath separates topology-dependent channel analysis from demand-dependent scheduling for PCIe GPU AlltoAllv. Across four platforms and deployments up to 256 GPUs, it outperforms existing systems and reduces Qwen3 prefill and Wan2.1 generation latency.
7 Conclusion
- VarioPath compresses feasible channels into reusable families offline, then schedules demand through channel activations online without repeating topology search.This separates topology-dependent analysis from demand-dependent scheduling.Source: §7 · p. 12
- The online scheduler decomposes each invocation’s demand into channel activations.The decomposition avoids repeating topology search during scheduling.Source: §7 · p. 12
- Across four PCIe platforms and deployments up to 256 GPUs, VarioPath outperforms existing systems and reduces Qwen3 prefill latency by up to 27.2% and Wan2.1 generation latency by 6.1%.The framework is evaluated at scales up to 256 GPUs.Source: §7 · pp. 12–13