Source-linked AI summary
DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
Wanqian Li, Jintao Peng, Zongfei Jing, Tianyu Zhang, Ze Long, Xianjie Qiao, Xiaoming Chen, Dongxu Yang, Kefeng Duan, June Yang
TL;DR
Multi-GPU LLM inference strategies require layer-boundary synchronization, which can turn workload imbalance into global waiting. DWDP keeps ranks data-parallel while fetching remote MoE weights on demand, allowing independent execution. On GB200 NVL72, it improves end-to-end output TPS/GPU by 8.8% at comparable TPS/user in 20–100 TPS/user 8K/1K serving.
Problem
LLM growth pushes inference beyond single-GPU memory, while existing multi-GPU parallelization requires layer-wise coordination that makes workload imbalance costly.
Source: §1 · p. 1Method
Results
8.8% higher end-to-end output TPS/GPU at comparable TPS/user in 20–100 TPS/user serving with 8K input and 1K output sequences.
Source: §1 · p. 2Takeaways & Limitations
DWDP supports flexible GPU-count configurations for finer-grained resource provisioning and rate matching between context and generation servers in disaggregated serving.
Source: §6 · p. 12At high TPS/user, generation bottlenecks limit context tokens available to amortize prefetch overhead, so DWDP may provide limited benefit or slightly regress.
Source: §5.3 · p. 11
Abstract
from arXiv · showhide
Large language model (LLM) inference increasingly depends on multi-GPU execution, yet existing inference parallelization strategies require layer-wise inter-rank synchronization, making end-to-end performance sensitive to workload imbalance. We present DWDP (Distributed Weight Data Parallelism), an inference parallelization strategy that preserves data-parallel execution while offloading MoE weights across peer GPUs and fetching missing experts on demand. By removing collective inter-rank synchronization, DWDP allows each GPU to progress independently. We further address the practical overheads of this design with two optimizations for split-weight management and asynchronous remote-weight prefetch. Implemented in TensorRT-LLM and evaluated with DeepSeek-R1 on GB200 NVL72, DWDP improves end-to-end output TPS/GPU by 8.8% at comparable TPS/user in the 20-100 TPS/user serving range under 8K input sequence length and 1K output sequence length.
1 Introduction
DWDP addresses layer-wise synchronization under imbalanced multi-GPU inference by keeping ranks data-parallel and fetching peer-hosted MoE weights on demand. In DeepSeek-R1 on NVL72, it raises output TPS/GPU by 8.8% at comparable TPS/user in 20–100 TPS/user 8K/1K serving.
1 Introduction
- Model growth pushes LLM inference beyond single-GPU memory, while model-parallel strategies require layer-boundary coordination that scheduling and load balancing do not remove.Source: §1 · pp. 1–2
- At 20% per-rank sequence-length variation, synchronization overhead reaches approximately 12%, as request- and weight-level imbalance amplifies local latency into global waiting.Figure 1 quantifies the synchronization overhead under DeepSeek-R1 on GB200 with 8K/1 input lengths ranging from 0.8×8K to 8K.Source: §1 · pp. 1–2, Figure 1 · p. 2
- DWDP preserves data-parallel ranks while offloading MoE weights across peer GPUs and fetching missing experts on demand, removing collective synchronization from the critical path.Source: §1 · p. 2
- DWDP introduces split-weight management and asynchronous-prefetch overheads, which the paper addresses with two optimizations and analysis of communication-computation interference.Source: §1 · p. 2
- 8.8% higher end-to-end output TPS/GPU at comparable TPS/user in 20–100 TPS/user 8K/1K serving with DeepSeek-R1 on GB200 NVL72.Source: §1 · p. 2
2 Methodology Overview
DWDP keeps attention weights replicated and partitions MoE experts across peer GPUs, fetching missing experts as needed. Asynchronous transfers overlap computation so ranks can progress independently, while flexible expert placement supports finer-grained GPU provisioning.
2 Methodology Overview
- DWDP replicates attention weights on each rank and partitions MoE experts across peer GPUs, fetching missing experts to assemble each layer’s weights.Source: §2 · p. 3
- Ranks prefetch remote experts for layer l+1 while executing layer l’s MoE block and layer l+1’s attention block, using double buffering to sustain the pipeline.Source: §2 · p. 3
- Instead of synchronizing collectives, each rank pulls remote experts from peers with cudaMemcpyAsync and proceeds independently once a layer’s required experts arrive.Source: §2 · p. 3
- Expert placement need not be perfectly disjoint or divisible by group size; permitted redundancy supports single-rank provisioning and can reduce remote prefetch overhead when memory allows.Source: §2 · p. 3
- The design’s practical challenges include split-weight management, communication-computation interference, and source-rank many-to-one contention during prefetch.Source: §2 · p. 3
3 Preliminary Analysis
The preliminary roofline analysis compares DWDP4 with DEP4 for DeepSeek-R1’s context phase on GB200, modeling whether remote-weight prefetch can be hidden by computation. It finds that DWDP’s advantage depends on context length and batch size, and is bounded when computation dominates.
3 Preliminary Analysis
- The layer-wise roofline model estimates DWDP latency as max(compute, prefetch) and DEP latency as compute plus expert-parallel all-to-all.It compares compute-to-prefetch ratio and DEP-to-DWDP runtime ratio across input sequence lengths.Source: §3 · p. 4
- At batch size 1, DWDP begins to outperform DEP at around 16K tokens, when longer contexts provide enough compute time to hide remote-weight prefetch.Increasing batch size enlarges the compute window and can make DWDP beneficial at shorter contexts.Source: §3 · p. 4
- At very long sequence lengths, computation dominates both methods, so synchronized all-to-all is a smaller share of DEP latency and DWDP’s marginal speedup declines.Figure 3 compares DWDP4 and DEP4 at batch size 1; its y=1 line marks the prefetch-hidden and DWDP-outperformance boundary.Source: §3 · pp. 4–5, Figure 3 · p. 4
- The analysis assumes perfectly balanced workloads and omits both DWDP’s additional synchronization benefit under imbalance and practical prefetch-overlap issues such as contention.The authors discuss these implementation issues and corresponding optimizations in the following section.Source: §3 · p. 5
4 Design Challenges and Optimizations
The section identifies baseline DWDP’s remaining costs—split-weight merging, communication-computation interference, and source-side contention—and presents techniques to address them. Its contention analysis motivates round-robin, sliced transfers to improve progress across concurrent destination pulls.
4.1.1 Kernel Breakdown Analysis
- The baseline DWDP implementation yields an 11.69% net iteration-latency improvement over DEP after regressions reduce a 21.86% gross reduction; it adds 34.00 μs of D2D-copy time.The breakdown attributes regressions to D2D copying and slower Attention and Others categories, with overlap interference dominated by power-induced frequency throttling in this setting.Source: §4.1.1 · p. 5, Table 1 · p. 5
4.1.2 Runtime-Trace Analysis
- Concurrent remote-weight pulls from different destinations can target one source rank, serializing at its copy engine and exposing compute bubbles when the compute window is comparable to prefetch time.Figure 4 shows a representative case under max_num_tokens=16384 and input sequence lengths of 4K–8K.Source: §4.1.2 · p. 6, Figure 4 · p. 6
4.2 Eliminating Split-Weight Merge Overhead
- DWDP’s split local and prefetched remote weights require a contiguous merge for conventional groupedGEMM; extending the kernel to consume multiple buffers directly eliminates that merge overhead.The TensorList-based design remains compatible with existing layouts and sharding schemes, with no meaningful performance regression reported.Source: §4.2 · p. 6
4.3.1 Theoretical Analysis
- The analysis models each rank’s N−1 serial remote pulls and counts concurrent pulls to a shared source, but does not translate contention probabilities into exact communication time.A fully serialized equal-size approximation would give a pull with contention degree C roughly Cτ latency; actual latency depends on system factors.Source: §4.3.1 · p. 7
- Under the random asynchronous model, low-order contention is most likely, while the probability mass of higher-order contention grows gradually with group size.The model computes contention probability rather than exact communication time; actual latency depends on system factors.Source: §4.3.1 · p. 7, Table 2 · p. 7
4.3.2 Copy with Time-Division Multiplexing
- Fixed-size transfer slices scheduled round-robin across destination queues let the pipelined copy engine interleave progress instead of letting one large pull monopolize it.With two small slices in flight, slowdown appears only when both encounter contention degree at least 3; contention degree 2 on both does not slow the rank-level pull in this model.Source: §4.3.2 · pp. 7–8
5 Experimental Results
Experiments evaluate DWDP in context-only and disaggregated end-to-end serving. DWDP improves output throughput per GPU at similar user throughput, while its gains depend on workload and serving conditions.
5.1 Experimental Setup
- On GB200 NVL72, DeepSeek-R1 experiments use NVFP4 MoE weights, FP8 KV cache, and a disaggregated-serving DEP baseline.DWDP is applied to the context server; the generation-server configuration is kept unchanged.Source: §5.1 · p. 8
5.2 Context-Only Results
- Across ISLs of 1K–32K, DWDP achieves 1.09–1.11× TPS/GPU and 1.11–1.27× TTFT speedup; larger MNT settings also yield higher speedups.The larger token budget creates a wider compute window for hiding prefetch overhead.Source: §5.2 · p. 9, Table 3 · p. 9
- As input-length imbalance increases, DWDP achieves 1.08–1.15× TPS/GPU and 1.11–1.18× TTFT speedup; DWDP3 and DWDP4 have nearly identical TPS/GPU speedup.DWDP3 has worse TTFT speedup but supports finer-grained resource allocation than conventional DEP configurations.Source: §5.2 · p. 9, Table 3 · p. 9
- Split-weight merge elimination improves DWDP TPS/GPU by about 3% by removing a D2D merge copy from the critical path.It does not meaningfully regress groupedGEMM execution.Source: §5.2 · p. 10
- Contention mitigation adds the most benefit when MNT is smaller and the ISL ratio is lower, where short compute windows leave less communication hidden.At ISL ratio 0.5 and MNT 16K, merge-elimination-only DWDP is slightly below DEP, while full DWDP is substantially higher; the added benefit is smaller at MNT 32K.Source: §5.2 · p. 10, Table 4 · p. 10
5.3 End-to-End Results
- DWDP achieves higher output TPS/GPU than the baseline at similar TPS/user across most target ranges.Figure 5 presents the Pareto comparison, and Table 5 summarizes average speedup by TPS/user range.Source: §5.3 · p. 10, Figure 5 · p. 11, Table 5 · p. 11
- DWDP typically uses fewer context GPUs than the baseline at comparable Pareto points, raising output TPS/GPU through improved throughput per context GPU.The end-to-end evaluation uses 8K maximum input and 1K output lengths, with DWDP applied only to the context server.Source: §5.3 · pp. 10–11
- DWDP increases median TTFT across evaluated TPS/user ranges, and efficiency gains can shrink or regress at high TPS/user when prefetch overhead is difficult to amortize.Reducing context GPUs can lower context service rate and increase queueing delay before the first token.Source: §5.3 · pp. 11–12
6 Conclusion
DWDP enables asynchronous execution across ranks and supports finer-grained GPU provisioning. It requires high-bandwidth communication, and efficient implementation calls for co-design across communication, kernels, and runtime.
6 Conclusion
- DWDP enables asynchronous execution across ranks and supports finer-grained GPU-count configurations for resource provisioning and context-generation rate matching.The conclusion frames these as benefits for increasingly diverse and imbalanced LLM serving workloads.Source: §6 · p. 12
- High-bandwidth communication, such as that provided between GPUs within NVL72, is a key requirement for hiding remote-weight prefetch behind computation.The authors identify communication, kernels, and runtime co-design as an ongoing efficiency challenge.Source: §6 · p. 12
A In-Depth Analysis of Communication-Computation Interference
Overlapping remote-weight prefetch with computation introduces hardware contention that reduces expected gains. The transfer uses shared memory-system resources on both GPUs, creating interference pathways beyond SM occupancy.
A In-Depth Analysis of Communication-Computation Interference
- Remote-weight prefetch overlapped with computation introduces hardware contention and reduces the expected gain.The copy engine does not occupy SM computation resources, but transfer traffic still traverses shared NoC, L2, DRAM, and power resources on both GPUs.Source: A In-Depth Analysis of… · p. 15, Figure 6 · p. 15
- NVLink transfers can interfere through NoC arbitration, L2 pressure, DRAM bandwidth, and power contention, potentially triggering DVFS frequency throttling.The transfer path fetches data from source DRAM through L2 and NoC to NVLink, then writes it into the destination memory hierarchy.Source: A In-Depth Analysis of… · p. 15
- Interference depends on whether communication overlaps with memory-bound kernels or compute-intensive kernels.Source: A In-Depth Analysis of… · p. 15
A.1 Contention with Memory-Bound Kernels: Memory Bandwidth Bottleneck
NVLink traffic competes with memory-bound kernels for bandwidth, and the measured slowdown is below the worst-case estimate because overlap is partial and L2 absorbs some activation traffic.
A.1 Contention with Memory-Bound Kernels: Memory Bandwidth Bottleneck
- At full NVLink utilization, memory-bound kernels could slow by up to 22.5%; measured Others overhead rises about 17.6%, from 241.69 μs in DEP4 to 284.32 μs in DWDP4.The observed slowdown is smaller because communication does not overlap all kernels and L2 cache absorbs part of the activation traffic.Source: §A.1 · pp. 15–16
A.2 Contention with Compute-Intensive Kernels: Power-Induced Frequency Bottleneck
The overlap study attributes heavy-compute and communication slowdown to power-induced frequency throttling, rather than L2, DRAM, or NVLink contention. Attention-kernel runtime tracks GPU frequency across overlap patterns, with Short-Duration Overlap most representative of DWDP.
A.2 Contention with Compute-Intensive Kernels: Power-Induced Frequency Bottleneck
- About 1.19× slower, DWDP4’s Attention kernel runs than DEP4’s during context-phase attention, indicating severe interference when CE communication overlaps heavy SM execution.The passage attributes the dominant interference in this compute-intensive workload to power-induced frequency throttling.Source: §A.2 · p. 16
- 114.4% of the TDP limit is the estimated combined power draw during overlap, exceeding the limit and triggering DVFS that reduces GPU frequency.The estimate combines attention at 96.7% TDP with two-sided communication at 30.5% TDP, subtracting the 12.9% idle baseline.Source: §A.2 · p. 16
- Short-Duration Overlap best matches real DWDP: tightly scheduled attention and small transfers repeatedly inject contention into power-constrained windows; the authors suggest avoiding fine-grained patterns that amplify power spikes.Intermittent Compute provides a no-overlap baseline, while Long-Duration Overlap preserves gaps that allow partial power recovery.Source: §A.2 · pp. 16–17, Figure 7 · p. 16
- 1.226× normalized attention-kernel runtime accompanies 0.798× normalized GPU frequency under Short-Duration Overlap; Long-Duration Overlap is 1.049× and 0.963×, respectively.Both measures are normalized to the Intermittent Compute baseline, and the curves closely follow each other.Source: Table 7 · p. 17, §A.2 · p. 17, Figure 8 · p. 17
- L2 and DRAM utilization stays at 45%–50%, adjusted L2 traffic remains near 1.6–1.8 GB against about 24 GB total, and NVLink throughput shows no noticeable fluctuation.Together, these observations argue against memory-system or NVLink contention as the dominant slowdown source.Source: §A.2 · p. 17