PaperlayerView exact revision on arXiv

Source-linked AI summary

fabric-lib: RDMA Point-to-Point Communication for LLM Systems

Nandor Licker, Kevin Hu, Vladimir Zaytsev, Lequn Chen

arXiv:2510.27656v2cs.DC

TL;DR

LLM workloads such as disaggregated inference and MoE need flexible point-to-point communication, but existing RDMA solutions are hardware-specific. fabric-lib provides a common interface across ConnectX and EFA, and its production systems demonstrate portable communication across three workloads, including trillion-parameter weight updates and MoE dispatch/combine.

  • Problem

    Emerging LLM workloads need point-to-point communication beyond collectives, while existing RDMA solutions lack viable portability across hardware such as EFA.

    Source: §1 · p. 1

  • Method

    fabric-lib abstracts heterogeneous RDMA hardware behind a common interface, using WRITEIMM and IMMCOUNTER completion notification without ordering assumptions and managing multiple NICs per GPU.

    Source: §1 · pp. 1–2

  • Results

    Across three production systems, fabric-lib supports KvCache transfer, 1.3-second weight updates for trillion-parameter models, and state-of-the-art MoE decode latency on ConnectX-7 with the first viable EFA implementation.

    Source: §1 · p. 2, §9 · p. 12

  • Takeaways & Limitations

    Portable point-to-point communication complements collectives for modern LLM architectures while avoiding vendor lock-in.

    Source: §9 · p. 12

    The host-proxy design targets current hardware, while next-generation GPUs with wide NVLink domains may shift MoE communication off RDMA.

    Source: §8 · p. 12

Abstract

from arXiv · show

Emerging Large Language Model (LLM) system patterns, such as disaggregated inference, Mixture-of-Experts (MoE) routing, and asynchronous reinforcement fine-tuning, require flexible point-to-point communication beyond simple collectives. Existing implementations are locked to specific Network Interface Controllers (NICs), hindering integration into inference engines and portability across hardware providers. We present fabric-lib, which bridges the functionality of common NICs to expose a uniform interface. fabric-lib exposes one-sided WriteImm operations with a ImmCounter primitive for completion notification, without ordering assumptions of network transport, transparently managing multiple NICs per GPU. We demonstrate peak throughput of 400 Gbps on both NVIDIA ConnectX-7 and AWS Elastic Fabric Adapter (EFA). We showcase fabric-lib through three production systems: (1) KvCache transfer for disaggregated inference with dynamic scaling, (2) RL weight updates achieving 1.3 seconds for trillion-parameter models, and (3) MoE dispatch/combine implementation exceeding DeepEP decode latency on ConnectX-7, with the first viable latencies on EFA. We demonstrate that our portable point-to-point communication complements collectives while avoiding lock-in. fabric-lib is open-sourced at https://github.com/perplexityai/pplx-garden/

1 INTRODUCTION

Emerging LLM workloads need flexible point-to-point communication beyond collectives, but existing RDMA solutions are hardware-specific and often unsuitable for dynamic or sparse transfers. fabric-lib offers a portable interface across ConnectX and EFA, demonstrated in three production systems.

1 INTRODUCTION

  • Collective-based communication constrains dynamic scaling and sparse transfers, while existing point-to-point primitives often cannot be composed for viable latency.Fixed membership, synchronized initialization, and uniform buffer sizes are cited constraints.Source: §1 · p. 1
  • fabric-lib provides a common ConnectX/EFA interface using one-sided WRITEIMM and IMMCOUNTER completion notification without relying on message ordering.It transparently manages multiple NICs per GPU, including the four 100 Gbps EFA NICs needed to aggregate 400 Gbps.Source: §1 · pp. 1–2
  • KvCache transfer supports unrestricted prefiller-decoder communication and elastic scaling without synchronized initialization or fixed membership.The system was production-tested on EFA with full CUDA Graph support and layer-by-layer transfers.Source: §1 · p. 2
  • 1.3 seconds for trillion-parameter model weight updates, over 100× faster than existing RL frameworks.One-sided RDMA writes use cluster bandwidth while pipelining H2D memcpy, weight preparation, and transfer.Source: §1 · p. 2
  • MoE dispatch/combine achieves state-of-the-art decode latency on ConnectX-7, competitive with specialized DeepEP kernels, and provides the first viable implementation on EFA.Parallel token and route transfers hide device-to-host and network latency on EFA.Source: §1 · p. 2
  • The demonstrated production systems span paged writes, bulk transfers, and coordinated scatter operations on heterogeneous hardware, complementing collectives while avoiding vendor lock-in.These systems cover KvCache transfer, RL weight updates, and MoE dispatch/combine.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The background describes RDMA operations and heterogeneous transports, then reviews how collective interfaces and existing libraries address or constrain point-to-point communication. Related work spans disaggregated inference, transfer libraries, storage, and framework integration.

2.1 Network Technology

  • RDMA separates OS-assisted control-plane setup from kernel-bypassing data transfer and completion polling; it supports two-sided SEND/RECV and one-sided WRITE, with WRITEIMM adding a receiver notification.READ and atomic-operation latencies are described as unsuitable for the paper's purposes.Source: §2.1 · p. 2

2.2 Programming Interface

  • Collectives support structured transfers but constrain point-to-point use through fixed membership, synchronized initialization, operation ordering, and uniform transfer shapes.GPUDirect RDMA accesses GPU memory directly; GPU-initiated transfer is available through IBGDA only on ConnectX in the described setting.Source: §2.2 · p. 2

2.3 Related Work

  • Related work includes disaggregated inference systems, point-to-point transfer libraries, distributed KvCache storage, and framework integrations, with cited gaps including EFA support or performance.The passage also distinguishes collective-optimization work as orthogonal to RDMA primitives.Source: §2.3 · p. 3

3 TRANSFERENGINE

TransferEngine provides a portable RDMA interface across heterogeneous NICs, using reliable-but-unordered transport and IMMCOUNTER completion notifications. Its API and hardware-specific domains coordinate transfers across the NICs attached to GPUs.

3 TRANSFERENGINE

  • TransferEngine abstracts heterogeneous RDMA hardware behind a portable interface, with IMMCOUNTER notifications that do not rely on ordering guarantees.The engine also exposes SEND/RECV, WRITE, scatter, and barrier operations.Source: §3 · p. 3, §3.1 · p. 3

3.1 Overview and Design Goals

  • The minimal interface supports EFA and libibverbs NICs, transparently handling multiple NICs per GPU and using zero-copy operations and hardware-specific optimizations.A ConnectX-7 NIC provides 400 Gbps; AWS p5 instances require four 100 Gbps EFA NICs to reach equivalent bandwidth.Source: §3.1 · p. 3

3.2 Architecture

  • Each GPU has a worker coordinating its NICs, while all peers must use the same number of NICs per GPU so transfers can be sharded or balanced.A DOMAINGROUP coordinates typically 1–4 NICs; each DOMAIN manages one NIC.Source: §3.2 · p. 4

3.3 API Design

  • The API registers host or GPU memory and supports one-sided contiguous or paged writes, peer scatter, barriers, asynchronous completion notifications, and GPU-progress watchers.The caller coordinates operations across multiple devices; synchronization uses counters because operations have no other ordering guarantees.Source: §3.3 · pp. 4–5

3.4 Implementation

  • TransferEngine pins workers and domain data to NUMA-local resources, shards work across NIC domains, and polls completions before delivering aggregated notifications through a shared callback thread.Transfers can target specific NICs, and paged, scatter, and barrier operations can shard across all NICs.Source: §3.4 · p. 5

3.5 Hardware-Specific Optimizations

  • EFA uses libfabric with valid descriptors for every transfer, while ConnectX uses separate queue pairs for SEND/RECV and WRITE/WRITEIMM plus work-request chaining and relaxed ordering.EFA bulk transfers and peer groups use templated work requests; ConnectX chains up to four work requests to reduce NIC doorbell rings.Source: §3.5 · p. 5

4 KVCACHE TRANSFER

The disaggregated inference workflow transfers KV pages and context from prefiller nodes to decoder nodes. It coordinates page allocation, transfer progress, and cancellation so decoding can begin after expected transfers complete.

4 KVCACHE TRANSFER

  • A decoder pre-allocates KV pages and context storage, then dispatches the request to a prefiller with the destination page indices.The prefiller runs prefill and transfers KV pages and context, including hidden states and logits when needed.Source: §4 · p. 5
  • During chunked prefill, GPU progress triggers layer-by-layer paged writes; final context is sent separately, with layout and offsets handling differing KV-cache sharding or replication.MLA replicas are matched across ranks; GQA uses page-wise offsets and strides to copy the appropriate slices.Source: §4 · p. 6
  • The decoder starts decoding after its expected transfer count is reached, without an explicit completion message from the prefiller.The decoder uses expect imm count to receive transfer-completion notification.Source: §4 · p. 6
  • Cancellation requires prefiller confirmation before KV pages can be reused, while heartbeats and timeouts handle transport failures and pending operations are awaited before confirmation.A per-request cancellation token stops future transfers and waits for pending operations.Source: §4 · p. 6

5 RL ROLLOUT WEIGHT TRANSFER

The paper uses one-sided RDMA writes and pipelined execution to update inference weights from training GPUs. For models up to trillion-parameter scale, it reports 1.3-second cross-machine updates.

5 RL ROLLOUT WEIGHT TRANSFER

  • The solution achieves 1.3-second cross-machine parameter updates for Kimi-K2 (1T parameters), DeepSeek V3 (671B), and Qwen3 (235B), transferring weights from 256 bf16 training GPUs to 128 fp8 inference GPUs.Source: §5 · p. 6

5.1 Point-to-Point Weight Transfer

  • Each training GPU sends weights directly to inference GPUs with one-sided RDMA WRITE, using NIC bandwidth across the cluster instead of gathering and broadcasting through training Rank0.A controller gathers parameter metadata, computes a static transfer schedule, and broadcasts it to training GPUs.Source: §5.1 · p. 6

5.2 Pipelined Weight Transfer Execution

  • Weight-transfer tasks overlap host-to-device copying, parameter preparation, RDMA transfer, and a global barrier; a configurable memory watermark limits in-flight tasks.Parameters within each MeshGroup transfer in parallel, while MeshGroups are processed sequentially.Source: §5.2 · pp. 6–7

6 MOE DISPATCH/COMBINE

The section presents low-latency MoE dispatch and combine kernels coordinated by a host proxy. They use staged transfers and shared routing information, and are described as more portable than DeepEP while retaining state-of-the-art decode performance.

6.4 Comparison with DeepEP

  • The proxy-based kernels achieve state-of-the-art decode performance and support both EFA and ConnectX, while DeepEP is tied to ConnectX.DeepEP relies on IBGDA and the mlx5 driver; the section describes the added proxy overhead in the proposed kernels.Source: §6 · p. 7, §6.4 · p. 8

6.1 Architecture

  • Dispatch splits routing into sender and receiver kernels, while a GDRCopy-polled host proxy invokes the TransferEngine when source buffers are ready.The sender prepares data in send buffers; the receiver shuffles incoming tokens into tensors used by later kernels.Source: §6.1 · p. 7, §6.2 · p. 7
  • Routing counts let peers assign contiguous receive-buffer ranges; speculative private-buffer transfers hide routing latency, and combine reuses the routing information for a single scatter.Each inter-node peer issues up to two WRITEs for dispatch and one for combine.Source: §6.1 · p. 7

6.2 Dispatch

  • Around 15 µs after dispatch-kernel launch, the first transfer begins for decode-sized batches at EP=64; the initial token batch is chosen to hide host-side routing work.Routes and tokens saturate NIC bandwidth, then remaining tokens are placed contiguously after routing information arrives.Source: §6.2 · p. 7

6.3 Combine

  • Combine uses cached routing offsets to receive tokens, computes their weighted average locally, and reuses dispatch buffers after prior operations complete.The combine scatter can be prepared during the idle interval between send and combine, typically while the GPU executes grouped GEMM.Source: §6.3 · p. 8

6.4 Comparison with DeepEP

  • DeepEP starts its first transfer sooner, while the proposed kernels spend longer before the first transfer but achieve better network utilization with bulk transfers.DeepEP sends tokens one by one and uses ATOMICs for counts and completion; the proposed kernels incur additional GPU-CPU-NIC communication overhead over PCIe.Source: §6.4 · p. 8
  • For prefill, DeepEP achieves better latency by pre-accumulating tokens over NVLink, but its batched accumulation has accuracy and determinism implications.The accumulation is not entirely in fp32 or performed in a fixed order.Source: §6.4 · pp. 8–9

7 EVALUATION

The evaluation measures communication libraries and end-to-end systems on EFA and ConnectX-7 clusters. It reports fast trillion-parameter weight transfers, high point-to-point bandwidth, and KvCache transfer behavior under CUDA Graph.

7 EVALUATION

  • Evaluations use 8×H200 nodes with 2×200 Gbps EFA per GPU and 8×H100 nodes with 400 Gbps ConnectX-7 per GPU.End-to-end evaluations run on a custom inference engine built on PyTorch.Source: §7 · p. 9

EFA CX-7

  • For 256 KiB single WRITE, TransferEngine achieves 54 Gbps on EFA and 116 Gbps on ConnectX-7; 64 KiB paged WRITE saturates bandwidth on both.Figure 8 compares fractions of peak NIC bandwidth; single WRITE requires at least 16 MiB to saturate, while paged WRITE requires 32 KiB for TransferEngine and 64 KiB for NIXL.Source: EFA CX-7 · p. 9, Figure 8 · p. 9, Table 2 · p. 9

7.2 KvCache Transfer

  • For Qwen3-235B on H200 with TP4 and EFA, layer-by-layer KvCache transfer is hidden by computation; TTFT slowdown mainly reflects an extra decode pass.The evaluation uses 32 kB pages, up to 16,384-token chunk-prefill, CUDA Graph, and UvmWatcher; 1,024 pages do not saturate RDMA bandwidth.Source: §7.2 · p. 9, Table 3 · p. 9
  • Under CUDA Graph, Rust UvmWatcher callback latency is tightly bounded just above 2 µs to 5 µs PCIe latency; Python overhead is described as acceptable.Table 4 reports UvmWatcher callback tail latency.Source: §7.2 · p. 9, Table 4 · p. 9

7.3 RL Weight Transfer

  • The profiled transfer from FSDP/PP/EP=16/2/8 to 128 inference GPUs (FP8, EP=32) completes in 1.2 s, an order of magnitude faster than prior systems’ 10–100 s for comparable model sizes.Full tensor() for FSDP parameter unsharding took 518 ms and synchronization across ranks took 357 ms on the critical path; only 42 ms of RDMA transfer time fell outside pipeline overlap.Source: §7.3 · p. 10

7.4 MoE Dispatch/Combine

The MoE dispatch/combine evaluation finds strong end-to-end decode performance on both EFA and ConnectX-7, then examines overlap, kernel latency, and the mechanisms and costs behind those results.

7.4.1 End-to-End Decode Speed

  • 3–6× higher throughput than pplx-kernels on EFA, while matching or slightly exceeding DeepEP across batch sizes on ConnectX-7.The comparison is end-to-end DeepSeek-V3 decode with MTP at EP=DP=64.Source: Table 6 · p. 10, §7.4.1 · p. 10

7.4.2 Computation–Communication Overlap

  • Dual-batch overlap gives our kernels only modest gains and consistently degrades pplx-kernels; small-batch latency-sensitive workloads still require low communication latency.Overlap is effective only when per-GPU batch size is sufficiently large.Source: §7.4.2 · p. 10, Table 7 · p. 10

7.4.3 Kernel Microbenchmarks

  • On decode-sized batches, the kernels make EFA viable, outperform IBGDA-based kernels on ConnectX-7, and trail by only 30% on EFA.At 16 and 32 ranks they outperform DeepEP on dispatch and combine; at 64 ranks combine still wins, while dispatch slows from proxy overhead.Source: §7.4.3 · pp. 10–11, Figure 9 · p. 11
  • Decode-optimized kernels lack transfer chunking, so their memory overhead limits which models can viably use them for prefill.The prefill latency figure reports means with percentile error bars; the supplied text gives no comparative outcome.Source: §7.4.3 · p. 11, Figure 10 · p. 11
  • DeepEP is faster for intra-node dispatch and combine, using token replication over NVLink and sender-side partial sums to reduce RDMA bytes.The dispatch advantage tapers at EP64 as fewer replicas land on the same node; the combine comparison uses reduced bf16 accumulation precision.Source: §7.4.3 · pp. 10–11

7.4.4 Ablation: Private Buffer Size

  • At least around 32 private-buffer tokens are needed intra-node; inter-node ConnectX-7 can use 24, while EFA drops off below 32.Smaller buffers leave too little capacity to hide route-exchange latency.Source: Figure 11 · p. 11, §7.4.4 · p. 11

7.4.5 Ablation: Send and Receive Latency

  • Kernel execution takes under 15% of transfer time; dispatch receive is an outlier because it pulls data with NVLink loads.Combine receive is faster due to faster accumulation, and shuffling adds about 15 µs of idle time.Source: Figure 12 · p. 11, §7.4.5 · pp. 11–12

7.4.6 Ablation: Host-Proxy Overhead Breakdown

  • At EP64, total WRITE posting time stays under 10 µs at p50 on ConnectX-7 and under 28 µs on EFA, without preventing DeepEP parity or better.Posting time increases roughly linearly with expert parallelism; posting WRITEs is the dominant cost, including libfabric overhead on EFA.Source: Table 9 · p. 12, §7.4.6 · p. 12

8 DISCUSSION

The discussion positions the host-proxy design for current hardware, while noting that GPU-initiated RDMA availability and future GPU interconnects bound its role.

8 DISCUSSION

  • GPU-initiated RDMA is unavailable on most cloud instances, while wide-NVLink GPUs may shift MoE communication off RDMA; RDMA latency is already hidden by computation for KvCache and RL transfers.Additional NIC support would require per-hardware tuning, not a library redesign, and TransferEngine applications would remain unchanged.Source: §8 · p. 12

9 CONCLUSION

fabric-lib layers a reliable, unordered abstraction over heterogeneous RDMA hardware. Production examples span KvCache transfer, RL weight updates, and MoE dispatch/combine, supporting portable point-to-point communication alongside collectives.

9 CONCLUSION

  • fabric-lib addresses RDMA vendor lock-in with a reliable abstraction that does not assume ordering and supports ConnectX and EFA.Its focus is extending support across heterogeneous NICs.Source: §9 · p. 12
  • KvCache transfer applies the library to disaggregated inference.Source: §9 · p. 12
  • RL weight updates reach 1.3 seconds for trillion-parameter models.Source: §9 · p. 12
  • MoE dispatch/combine delivers state-of-the-art latency on ConnectX-7 and the first viable implementation on AWS EFA.The paper concludes that portable point-to-point communication complements collectives while avoiding vendor lock-in.Source: §9 · p. 12

A KVCACHE TRANSFER PSEUDOCODE

The pseudocode describes decoder-to-prefiller dispatch and prefiller-to-decoder transfers of KV pages and tail context. It uses completion counting and a UVM watcher to overlap transfers with computation, while omitting several production details.

A KVCACHE TRANSFER PSEUDOCODE

  • Although presented synchronously and blocking, the operations run in practice as asynchronous tasks on a Tokio event loop.The pseudocode describes the disaggregated KV-cache transfer workflow using the TransferEngine API.Source: A KVCACHE TRANSFER PSEUDOCODE · p. 17
  • Tail context means auxiliary GPU tensors beyond the KV cache, such as output logits, whose contents depend on the inference engine.The transfer pattern remains the same across tail-context contents.Source: A KVCACHE TRANSFER PSEUDOCODE · p. 17
  • The pseudocode omits tensor-parallel and MLA sharding, multiple KV buffers, chunked-prefill bookkeeping, prefix-cache reuse, and WRITE stride and offset calculations.Chunked prefill leaves the total transferred pages and expected IMM count unchanged but requires additional bookkeeping.Source: A KVCACHE TRANSFER PSEUDOCODE · p. 17
  • The decoder dispatch message carries KV and tail memory descriptors plus page indices, enabling direct writes into decoder GPU memory.The prefiller receives commands, and the final context is transferred separately as a single write.Source: A KVCACHE TRANSFER PSEUDOCODE · p. 17
  • The decoder preregisters an IMMCOUNTER for the expected WRITEIMM completions before dispatch, eliminating the need for an explicit completion message from the prefiller.The decoder’s message supplies memory descriptors and page indices so the prefiller can write directly into decoder GPU memory.Source: A KVCACHE TRANSFER PSEUDOCODE · p. 17
  • A UVM watcher triggers per-layer KV page transfers after attention, while the next layer computes.A CPU-side callback submits WRITEIMM for that layer; the pseudocode waits until all page and tail writes complete.Source: A KVCACHE TRANSFER PSEUDOCODE · p. 17

B RL WEIGHT TRANSFER PSEUDOCODE

The RL weight-transfer workflow precomputes routing from training parameters to inference memory regions. Training workers then reuse that routing at each training step without replanning or coordination.

B RL WEIGHT TRANSFER PSEUDOCODE

  • The controller gathers parameter metadata and memory-region descriptors, computes and broadcasts a static routing table, and workers reuse it for each weight transfer without replanning or coordination.Figure 16 depicts the RL training and weight-update workflow.Source: B RL WEIGHT TRANSFER PSEUDOCODE · p. 17, Figure 16 · p. 17
Loading 2510.27656v2…