Source-linked AI summary
Deploying RDMA for Distributed Inference and Training At Scale
Yibo Huang, Yi Cui, Yukai Huang, Huaping Zhou, Jiancong Tong, Zhaoyu Wu, Gen Miao, Haoran Wei, Boyan Ding, Zhuo Jiang, Jianxi Ye, Yibo Zhu, Chuanxiong Guo, Yang Xu, Jie Wu
TL;DR
Distributed inference and training need fast communication, but production RDMA must also handle variable buffers, concurrent transfers, incast, and service failures. ByteExpress combines buffer and transfer management with resource controls and RPC-assisted recovery. In production, it lowers inference fetch latency and CPU usage versus TCP, and lowers training P99 push latency with no PFC pauses observed during the reported interval.
Problem
Production RDMA integration for parameter-server inference and training must address variable buffers, concurrent connections, bursty incasts, and service failures beyond hardware offload.
Source: §3 · p. 3Method
ByteExpress is a software communication layer over hardware RDMA that manages buffers and transfers, uses an RDMA-aware allocator and zero-copy, and combines resource controls with RPC-level recovery.
Source: §1 · p. 1, §6.3 · p. 9, §11 · p. 11Results
Relative to TCP, inference average PS fetch latency fell 35.7%, P99 latency fell 43.7%, and total CPU usage fell 10.69%; training P99 push latency fell 74.0%, with no observed PFC pauses.
Source: §1 · p. 2Takeaways & Limitations
The deployment supports assigning distinct responsibilities across layers and evaluating the resulting communication path under the concurrency and failure patterns of the target workload.
Source: §11 · p. 11The evaluation uses deployed Linux kernel TCP as its baseline and does not compare all TCP tuning choices or alternative user-space transports.
Source: §8 · p. 10
Abstract
from arXiv · showhide
RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP. Variable-size messages, concurrent connections, bursty incasts, and service failures introduce challenges that hardware offload alone cannot address. This paper reports our experience deploying ByteExpress, a software communication layer over hardware RDMA, in ByteDance's parameter-server-based inference and training systems. We present three lessons from the deployment: efficient RDMA requires buffer management across application and transport layers; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer. ByteExpress combines an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and RPC-assisted recovery. In production inference, ByteExpress reduces mean and P99 model parameter fetch latency by 35.7% and 43.7%, respectively, and total CPU usage by 10.69% compared with kernel TCP. In training, ByteExpress reduces P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime.
1 Introduction
ByteDance’s parameter-server inference and training systems put communication on the critical path, but production RDMA integration must address variable buffers, concurrency, and failures. ByteExpress combines cross-layer buffer and transfer management with resource controls and RPC-level recovery.
1 Introduction
- ByteExpress manages buffers and transfers over hardware RDMA while Archon retains the RPC interface and request-level recovery responsibilities.The RNIC offloads transport; ByteExpress manages buffers, schedules transfers, and bounds outstanding work.Source: §1 · p. 1
- Variable message sizes and buffer lifetimes, concurrent fan-out and fan-in transfers, and failed nodes or replicas make production integration challenging.Transport reliability alone cannot recover from every node failure, stalled connection, or unavailable service replica.Source: §1 · p. 1
- bemalloc makes dynamically allocated application buffers RDMA-capable, enabling the communication layer to hand received buffers to application threads without copying.Batched work requests and completion-based receive notification reduce engine overhead.Source: §1 · p. 1
- QP multiplexing and thread domains address connection-state pressure and shared-driver contention, while receive credits and a per-engine send window control distinct resources.Receive credits protect posted receive capacity; the send window bounds outstanding bytes during incasts.Source: §1 · p. 1
- 35.7% lower average PS fetch latency, 43.7% lower P99 latency, and 10.69% lower total CPU usage; training P99 push latency fell 74.0% with no observed PFC pauses, relative to TCP.The training result used the software send window.Source: §1 · p. 2
2 Background
The paper’s parameter-server systems exchange sharded model parameters for inference and training, using RDMA transports whose reliability, connection model, and message limits differ. These workload patterns motivate the communication design.
2.1 Inference and Training Systems
- Workers act as rankers during inference and trainers during training; model parameters are partitioned into shards stored across parameter servers.Inference requests can generate concurrent fetches to multiple servers, while training includes parameter pushes.Source: §2.1 · p. 2
- Inference requests may evaluate multiple models and produce concurrent subrequests, with request and response sizes ranging from kilobytes to megabytes.Training exchanges parameters and pushes updates, generating incasts at workers and servers.Source: §2.1 · p. 2
2.2 RDMA: A Primer
- RDMA offloads transport processing and exposes user-space operations, but software still allocates buffers, posts operations, and processes completions.QP, send queue, receive queue, verbs, and completion queue form the endpoint and operation interface.Source: §2.2 · p. 2
- RC and UC are connection-oriented and support large messages, whereas UD is connectionless, MTU-limited, and requires reliability above the transport.ByteExpress uses RC for hardware reliability and large-message support while managing connection count in software.Source: §2.2 · pp. 2–3
- In the deployment, RC reliability, DCQCN sending-rate control, and PFC link-level pausing serve distinct purposes.The paper describes DCQCN as responding to congestion feedback and PFC as pausing traffic classes to limit buffer overflow.Source: §2.2 · p. 3
3 Deployment Requirements
Production deployment requires improving communication performance while retaining scalability and availability. Variable buffers, receive capacity, incast congestion, and service recovery call for controls at different layers.
3 Deployment Requirements
- Variable message sizes and buffer lifetimes complicate fixed receive-buffer layouts, while registering each message separately adds overhead and memory-region pressure.The design uses dynamic allocation from reusable registered memory; RC supports large transfers, while logical connections are multiplexed to limit QP-state pressure.Source: §3 · p. 3
- Receive capacity and fabric congestion are separate constraints: receive credits prevent receiver-not-ready events, while a software send window limits outstanding data.The deployment retains DCQCN and PFC alongside the software bound.Source: §3 · p. 3
- Applications use Archon RPCs while ByteExpress manages buffers and message transfers over hardware RDMA.Figure 2 depicts the communication stack.Source: Figure 2 · p. 3
- Aggregate traffic from concurrent flows can exceed switch buffering, and hardware congestion control alone did not prevent sustained PFC pauses and poor training request latency.This observation motivates bounding outstanding data rather than assuming the fabric absorbs every burst.Source: §3 · p. 3
- Reliable delivery on a functioning connection cannot select another service replica or decide how to retry a failed RPC.Archon’s RPC abstraction supplies application context for timeouts, retries, replica selection, and TCP fallback.Source: §3 · p. 3
4 System Overview
ByteExpress keeps applications on Archon’s RPC interface while translating transfers into RDMA operations and separating logical connections from physical QPs. It coordinates buffers and transport resources in BE, with Archon providing recovery above the transport layers.
4 System Overview
- Applications retain Archon RPCs through socket-like endpoints, while BE translates message transfers into RNIC verbs.Applications need not directly manage verbs, QPs, or remote addresses.Source: §4 · p. 3
- Sockets represent logical connections, while engines own RC QPs; multiplexing sockets to the same destination reduces QP count.A scheduler coordinates transfers across the sockets sharing an engine.Source: §4 · p. 3
- bemalloc supplies registered buffers to both layers, and BE uses metadata exchange and fragmented RDMA writes for large outgoing messages.Fragmentation lets the scheduler interleave sockets and enforce a per-engine send window.Source: §4 · p. 3
- Receive credits protect posted receive capacity, a software send window limits outstanding bytes, and RNIC DCQCN controls hardware sending rate.PFC remains enabled as link-level protection; Archon handles timeouts, retries, replica selection, and TCP fallback.Source: §4 · pp. 3–4
5 Building a High-Performance Communication Engine
ByteExpress’s communication engine combines RC transport with registered-buffer management and scheduling to support large, variable-size messages and application-controlled buffer lifetimes. Its design uses allocator-backed zero-copy ownership and completion-based notification for large transfers.
5.1 Choosing RC for Large-Message IO
- At 100 Gbps/1,024-byte MTU, RC reaches 92.26 Gbps versus UD’s 48.29 Gbps; at 200 Gbps/4,096-byte MTU, RC reaches 191.36 Gbps versus UD’s 114.70 Gbps.At 25 Gbps/1,024-byte MTU both approach link rate, and at 100 Gbps/4,096-byte MTU both are close to line rate; the measurements motivate RC for the evaluated workload, not a general claim about UD.Source: §5.1 · p. 4
5.2 RDMA Memory Management
- bemalloc extends rpmalloc with registered blocks and O(1) memory-region lookup while keeping registration details behind an allocation interface.Registration and deregistration costs are amortized across allocations from each block; cached blocks serve steady-state allocation and deallocation without further registration.Source: §5.2 · p. 4
- On one inference node, registered memory rises to approximately 2.2 GB and stabilizes while cached blocks continue serving allocations and deallocations.The footprint is sampled every 30 seconds from startup.Source: §5.2 · p. 5, Figure 5 · p. 5
- In allocator tests, bemalloc tracks rpmalloc in operation rate and peak memory, while reaching the highest operation rates at up to approximately 50% higher peak memory than alternatives.The tests vary thread count from one to eight and include approximately 10% cross-thread operations.Source: §5.2 · p. 5
5.3.1 Zero-Copy Buffer Ownership
- bemalloc lets the RNIC place received data directly into registered buffers that applications can retain, transfer between threads, and free after processing.This avoids coupling application buffer lifetimes to a fixed receive ring; zero-copy here means avoiding extra copies between application and communication buffers, not all processing or serialization copies.Source: §5.3.1 · p. 5
5.3.2 Transferring Large Messages
- For large messages, BE exchanges destination-buffer metadata, writes payload fragments that the scheduler can interleave, and uses a final WRITE_WITH_IMM completion to notify the receiver.The metadata exchange costs one RTT unless valid remote-buffer metadata is supplied in advance; completion semantics provide data visibility even with relaxed ordering.Source: §5.3.2 · pp. 5–6, Figure 6 · p. 5
5.4 RDMA Engine
ByteExpress engines manage a QP, receive resources, and work requests, while schedulers coordinate socket I/O. In synthetic tests, ByteExpress achieves latency and throughput comparable to or better than the plotted alternatives, with latency and throughput plots using different message-size settings.
5.4 RDMA Engine
- Each engine owns one QP, its work-request chain, and receive buffers, while sockets represent logical connections coordinated by a scheduler.The engine posts verbs operations and processes completions.Source: §5.4 · p. 6
5.4.1 Work Request I/O
- A circular work-request chain tracks in-flight, pending, and free entries, enabling batched submissions and completion processing without exceeding send-queue depth.Only the last request in a batch asks for a completion notification.Source: §5.4.1 · p. 6
5.4.2 Flow Control
- Receive credits track peer receive requests consumed by SEND and WRITE_WITH_IMM, and batch replenishment reduces posting calls and credit messages.The sender waits without the required credit; choosing k = d/2 leaves capacity available while updates are in transit, though delayed updates can still stall it.Source: §5.4.2 · p. 6
5.5 Evaluation
- ByteExpress achieves latency and throughput comparable to or better than the plotted alternatives in the tested configurations.Figure 8 compares mean and P99 latency against eRPC for 32 B–16 KB messages with one outstanding message; Figure 9 compares peak throughput across four link-rate/MTU settings.Source: §5.5 · p. 6, Figure 8 · p. 7, Figure 9 · p. 7
- Latency and throughput plots use different message-size settings and should be interpreted separately.The latency setup allows one outstanding message, whereas the throughput setup uses 1 MB messages with sufficient concurrency to saturate the path.Source: §5.5 · pp. 6–7
6 Scaling Connections & Concurrent PS Ops
This section scales concurrent parameter-server communication by coordinating connection state, thread-owned resources, receive capacity, and outstanding traffic. Its mechanisms address different bottlenecks rather than relying on hardware congestion control alone.
6.1 QP Management
- ByteExpress multiplexes logical connections onto QPs to reduce pressure on limited RNIC QP-context cache, while message fragmentation lets the scheduler interleave sockets’ work.Approximately 2 KB fragments saturated the tested 25 Gbps link; fragments larger than 8 KB were needed for near-line-rate throughput on the tested 100 Gbps link.Source: §6.1 · p. 7, Figure 10 · p. 7
6.2.1 Thread Ownership and Provider Resources
- Thread domains address provider-side contention that persisted despite single-thread engine ownership, while per-thread completion queues allow batched polling across assigned QPs.In the eight-thread experiment, 512-byte message rate dropped sharply beyond 64 QPs; profiling found substantial time in ibv_post_send() spin locks.Source: §6.2.1 · pp. 7–8, §6.2.2 · p. 8, Table 3 · p. 8
6.1 QP Management
- With 4 KB fragments, small-flow latency still rose under contention but remained substantially below the unfragmented case, so fragmentation mitigates rather than eliminates scheduling head-of-line blocking.Transport ordering, packet loss, and congestion control can still cause blocking on the shared QP.Source: §6.1 · p. 7, Figure 13 · p. 8
6.2.2 Sharing Queues Within an I/O Thread
- A per-thread shared receive queue improves buffer utilization, but its shared pool lacks the dedicated-queue scheme’s per-peer receive-capacity bound.Larger queues and RNR timeout/retry settings mitigate receive-buffer exhaustion but do not enforce that bound.Source: §6.2.2 · p. 8
6.3 Application-Level Congestion Control
- Training’s repeated synchronization can create incasts and PFC pauses; ByteExpress adds a per-engine outstanding-byte window alongside DCQCN rate control and fabric PFC.Request P99 latency is used as a proxy for communication stalls, not as a direct measure of end-to-end training throughput.Source: §6.3 · pp. 8–9
- Reducing the software window raised the reported server count before PFC from 63 without software control to 1,218 with a 4 KB window in the synthetic setup.The experiment illustrates this configuration’s effect, not a universal PFC-free scale limit.Source: Table 4 · p. 9, §6.3 · p. 9
- The configured fragment and window sizes suit relatively stable training traffic, but the per-engine bound does not cap aggregate traffic as active engines grow.The paper does not establish stability or fairness of the combined controls across arbitrary workloads.Source: §6.3 · p. 9
7 Maintaining Service Availability
Transport reliability on a functioning connection is not enough for service recovery from failed nodes, persistent path failures, or unavailable replicas. ByteExpress retains RPC-level recovery and adds connection monitoring, path changes, replica selection, and TCP fallback.
7.1 Transport Reliability & Service Recovery
- Fan-out operations are sensitive to slow or failed sub-requests, so a single stalled destination can delay a larger inference or training operation.Recovery from node failures, persistent path failures, and unavailable replicas requires decisions above the transport layer.Source: §7.1 · p. 9
7.2 Deployment and Recovery Mechanisms
- Archon preserves RPC timeouts, probabilistic retries, replica selection, and TCP fallback, while ByteExpress detects QP errors and peer failures through completions, verbs results, and keepalives.Example thresholds are 10 ms for connection timeout and 40 ms for request timeout; these serve different purposes from the keepalive mechanism.Source: §7.2 · p. 9
- Replication across mini-pods provides alternative service destinations when a node or path fails, while integrated network monitoring helps locate the underlying problem.Rankers and parameter servers are deployed across multiple geographic data centers.Source: §7.2 · p. 9
- Recovery tries an RNIC-supported path change, then a nearby replica in another mini-pod, and finally TCP if RDMA remains unavailable.In the reported deployment, TCP fallback restored high throughput within one minute.Source: §7.2 · p. 9
- The recovery mechanisms are not an end-to-end availability guarantee, and the measurements do not establish exactly-once execution or availability across all failure modes.Retry safety for state-changing operations such as Push depends on RPC and application semantics.Source: §7.2 · p. 10
8 Production Deployment and Evaluation
The production evaluation compares RDMA with the deployed Linux kernel TCP/IP stack in ByteDance inference and training workloads. RDMA with the software window lowers measured parameter-fetch latency and CPU usage in inference, while the windowed training configuration lowers P99 push latency.
8 Production Deployment and Evaluation
- The evaluation reports production CPU utilization and request latency after RDMA deployment across tens of data-center clusters for more than two years.The deployed baseline is Linux kernel TCP/IP; the results do not compare all TCP tuning choices or alternative user-space transports.Source: §8 · p. 10
8.1 Online Inference
- The inference measurements cover a Toutiao service with 300 parameter servers and 124 rankers, using matched operation loads for TCP and RDMA.Mean and P99 parameter-fetch latency were measured; these are not end-to-end recommendation latency results.Source: §8.1 · p. 10
8.2 Offline Training
- 74.0%: RDMA with the software window reduces P99 push latency relative to TCP’s 1.03 s, with no PFC pauses reported during the observation interval.Table 7 reports the busiest trainers; the measurements do not establish that the fabric is congestion-free or that PFC can safely be disabled.Source: Table 7 · p. 10, §8.2 · p. 10
8.1 Online Inference
- Relative to TCP, mean parameter-fetch latency falls by 4.5 ms (35.7%) and P99 latency by 27.6 ms (43.7%), while total CPU usage falls 10.69%.The latency figures are parameter-fetch measurements, not end-to-end recommendation latency; the largest relative CPU reductions are in software-interrupt and system CPU time.Source: §8.1 · p. 10
9 Deployment Lessons in Production
The deployment lessons center on cross-layer buffer ownership, distinct controls for receive capacity and outstanding traffic, and resource sharing aligned with I/O-thread ownership. Service recovery remains tied to RPC-level context.
Buffer ownership
- A consistent buffer-lifetime contract across allocation, serialization, transfer, and reclamation lets applications use registered memory and zero-copy transfers beneath Archon’s RPC interface.bemalloc builds on rpmalloc and lets the communication layer hand received buffers to application threads without copying.Source: §9 · p. 10
Traffic control
- Receive credits protect posted receive capacity, while the software send window bounds outstanding data above DCQCN and PFC.In production training, adding the window reduced P99 push latency to 268 ms, with no PFC pauses observed during the measurement interval.Source: §9 · p. 11
Resource sharing
- Thread domains align provider resources with application I/O-thread ownership, while per-thread queue sharing reduces resource overhead.Assigning each engine to one I/O thread removed contention in application data structures, but provider-side spin locks still limited message rate as QP counts grew.Source: §9 · p. 11
Service recovery
- Archon retains timeouts, replica selection, and TCP fallback, while BE supplies connection-error and keepalive signals for recovery.Service context includes replica placement across failure domains and excluding repeatedly failing destinations.Source: §9 · p. 11
10 Related Work
Related work spans RDMA systems, connection scaling, and congestion control. ByteExpress focuses on operational co-design across memory allocation, transport, resource sharing, and RPC-level recovery for parameter-server inference and training.
RDMA systems and machine learning
- Prior RDMA applications include key-value stores, distributed transactions, storage, cloud security, disaggregation, and RPC [22].EFLOPS and Xue et al. study RDMA for distributed training and machine learning, respectively.Source: §10 · p. 11
Connection scaling
- Connection-scaling approaches include hardware-offloaded sharing in SkyRDMA, RNIC resource scheduling, GPU fabrics, and Dynamically Connected transport.BE instead multiplexes logical sockets over RC QPs and aligns provider resources with I/O-thread ownership.Source: §10 · p. 11
Congestion control
- DCQCN, TIMELY, Swift, and HPCC explore congestion control, while BE adds an application-level send window above existing DCQCN.The software window controls outstanding RC traffic and is presented as complementary to prior transport-control designs.Source: §10 · p. 11
11 Conclusion
The conclusion frames production RDMA deployment as coordination across application interfaces, memory ownership, transport resources, and service recovery. ByteExpress combines these mechanisms, and the deployments report lower communication costs against the deployed TCP baseline.
Conclusion
- ByteExpress combines an RDMA-aware allocator and zero-copy transfers with QP multiplexing, thread-aware resource management, receive credits, and a software send window.Archon retains RPC-level context for retries, replica selection, and TCP fallback.Source: §11 · p. 11
- The reported deployments show lower parameter-fetch latency and CPU usage relative to the deployed TCP baseline, while hardware offload alone does not resolve all operational limits.The conclusion emphasizes evaluating the complete communication path under the target workload’s concurrency and failure patterns.Source: §11 · p. 11
- The broader deployment lesson is to assign each layer a clear responsibility across the communication path.The paper identifies application interfaces, memory ownership, transport resources, and service recovery as areas requiring coordination.Source: §11 · p. 11