PaperlayerView exact revision on arXiv

Source-linked AI summary

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang

arXiv:2602.21548v2cs.DC

TL;DR

Agentic LLM inference is constrained by repeated, high-volume KV-Cache retrieval, while PD-disaggregated systems concentrate storage traffic on prefill engines and leave decode-engine bandwidth underused. DualPath adds a storage-to-decode loading path with RDMA transfer to prefill engines and workload-aware scheduling, improving offline throughput by up to 1.87× and online serving throughput by 1.96× on average.

  • Problem

    Agentic workloads repeatedly reuse long contexts, making KV-Cache retrieval I/O-intensive and exposing storage-bandwidth imbalance in PD-disaggregated inference.

  • Method

    DualPath loads KV-Cache through either storage-to-prefill or storage-to-decode paths, then uses RDMA transfer, traffic isolation, and dynamic scheduling.

  • Results

    1.87× offline inference throughput improvement and 1.96× average online serving throughput improvement are achieved on representative agentic workloads.

  • Takeaways & Limitations

    Redistributing storage-network load across prefill and decode engines improves throughput for long-context, high-cache-reuse agentic inference workloads.

  • Takeaways & Limitations

    Large-scale experiments did not demonstrate additional JCT or serving-capacity gains over equivalent-cost multiple small-scale units because parallelism settings and P/D ratios were not fine-tuned.

Abstract

from arXiv · show

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput. We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines. Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87$\times$ on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96$\times$ without violating SLO.

1 INTRODUCTION

Agentic LLM inference involves long, multi-turn sessions whose accumulating contexts make KV-Cache storage I/O a central bottleneck. DualPath addresses prefill-side bandwidth pressure by loading caches through both storage-to-prefill and storage-to-decode paths, with traffic isolation and dynamic scheduling.

  • Motivation: Agentic systems accumulate context across dozens or hundreds of tool-mediated turns, making multi-turn inference a critical production workload.Examples include coding assistants and autonomous systems.
  • Existing bottleneck: Prefill engines load KV-Cache from remote storage while decode engines often retain unused storage bandwidth, creating an uneven bottleneck.This imbalance makes simply provisioning more prefill-side bandwidth costly or impractical.
  • DualPath: DualPath loads KV-Cache either directly into prefill engines or into decode engines for RDMA transfer to prefill engines.Dynamic path selection redistributes storage-network load and alleviates prefill-side pressure.
  • System design: DualPath isolates KV-Cache traffic from latency-sensitive model-execution communication and dynamically balances computation and network utilization.The design targets interference involving collective operations and heterogeneous workloads.
  • Results: DualPath improves offline end-to-end throughput by up to 1.87× and online serving throughput by 1.96× on average.The evaluation uses representative agentic workloads with long contexts and high cache reuse.

2 BACKGROUND

Agentic LLM workloads repeatedly reuse long contexts while appending short tool or user outputs, making external KV-Cache storage important. Modern infrastructure separates compute and storage networks and commonly uses prefill–decode disaggregation and layerwise prefill.

  • LLM inference: Decoder-only LLMs store attention keys and values as KV-Cache in HBM to avoid recomputation.Attention enables token interactions, while feed-forward networks process tokens independently.
  • PD-disaggregated inference: PD disaggregation assigns prompt processing to prefill engines and token generation to decode engines with distinct compute and memory patterns.Prefill is compute-intensive and batched, whereas decode is memory-bound and latency-sensitive.
  • Layerwise prefill: Long-context layerwise prefill allocates and frees KV-Cache by layer, reducing HBM requirements and increasing effective batch size.This exploits the locality that each layer needs only its own layer-specific cache.
  • Agentic workloads: Agent trajectories can span dozens or hundreds of turns, with contexts reaching up to one million tokens and typically more than 95% of tokens reused across rounds.Only newly appended context generally requires prefill computation.
  • Storage requirements: External SSD-based storage is needed for large agentic working sets because DRAM and HBM can store only a small proportion of long-run KV-Caches.RL rollout states further constrain DRAM available for KV-Cache.
  • Data-center architecture: AI data-center nodes provide separate high-bandwidth compute NICs and storage NICs, with the compute and storage fabrics isolated from each other.The separation is intended to reduce interference with inter-GPU communication.

3 BOTTLENECK & MOTIVATION

Agentic inference under PD disaggregation is bottlenecked by high KV-Cache retrieval demand, declining I/O relative to compute, and storage-bandwidth imbalance across engine types. DualPath exploits decode-node storage bandwidth and the faster compute network to relieve this bottleneck.

  • Bottleneck: Agentic inference shows severe GPU underutilization because KV-Cache loading speed is limited by the single storage NIC on each node.The bottleneck reflects high cache-retrieval demand and uneven storage-network utilization.
  • Workload characteristics: Representative coding traces average 157 rounds, with 32.7k-token contexts, 429-token append lengths, and a 98.7% KV-Cache hit rate.These workloads require substantial cache I/O but relatively little new prefill computation.
  • I/O intensity: The cache-compute ratio for DeepSeek-V3.2 is approximately 22 GB/PFLOP at 429 appended tokens, placing significant pressure on storage bandwidth.Models with larger KV-Cache sizes face an even worse ratio according to the passage.
  • Hardware trends: From NVIDIA Ampere to Blackwell, the I/O-compute ratio decreases by 14.4×, while limited NIC bandwidth and HBM capacity constrain utilization.The hardware trend makes communication and memory capacity increasingly mismatched with GPU FLOPS.
  • Bandwidth imbalance: PD-disaggregated systems centralize hit-token storage reads on prefill engines, leaving decode-engine storage NICs largely idle.Consequently, aggregate storage-network bandwidth cannot be fully harnessed.
  • Opportunity: DualPath uses decode-node storage NICs to load KV-Cache and transfers it back to prefill nodes over the higher-bandwidth compute network.This dual-path architecture directly targets the prefill-side storage I/O bottleneck.

4 DUALPATH SYSTEM OVERVIEW

DualPath rethinks KV-Cache retrieval in PD-disaggregated inference by adding a storage-to-decode path alongside the conventional storage-to-prefill path. Its buffering, transfer, and scheduling design aggregates storage bandwidth while targeting bottleneck-free operation under stated hardware and workload assumptions.

  • 4.1 Dual-Path Loading: DualPath adds a storage-to-decode path that transfers KV-Cache to prefill engines over RDMA, alongside the conventional storage-to-prefill path.The system dynamically distributes traffic across both paths to use decode-side storage bandwidth.
  • 4.1 Dual-Path Loading: The dual-path design aggregates storage NIC bandwidth across engines, including otherwise-idle decode-side NICs, removing asymmetric bandwidth saturation.This turns storage I/O into a globally pooled and schedulable capacity.
  • 4.1 Dual-Path Loading: KV-Cache buffers on prefill and decode engines support layerwise streaming and overlap transfers with computation during prefill.The PE and DE read paths use different buffer flows, while decode begins after the complete prompt KV-Cache is available.
  • 4.1 Dual-Path Loading: DualPath uses Full Blocks for storage interactions and Layer Blocks for layerwise transfers between buffers and GPU HBM.This layout supports streaming in both PE and DE read paths.
  • 4.2 Bottleneck-Free Analysis: Under load-balanced scheduling, DualPath analyzes storage, compute-NIC, PCIe, and DRAM pressures to identify configurations without hardware bottlenecks.The analysis assumes a well-configured PCIe topology, no compute-network congestion, and fully utilized storage read bandwidth.
  • 4.2 Bottleneck-Free Analysis: For g = 8, s = 1, M ≈ 500 GB/s, and B_s ≈ 50 GB/s, the bottleneck-free range is 1/7 ≤ P/D ≤ 7/2.The stated range covers most practical configurations.
  • 4 DUALPATH SYSTEM OVERVIEW: DualPath addresses fine-grained transfer overhead and potential interference with latency-sensitive collective communications as core system challenges.These challenges arise from fragmented layerwise KV-Cache data and additional compute-network and PCIe traffic.

5 CNIC-CENTRIC TRAFFIC MANAGER

The CNIC-centric traffic manager routes GPU-related data traffic through paired CNICs and uses compute-network QoS to isolate KV-Cache transfers from model-execution communication. This design also reduces small-transfer submission overhead relative to cudaMemcpyAsync.

  • Traffic Isolation: DualPath routes GPU H2D and D2H traffic through paired CNICs using GPUDirect RDMA, consolidating traffic onto the compute network.Native compute-network QoS then differentiates traffic classes.
  • Traffic Isolation: Inference communication uses a high-priority virtual lane, while KV-Cache transfers use a separate low-priority lane with weighted bandwidth arbitration.The InfiniBand configuration reserves approximately 99% of total bandwidth for high-priority traffic.
  • Traffic Isolation: The same isolation principles can extend from InfiniBand to RoCE and emerging interconnects through traffic classes, packet markings, and hardware queues.The paper specifically identifies TC and DSCP markings for RoCE.
  • Traffic Isolation: Existing GPUDirect Storage and CUDA copy-engine methods do not isolate KV-Cache traffic from latency-sensitive collective communications.The paper identifies this interference as a source of degraded inference performance.
  • CNIC-Centric Data Transfer: The CNIC-assisted path reads KV-Cache into host DRAM, uses an RDMA Write for local H2D transfer, and reverses the process for persistence.The CNIC becomes the QoS scheduler for GPU PCIe traffic.
  • CNIC-Centric Data Transfer: CNIC-assisted H2D and D2H provide a practical way to prevent KV-Cache transfers from degrading critical model-execution communication.The paper notes that this path may detour through host memory compared with direct alternatives.
  • CNIC-Centric Data Transfer: A single cudaMemcpyAsync copy incurs approximately 5-7 μs, whereas one RDMA Write work request takes around 1 μs for small data chunks.The RDMA submission involves a few user-space MMIO writes to NIC registers.

6 ADAPTIVE REQUEST SCHEDULER

DualPath uses adaptive inter-engine and intra-engine scheduling to balance NIC traffic, GPU utilization, request counts, HBM capacity, and attention-layer execution time. Its policies select paths and batches using queue, token, memory, and compute signals.

  • 6 ADAPTIVE REQUEST SCHEDULER: Scheduling has two levels: inter-engine assignment selects a PE-DE pair and read path, while intra-engine scheduling selects requests for each forward batch.The two levels jointly address NIC traffic and GPU utilization balance.
  • 6.1 Inter-Engine Scheduling: Engine groups reduce scheduler pressure by having Leader Engines interact with the scheduler and engines report unfinished requests, token counts, and disk-queue lengths.Token count is used as a proxy for balancing load across engines.
  • 6.1 Inter-Engine Scheduling: PE scheduling prioritizes engines with short disk queues and acceptable token load, avoids overloaded engines, and assigns requests to the minimum-token PE.The policy uses thresholds α and β to form engine categories.
  • 6.1 Inter-Engine Scheduling: DE scheduling first assigns requests across groups by minimum total token count, then places them within a group according to remaining HBM and token pressure.The within-group policy uses a high-token threshold Z and avoids assignments without sufficient HBM.
  • 6.1 Inter-Engine Scheduling: After choosing a PE-DE pair, the scheduler reads KV-Cache on the side with the shorter reading queue.Splitting one request across both sides is left as future work.
  • 6.2 Intra-Engine Scheduling: Compute-quota scheduling targets similar attention execution times across GPUs, reducing synchronization bubbles caused by workload imbalance.This matters when data parallelism assigns different requests to GPUs that must synchronize before the FFN stage.
  • 6.2 Intra-Engine Scheduling: Intra-engine scheduling uses FIFO packing and profiled attention-layer time estimates to keep each batch within a predefined compute quota.If necessary, binary search finds a smaller token workload for chunked prefill.

7 EVALUATION

DualPath is evaluated on three models and production agent-trace workloads using offline and online throughput metrics. It consistently improves throughput over Basic, with gains from dual-path loading and scheduling, while large-scale experiments show near-linear scaling.

  • 7.1 Implementation: Experiments use three models, production agent traces, InfiniBand GPU servers, and 3FS storage with physically isolated computation and storage networks.The evaluated models are DS 660B, DS 27B, and Qwen 32B; each dataset contains 500 trajectories.
  • 7.2 Experimental Setup: Offline inference measures job completion time, while online serving measures TTFT, TTST, and TPOT.Offline experiments model simultaneous agent rollouts and measure completion when all requests finish.
  • 7.3 Offline Batch Inference: 1.87×: DualPath improves DS 660B offline throughput over Basic, while DS 27B improves by up to 1.78× and Qwen 32B shows similar trends.DualPath benefits more from larger batches and longer maximum agent lengths; SGL(MC) failed on some large configurations.
  • 7.3 Offline Batch Inference: 1.64× average speedup: DualPath improves DS 27B performance across 1P1D, 2P1D, and 1P2D configurations, reaching up to 2.46×.Across append-length scales, DualPath achieves 1.82−1.99× speedup over Basic.
  • 7.4 Online Serving: 2.25×: DualPath increases DS 660B APS capacity over Basic, with comparable TTST and no additional decoding overhead according to TPOT.For DS 27B, both systems have substantially higher TPOT than Oracle, indicating considerable basic P-D transfer overhead in that case.
  • 7.5 Ablation Study: 45.62%: combining dual-path loading with scheduling reduces JCT versus Basic, compared with 38.19% for dual-path loading alone.Scheduling also improves storage-NIC balance from 1.53 to 1.18 and maintains an attention-layer Max/Avg ratio as low as 1.06 early in the task.
  • 7.6 Large-Scale Scalability: 22×: the 44P88D online configuration raises throughput from 0.4 to 8.8 APS while maintaining similar latency, while offline scaling from 2P4D to 48P96D is near-linear.The corresponding offline JCTs are 3,167s and 3,201s; scheduler CPU usage remains below 10 cores.
  • 7.6 Large-Scale Scalability: Large-scale experiments do not demonstrate additional JCT or serving-capacity gains over equivalent-cost small-scale units because parallelism and P/D ratios were not fine-tuned.The authors still identify reduced fragmentation and more scheduling opportunities as benefits of large-scale deployment.

8 DISCUSSION

The discussion identifies dynamic workloads and working-set growth as important deployment constraints. Evaluation assumptions about inter-arrival and tool-call gaps can substantially enlarge storage requirements beyond the measured setting.

  • 8.1 Potential Future Work: Offline agentic workloads are highly dynamic, with prefill pressure typically higher during the first half of execution than the second.The authors call for adaptive parallelism and P/D-ratio configuration, including simulators or online adjustment mechanisms.
  • 8.2 Working Set Analysis: 69 GB to 681 GB: DualPath’s estimated DS 660B serving KV-Cache working set grows across APS 0.1 to APS 0.45.The approximation uses arrival rate, mean JCT, and average total trajectory length.
  • 8.2 Working Set Analysis: Realistic inter-arrival and tool-call gaps would enlarge the working set beyond the evaluation estimate, potentially exceeding available memory and reducing distributed-memory hit rate.If JCT increases by r times, the analysis states that working-set size expands by r^2 times and storage cost scales as r^3.

9 RELATED WORK

Prior KV-Cache optimization largely targets individual data paths or caching tiers, while DualPath instead balances storage traffic across all storage NICs and reduces DRAM dependence.

  • Distributed Memory Cache Pools: Mooncake and TokenLake use distributed cache pools, whereas DualPath targets the storage backend directly.DualPath can also use a middle DRAM cache, but the reported performance gain is marginal.
  • KV-Cache I/O Optimization: Prior I/O optimizations address hierarchical-storage bottlenecks, recomputation overlap, or layer-granular hybrid quantization.These approaches primarily optimize a single data path or reduce bandwidth requirements on that path.
  • LLM Inference System: PD-disaggregated inference separates prefill and decode across GPUs, enabling distinct parallel strategies and hardware configurations.The separation also reduces performance interference between the two stages.
  • Attention Mechanisms: Dense attention keeps computation-to-KV-Cache size proportional because both scale linearly with sequence length.The passage identifies MHA, MQA, GQA, and MLA as dense-attention variants.

10 CONCLUSION

DualPath addresses imbalanced KV-Cache reading in PD-disaggregated inference through dual-path loading. It achieves substantial throughput improvements in both offline inference and online serving.

  • DualPath addresses imbalanced KV-Cache reading in PD-disaggregated architectures through dual-path KV-Cache loading.
  • 1.87× is the maximum throughput improvement reported for offline inference.
  • 1.96× is the average improvement in agent runs per second for online serving.

A.1 Traffic Isolation Configuration Details

The appendix describes traffic-isolation settings, agent-task replay, experiment configuration, cache-hit accounting, and block formats supporting layerwise prefill.

  • Traffic Isolation Configuration Details: InfiniBand traffic uses four virtual lanes with weighted arbitration, while qos_high_limit is set to 240.The listed high- and low-priority arbitration weights configure the four lanes.
  • Traffic Isolation Configuration Details: RoCE configures four lossless RDMA traffic classes with PFC and proportional scheduling weights for bandwidth isolation.
  • Agent Task Structure: Agent trajectories are replayed as multi-turn sequences whose rounds append tokens and specify the next round’s generated-token count.Additional trajectories can be created by prepending a synthetic round with random tokens and one generated token.
  • Configuration Parameters: DeepSeek experiments allocate 80GB DRAM per node, Qwen 32B allocates 320GB, and all configurations use 3FS storage.Speculative decoding is disabled across settings.
  • KV-Cache Hit Length Calculation: KV-Cache hit lengths are computed within trajectories for most systems, while SGL(MC) computes them from HiCache and Mooncake Store states.
  • KV-Cache Block Formats: Layer Blocks store one layer’s KV-Cache, and concatenating them produces Full Blocks stored in distributed storage through a trie.This design avoids manual memory-layout conversion during inference.
Loading 2602.21548v2…