Source-linked AI summary
Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter
Ruoyu Qin, Weiran He, Yaoyu Wang, Zheming Li, Xinran Xu, Yongwei Wu, Weimin Zheng, Mingxing Zhang
TL;DR
Conventional PD serving is constrained by KVCache transfer, which limits heterogeneous deployments to tightly coupled network domains. PrfaaS selectively offloads long-context prefill to standalone clusters and transfers KVCache over commodity Ethernet, combining model-side KV reduction with system-side scheduling and cache management. In a case study, it improves throughput and latency over homogeneous PD while retaining modest bandwidth use.
Problem
KVCache transfer keeps conventional PD serving within tightly coupled high-bandwidth domains, limiting heterogeneous deployment and resource flexibility.
Method
PrfaaS selectively routes long uncached prefills to standalone compute-dense clusters, then transfers their KVCache to local PD decode clusters using commodity Ethernet.
Results
54% higher throughput and 64% lower P90 TTFT than a 96-H20 homogeneous PD-only baseline, with approximately 15% higher throughput at equal cost.
Takeaways & Limitations
Cross-datacenter PD serving becomes practical by combining hybrid-model KVCache reduction with selective offloading and bandwidth-aware scheduling.
Takeaways & Limitations
The throughput model treats BS max and Tdecode as SLO-governed constants.
Abstract
from arXiv · showhide
Prefill-decode (PD) disaggregation has become the standard architecture for large-scale LLM serving, but in practice its deployment boundary is still determined by KVCache transfer. In conventional dense-attention models, prefill generates huge KVCache traffics that keep prefill and decode tightly coupled within a single high-bandwidth network domain, limiting heterogeneous deployment and resource elasticity. Recent hybrid-attention architectures substantially reduce KVCache size, making cross-cluster KVCache transport increasingly plausible. However, smaller KVCache alone does not make heterogeneous cross-datacenter PD serving practical: real workloads remain bursty, request lengths are highly skewed, prefix caches are unevenly distributed, and inter-cluster bandwidth fluctuates. A naive design that fully externalizes prefill can therefore still suffer from congestion, unstable queueing, and poor utilization. We present Prefill-as-a-Service (PrfaaS), a cross-datacenter serving architecture that selectively offloads long-context prefill to standalone, compute-dense prefill clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode. Rather than treating reduced KVCache as sufficient, PrfaaS combines model-side KV efficiency with system-side selective offloading, bandwidth-aware scheduling, and cache-aware request placement. This design removes the requirement that heterogeneous accelerators share the same low-latency RDMA fabric, enabling independent scaling of prefill and decode capacity across loosely coupled clusters. In a case study using an internal 1T-parameter hybrid model, a PrfaaS-augmented heterogeneous deployment achieves 54% higher serving throughput and 64% lower P90 TTFT than a homogeneous PD baseline, with approximately 15% throughput gain at equal cost, while consuming only modest cross-datacenter bandwidth.
1 Introduction
PrfaaS extends PD disaggregation across datacenters by selectively offloading long-context prefill and transferring reduced KVCache over commodity Ethernet. It combines hybrid-model KV efficiency with scheduling and cache management to support heterogeneous, independently scalable serving.
- Motivation: KVCache transfer confines conventional PD disaggregation to tightly coupled, high-bandwidth network domains, limiting heterogeneous deployment.Hybrid-attention models reduce KVCache growth, but this reduction alone does not resolve the remaining systems constraints.
- Architecture: PrfaaS dedicates standalone clusters to long-context prefill and transfers their KVCache to decode-capable PD clusters over commodity Ethernet.Short requests remain on the local PD path instead of being fully externalized.
- Scheduling: Selective offloading routes only sufficiently long requests to compute-dense prefill clusters, avoiding inefficient transfer of short requests.The design targets requests for which prefill acceleration matters most while reducing bandwidth pressure.
- Scheduling: PrfaaS combines length-based routing, bandwidth-aware scheduling, and cache management to address fluctuating links, request lengths, and prefix-cache placement.These mechanisms jointly determine when requests should remain local or be sent to remote prefill capacity.
- Evaluation: 54% higher serving throughput than homogeneous PD demonstrates the benefit of the heterogeneous PrfaaS deployment in the reported case study.The evaluation uses an internal 1T-parameter hybrid model and reports only modest cross-datacenter bandwidth consumption.
2 Background
PD disaggregation separates compute-intensive prefill from memory-bandwidth-intensive decode, but KVCache transfer keeps conventional deployments tied to tightly integrated network domains. Hybrid attention reduces KV throughput enough to make commodity-Ethernet cross-cluster serving plausible, provided bandwidth and request placement are managed selectively.
- PD disaggregation: PD disaggregation separates arithmetic-intensive prefill from memory-bandwidth-intensive decode, enabling phase-specific optimization but requiring KVCache export between nodes.The separation improves utilization, yet introduces network transport as a systems requirement.
- The bandwidth wall: Dense-attention KVCache grows with context length; MiniMax-M2.5 produces roughly 60 Gbps for a 32K-token request, exceeding typical cross-datacenter Ethernet capacity.This bandwidth demand keeps conventional PD deployments operationally tied to tightly integrated fabrics.
- Hybrid attention: Hybrid attention reduces KV throughput sharply; at 32K tokens, MiMo-V2-Flash reaches 4.66 Gbps versus 59.93 Gbps for MiniMax-M2.5, a 13× reduction.Hybrid stacks interleave a small number of full-attention layers with more efficient linear-complexity or sliding-window layers.
- Bandwidth-aware serving: Cross-cluster bandwidth depends on model KV throughput and the average uncached length of offloaded requests, which is shaped by request distribution, cache hits, and routing.Selective offloading and bandwidth-aware scheduling keep required egress within the available inter-datacenter bandwidth envelope.
- Prefill-as-a-Service: PrfaaS-PD selectively externalizes long-context prefill to standalone compute-dense clusters while decode remains in conventional PD clusters, extending heterogeneous serving beyond one RDMA domain.The architecture shifts the systems problem toward selecting worthwhile offloads and transporting their KVCache efficiently.
3 Disaggregation over Cross-Datacenter KVCache
PrfaaS-PD extends disaggregated serving across loosely connected clusters by selectively sending long prefills to dedicated clusters and transferring KVCache to local decode clusters. Its design combines heterogeneous cluster roles, hybrid cache management, threshold routing, bandwidth-aware scheduling, and throughput-based resource allocation.
- Architecture: PrfaaS clusters perform selective remote prefill, then transfer KVCache to local PD clusters for decode, while short or unsuitable requests stay local.The architecture uses commodity Ethernet between clusters and retains local PD processing for requests that do not benefit from remote prefill.
- Hybrid Prefix Cache Pool: Linear states and full-attention KVCache use separate groups backed by a shared block pool, with prefix-cache and transfer-cache blocks serving different reuse and transfer paths.Prefix-cache blocks support reuse within a cluster, whereas transfer-cache blocks hold KVCache for cross-cluster transfer and are discarded afterward.
- Hybrid Prefix Cache Pool: The global KVCache manager tracks prefix matches across clusters, chooses cache-affine placement, and can rebalance caches or transfer them when bandwidth permits.Routing jointly considers cache affinity and available inter-cluster bandwidth.
- Routing: Requests with incremental length above threshold t are routed to PrfaaS, while shorter requests use PD-P; only incremental uncached content is transferred for prefix-cache hits.The threshold avoids using bandwidth-constrained cross-cluster paths for short prefills and limits transfer demand.
- Throughput Model: The throughput model treats PrfaaS and PD-P as disjoint upstream producers feeding a single PD-D consumer, with system throughput limited by the slowest stage.PrfaaS throughput is constrained by the slower of prefill computation and egress transfer, while PD-P is compute-limited under the stated assumption.
- Dual-Timescale Scheduling: The scheduler optimizes threshold t and the PD-cluster prefill-to-decode ratio, then periodically reallocates prefill and decode nodes as traffic-driven bottlenecks shift.Long-term re-optimization uses queue depth and utilization to identify whether prefill or decode is the binding constraint.
4 Case Study: Bandwidth Demand of PrfaaS-PD Architecture
The case study evaluates PrfaaS-PD under realistic hybrid-model workloads and finds that selective long-request offloading can improve throughput and TTFT while keeping KVCache traffic modest. Grid-search optimization balances local prefill/decode capacity and identifies a 19.4K-token routing threshold.
- Evaluation Setup: The evaluation uses an internal 1T-parameter hybrid model, realistic long-context request lengths, and a 100 Gbps cross-cluster VPC link.Requests follow a truncated log-normal distribution with mean input length of approximately 27K tokens and fixed 1024-token outputs.
- Throughput Modeling: Exhaustive grid search selects a 19.4K-token routing threshold and a local PD allocation of Np = 3 and Nd = 5.At this operating point, approximately 50% of requests are offloaded to PrfaaS.
- Bandwidth Utilization: 13 Gbps of PrfaaS egress consumes only 13% of the 100 Gbps Ethernet link when 49.6% of requests are offloaded.The offloaded requests have an average input length of approximately 44K tokens, leaving substantial bandwidth headroom.
- Comparison with Homogeneous PD: PrfaaS-PD improves overall serving throughput by 54% over the homogeneous PD baseline.The configuration requires two fewer local prefill instances, freeing capacity for additional decode slots.
- Comparison with Homogeneous PD: Mean and P90 TTFT decrease by 50% and 64%, respectively, compared with homogeneous PD.Long requests avoid competing with short requests for local prefill capacity and are processed by the dedicated PrfaaS cluster.
- Comparison with Naive Heterogeneous PD: Naive heterogeneous PD reaches only 1.16× the homogeneous-baseline throughput, whereas selective routing avoids severe prefill/decode imbalance.The comparison identifies scheduling and selective offloading as necessary for effective heterogeneous deployment.
- Summary: The PrfaaS-PD configuration achieves 54% higher throughput and 64% lower P90 TTFT than a 96-H20 homogeneous baseline, with approximately 15% throughput gain at equal cost.The PrfaaS cluster uses 32 H200 GPUs and the local PD cluster uses 64 H20 GPUs.
- Summary: At larger scale, the PrfaaS cluster is currently compute-bound, and thousands of PrfaaS GPUs would require aggregate egress on the order of 1 Tbps.The passage presents this bandwidth level as within modern datacenter-fabric capacity.
5 Discussion
The discussion places cross-datacenter KVCache within a broader co-evolution of model architecture, cache management, and phase-specialized hardware. These trends reduce KVCache cost and align compute-intensive prefill with memory-bandwidth-intensive decode.
- Cross-Datacenter Serving: Cross-datacenter KVCache extends PD disaggregation from one tightly coupled cluster to loosely connected heterogeneous clusters.Its practicality depends on coordinated progress in model architecture, system design, and hardware.
- KVCache-Friendly Model Architecture: MLA, sliding-window attention, and linear attention reduce KVCache size and motivate optimizing models for KVCache transfer volume.The discussion frames KVCache transfer as an increasingly important inference cost as context windows grow.
- KVCache Compression and Reuse: KVCache compression, selective eviction, quantization, and reuse provide complementary ways to reduce memory footprints and transfer volume.The examples include H2O, KIVI, CacheGen, CacheBlend, and FusionRAG.
- Phase-Specialized Inference Hardware: Phase-specialized hardware maps compute-throughput accelerators to prefill and high-memory-bandwidth chips to decode.The discussion identifies NVIDIA Rubin CPX, LPU, and Taalas HC1 as examples of this trend.
6 Related Work
Related work has progressed from phase-disaggregated serving toward systems that optimize cost, goodput, scheduling, placement, and hardware or network heterogeneity. This work builds on those directions to address cross-cluster deployment.
- Disaggregated Serving: Disaggregated serving separates compute-intensive prefill from memory-bandwidth-intensive decode to reduce inter-phase interference and enable independent scaling.The related-work discussion presents this separation as a shift away from monolithic single-cluster engines.
- PD Disaggregation: Splitwise and DistServe study PD disaggregation through cost/power and goodput objectives with deployment, scheduling, and placement strategies.The passage reports improvements in throughput and cost under appropriate SLO and hardware constraints.
- Heterogeneous Inference: Helix, Hetis, and LLM-PQ incorporate heterogeneous GPUs and networks into disaggregated inference systems.These systems extend prior work toward hardware- and network-aware serving.
7 Conclusion
The paper proposes cross-datacenter KVCache and the PrfaaS-PD architecture to extend disaggregated serving across heterogeneous clusters connected by commodity Ethernet. It positions the approach as a basis for efficient large-scale LLM serving alongside evolving models, hardware, and networks.
- Contribution: Cross-datacenter KVCache extends disaggregated serving from single homogeneous clusters to cross-cluster heterogeneous deployments.The architecture addresses practical deployment challenges in heterogeneous disaggregated inference.
- Conclusion: PrfaaS-PD uses heterogeneous PrfaaS clusters connected via commodity Ethernet to augment serving throughput at low cost.The conclusion presents the paradigm as co-evolving with next-generation models, hardware, and networks.