PaperlayerView exact revision on arXiv

Source-linked AI summary

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu

arXiv:2407.00079v4cs.DCcs.AIcs.AR

TL;DR

LLM serving must handle overloaded conditions where resources are limited and not every request can be processed while meeting latency SLOs. Mooncake addresses this with a KVCache-centric disaggregated architecture and SLO-aware scheduling, achieving higher throughput and request capacity in long-context and real workloads.

  • Problem

    Existing LLM-serving research generally assumes sufficient resources and processes all requests, leaving overload-oriented scheduling under latency SLOs insufficiently explored.

  • Method

    Mooncake separates prefill and decoding resource pools, uses KVCache-centric scheduling based on predicted TTFT and TBT SLO satisfaction, and rejects requests early when future decoding capacity is unavailable.

  • Results

    Compared to the baseline method, Mooncake achieves up to a 525% increase in throughput while meeting SLOs and enables Kimi to handle 75% more requests under real workloads.

  • Takeaways & Limitations

    Mooncake is designed for efficient LLM serving in long-context and overloaded scenarios while balancing effective throughput against latency-related SLO requirements.

  • Takeaways & Limitations

    In current workloads, at most 50% of KVCache can theoretically be reused under unlimited storage capacity and TTFT SLO, although reuse can reach 90% in certain scenarios.

Abstract

from arXiv · show

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.

1 Introduction

Mooncake addresses diversified LLM workloads by combining KVCache-centric disaggregation with scheduling that balances effective throughput and latency SLOs. Its design also targets overloaded deployments through early rejection and load prediction, with strong reported results on long-context workloads.

  • Architecture: Mooncake separates prefill and decoding resources and restructures GPU-cluster resources into collaborative disaggregated pools.The architecture is motivated by the distinct computational characteristics of prefill and decoding and by the need to use available cluster resources efficiently.
  • Scheduling: KVCache-centric scheduling balances cache reuse and batch size against TTFT and TBT constraints.Remote cache reuse can prolong TTFT, while larger decoding batches can increase TBT.
  • Overload handling: Overload-oriented scheduling predicts future load and rejects requests early when later decoding capacity is unavailable.This policy aims to avoid wasted prefill computation, while straightforward rejection can itself cause load fluctuations.
  • Results: 525% higher throughput was achieved versus the baseline while meeting SLOs in some simulated scenarios.Under real workloads, Mooncake enabled Kimi to handle 75% more requests.
  • Results: Mooncake is presented as a platform that has handled exponential workload growth and highly overloaded workloads for Kimi.The evaluation uses replayed real-workload traces with a dummy model following the LLaMA2-70B architecture.

2 Preliminary and Problem Definition

LLM inference divides into prefill and autoregressive decoding, whose different computational behaviors motivate distinct latency SLOs. Under constrained resources, Mooncake treats request rejection as part of maximizing completed-request throughput while respecting those SLOs.

  • Inference stages: Prefill processes input tokens in parallel and produces the first token while storing intermediate keys and values as KVCache.Its attention computation grows quadratically with input length, making prefill time generally increase superlinearly.
  • Inference stages: Decoding generates one token per request at a time, making it memory-constrained and sensitive to batch size.Continuous batching adds newly arrived requests and removes completed requests between iterations.
  • SLOs: TTFT measures arrival-to-first-token latency, whereas TBT measures latency between successive generated tokens.These metrics correspond primarily to prefill and decoding, respectively.
  • SLOs: TTFTP 90 = 10× and TBTP 90 = 5× are used in the paper’s end-to-end experiment.The paper also describes fixed TTFT and TBT SLOs for real deployments.
  • Overload problem: Under constrained GPU supply, Mooncake counts only fully completed requests toward goodput and must decide which requests to reject.The objective is to maximize throughput while adhering to SLOs.

3 Overview of Mooncake’s Disaggregated Architecture

Mooncake disaggregates prefill, decoding, and KVCache resources, using CPU, DRAM, SSD, and RDMA resources to expand cache capacity and transfer bandwidth. Conductor coordinates cache reuse, incremental prefill, KVCache transfer, and SLO-aware decoding.

  • Resource architecture: Mooncake separates prefill and decoding nodes while pooling CPU, DRAM, SSD, and RDMA resources into a disaggregated KVCache.This uses underutilized resources to provide near-GPU prefix caching without additional costs.
  • Global scheduling: Conductor dispatches requests according to KVCache distribution and workload, and may replicate or swap blocks for future inference.It selects prefill and decoding instances for each request.
  • Request workflow: Prefix-cache reuse loads reusable blocks from remote CPU memory before incremental prefill begins.When uncached input exceeds prefill_chunk, prefill is split into pipelined chunks.
  • Request workflow: KVCache transfer overlaps incremental prefill by streaming layer-generated cache to the decoding node’s CPU memory.The overlap is intended to reduce waiting time and transmission overhead.
  • Request workflow: Decoding joins continuous batching after cache receipt, while Conductor and the local scheduler check TBT-related load.A changed anticipated load can cause rejection after prefill, wasting the corresponding prefill costs.

4 Sampled Real-world Request Trace

The sampled trace preserves timing, length, and prefix-cache relationships while exhibiting long-context, highly skewed reuse patterns. Cache capacity and policy substantially affect hit rates, and hot-block concentration motivates replication to avoid congestion.

  • Dataset: 23,608 entries record timestamps, input lengths, output lengths, and remapped block hashes for cache-reuse analysis.The dataset is sampled from one hour of online requests and prioritizes requests from the same session.
  • Dataset: Hash IDs encode prefix relationships over 512-token blocks, enabling reuse when blocks and their preceding tokens match.Identical hash IDs indicate reusable KVCache; the example shares 12 blocks totaling 6,144 tokens.
  • Workload characteristics: The trace averages 7,590 input tokens, 182 output tokens, and an input-output ratio of approximately 720.The authors characterize this as representative rather than universal across workloads.
  • Cache analysis: Increasing cache capacity from 1,000 to 50,000 blocks raises the cache hit ratio from 30% to 50%, while further increases provide minimal improvement in this trace.LRUCache performs best under the dataset’s observed temporal-utilization patterns.
  • Cache analysis: Over 50% of cache blocks remain unused while some are accessed tens of thousands of times, making hot-block replication important for avoiding transfer congestion.The popularity imbalance is shown in Figure 6.

5 Implementation of the Prefill Pool

Mooncake retains a disaggregated prefill pool and accelerates long-context requests with chunked pipeline parallelism and layer-wise KVCache transfers. This design reduces communication overhead, supports varied context lengths, and overlaps cache movement with computation.

  • Architecture: Mooncake retains separate prefill and decoding pools because long-context prefill needs different cross-node parallelism and can save VRAM.Prefill is inlined into decoding only when forwarding without chunking does not compromise the TBT SLO.
  • Multi-node prefill: Sequence parallelism partitions input sequences across nodes but requires at least one cross-node communication step per layer.These methods improve network consumption and MFU relative to some alternatives, but SP still has lower MFU than single-node TP.
  • Multi-node prefill: CPP partitions each request into chunks processed simultaneously by different nodes, parallelizing long-context prefill and reducing TTFT.Every X prefill nodes form a pipeline group, and chunks are bounded by prefill_chunk.
  • Multi-node prefill: CPP communicates across nodes only at pipeline-stage boundaries, allowing communication to overlap computation and reducing KVCache-transfer contention.The approach also fits short and long contexts without frequent dynamic node repartitioning.
  • Layer-wise prefill: Layer-wise prefill asynchronously loads and stores KVCache around attention computation, overlapping transfers with the computation-bound prefill process.Each layer waits for its load before attention, triggers the next load, and launches storage after attention completes.
  • Layer-wise prefill: The overlap allows prefill scheduling to disregard available VRAM size provided it can contain a single request, focusing instead on KVCache distribution and DRAM.The authors report that Figure 7 demonstrates reduced latency for long-context requests.

6 KVCache-centric Scheduling

Mooncake’s Conductor performs cache-aware global scheduling by balancing prefix-cache reuse, instance load, predicted execution time, and TTFT SLOs. It extends this with heuristic hot-spot migration to balance distributed KVCache load, and outperforms alternative scheduling strategies on TTFT metrics.

  • 6.1 Prefill Global Scheduling: Mooncake selects prefill instances using prefix-cache hit length, reusable KVCache distribution, predicted prefill time, and queueing time rather than request counts alone.The scheduler may sacrifice the longest cache match when another node better balances load and TTFT SLO attainment.
  • 6.1 Prefill Global Scheduling: For each request, Conductor hashes token blocks, matches them against instance caches, estimates TTFT, and assigns the request to the instance with the shortest predicted TTFT.If the SLO cannot be achieved, it returns HTTP 429 Too Many Requests.
  • 6.2 Cache Load Balancing: Because future KVCache usage changes substantially over time, Mooncake uses heuristic-based automated hot-spot migration instead of relying on accurate long-term usage prediction.The scheme enhances cache load balancing by backing up or migrating hot KVCache blocks.
  • 6.3 Evaluation: Mooncake’s KVCache-centric scheduling outperforms random and load-balancing scheduling on average TTFT and TTFT SLO attainment rate.The experiment compares random scheduling, request-load balancing, cache-aware scheduling, and cache load balancing.

7 Overload-oriented Scheduling

Mooncake addresses overloaded disaggregated serving by rejecting requests before prefill when future decoding capacity is insufficient. Prediction-based early rejection mitigates the load fluctuations introduced by basic early rejection, using system-level forecasts suited to overload conditions.

  • 7 Overload-oriented Scheduling: Overload-oriented scheduling is needed because existing LLM-serving work often assumes all requests are processed, while inference capacity can lag rapidly increasing demand.Mooncake therefore balances processing as many requests as possible against cost and user-experience constraints.
  • 7.1 Scheduling in Overload Scenarios: Mooncake measures prefill and decoding load by comparing predicted maximum TTFT and TBT against their respective SLO constraints.This yields separate decisions for accepting prefill and proceeding with decoding.
  • 7.2 Early Rejection: Early Rejection assesses decoding load before prefill and rejects requests based on the greater load between the prefill and decoding pools.The policy reduces ineffective prefill computation when decoding capacity is unavailable.
  • 7.3 Load Fluctuation: Basic Early Rejection creates anti-phase load fluctuations between prefill and decoding instances because scheduling responds to decoding load with a time lag.The effect becomes stronger with fewer prefill machines or longer prefill stages.
  • 7.4 Early Rejection Based on Prediction: Early Rejection Based on Prediction forecasts decoding load after incoming requests’ prefill stages and uses that forecast to decide acceptance.Mooncake currently uses system-level prediction with a uniform decoding time assumption, while request-level prediction remains future work.
  • 7.4 Early Rejection Based on Prediction: Request-level prediction is difficult because each request’s output length is unknown, especially when overload makes accurate prediction necessary under scarce resources.System-level prediction instead estimates aggregate batch count or TBT status with lower precision requirements.

8 Evaluation

Mooncake is evaluated against vLLM on public, simulated, and real workloads using throughput under TTFT and TBT SLOs. It improves throughput especially for long-context and real-trace workloads, while early rejection reduces unnecessary work under overload.

  • 8 Evaluation: P90 TTFT and TBT are used as ultimate metrics across RPS rates, with higher RPS indicating improved throughput.TTFT and TBT thresholds are set by multiplying the lowest observed RPS values by factors of 10 and 5, respectively.
  • 8.1.1 Public Datasets: 20% and 40% throughput improvements are achieved by Mooncake-[3P+1D] over vLLM-[4M] on ArXiv Summarization and L-Eval, respectively, while satisfying SLOs.Prefix caching further enhances Mooncake throughput on L-Eval by reducing prefill time.
  • 8.1.2 Simulated Data: 50% to 525% throughput enhancements are achieved by Mooncake over vLLM on simulated data under the same TTFT and TBT SLO constraints.Mooncake’s two-stage disaggregation minimizes prefill impact on decoding, while vLLM processes long-context requests individually rather than in batches.
  • 8.1.3 Real Workload: Approximately 75% more requests are processed by Mooncake-[10P+10D] than vLLM-[20M] under real request traces while adhering to the SLOs.Almost 100% of requests meet the TTFT SLO on both systems, while approximately 100% meet Mooncake’s TBT SLO versus 57% for vLLM.
  • 8.2 Overloaded Scenarios: 3,589 requests are rejected with Early Rejection based on Prediction, compared with 4,183 under the baseline strategy, in the overloaded-scenario experiment.Early rejection avoids unnecessary prefill computations, and prediction further reduces rejected requests relative to Early Rejection.

9 Related Work

Mooncake builds on prior work in LLM serving, disaggregated prefill and decoding, prefix caching, and hierarchical KVCache management. It extends these directions with KVCache-centric global scheduling and explicit overload-oriented SLO management.

  • LLM Serving Systems: Production systems and research improve LLM serving through scheduling, memory management, resource optimization, continuous batching, and dynamic KVCache management.Examples include FasterTransformer, TensorRT-LLM, DeepSpeed Inference, Orca, and vLLM.
  • Disaggregated Serving: Splitwise, DistServe, and TetriInfer separate prefill and decoding or optimize resources and parallel strategies for the two stages.These works motivate and corroborate Mooncake’s disaggregated architecture.
  • Prefix Caching: Prefix caching systems reuse KVCache to reduce computational overhead or inference latency, including Prompt Cache and SGLang’s RadixAttention.These approaches target reuse across multiple requests through precomputation, storage, or structured cache management.
  • KVCache Systems: Mooncake shares design choices with AttentionStore but adds KVCache-centric global scheduling for long-context inference, where KVCache requires high capacity and efficient transfer.Mooncake is also described as not being a standalone cache system.
  • Scheduling Scope: At most 50% of KVCache can be reused in current workloads under infinite storage capacity and TTFT SLO assumptions, although reusability can reach 90% in certain scenarios.The paper emphasizes overload-oriented scheduling subject to SLOs rather than throughput-only scheduling.

10 Future Work

Future work extends Mooncake’s disaggregation across heterogeneous accelerators and improves KVCache efficiency and scheduling flexibility. The proposed directions target memory-bound decoding, cache reuse, and diverse operational conditions.

  • Heterogeneous Accelerators: Heterogeneous accelerators could provide computation-oriented and bandwidth-oriented resource pools for more advanced disaggregation architectures.The paper highlights memory-oriented devices using process-in-memory or hybrid bonding as potential options for memory-bound decoding operations.
  • Heterogeneous Accelerators: Attention decoding has arithmetic intensity proportional to attention heads divided by key/value heads, and increasing batch size typically cannot raise this intensity.This motivates considering bandwidth-oriented accelerators for decoding.
  • KVCache Efficiency: KVCache compression, token selection, and cross-layer sharing can increase batch size and improve cache-hit ratios, reducing prefill costs.These algorithms are identified as an orthogonal direction that could benefit Mooncake.
  • Scheduling and Cache Management: Advanced scheduling will account for varying request priorities and TTFT/TBT SLOs, while cache management will address replication, migration, partial hits, and expiration.The paper also plans dynamic balancing of system resources under diverse operational conditions.

11 Conclusion

Mooncake is a KVCache-centric disaggregated architecture for efficient LLM serving, particularly with long contexts and overloaded workloads. Its design balances effective throughput against latency-related SLOs.

  • 11 Conclusion: Mooncake targets efficient LLM serving in long-context and overloaded scenarios through a KVCache-centric disaggregated architecture.The paper discusses the architecture’s necessity, challenges, and design choices.
  • 11 Conclusion: Mooncake balances maximizing overall effective throughput with meeting latency-related SLO requirements.This is the paper’s stated optimization focus.
Loading 2407.00079v4…