MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool

1 Introduction

• Context caching reuses dependencies across requests, while disaggregated inference and sequence parallelism exploit dependencies within a request.These optimizations extend the KV cache’s lifetime across requests or its domain across instances, challenging systems designed for per-request, single-instance cache management.Source: §1 · p. 1
• Existing serving systems cannot combine inter-request and intra-request optimizations because distributed sub-requests complicate KV-cache management and reuse.In particular, disaggregated inference lacks mechanisms to return KV cache from decode to prefill for future reuse.Source: §1 · p. 1
• Current schedulers route requests by load or session ID, rather than maximizing cache reuse across sessions with shared prompt prefixes.Context caching benefits when requests sharing a prefix run on the same serving instance.Source: §1 · p. 1
• MemServe introduces MemPool, an elastic pool for managing distributed memory and KV caches, and combines context caching with disaggregated inference through its APIs.The design also uses a global scheduler with locality-aware routing based on global prompt trees to maximize KV-cache reuse.Source: §1 · pp. 1–2
• MemServe develops the combined design in stages: disaggregation, caching on prefill-only instances, caching on decode-only instances, then decode-to-prefill transfer.The system co-optimizes memory layout and network transfer using huge pages to address overhead from discrete layouts and point-to-point primitives.Source: §1 · p. 2
• On ShareGPT, MemPool-based disaggregated inference improves JCT by up to 42% versus PD-colocated; adding context caching improves JCT by a further 29%.On LooGLE, disaggregation boosts JCT by up to 10.8% versus PD-colocated, with context caching potentially improving it by a further 26.9%.Source: §1 · p. 2

Generative LLM Inference

• Prefill processes the prompt and produces the KV cache; decode uses that cache to generate tokens iteratively, and cache size grows linearly with generated-token count.The KV cache comprises key-value pairs generated in self-attention.Source: §2 · p. 2

Inter-Request Optimization

• Context caching stores and reuses KV cache across requests with common prefixes or contexts to avoid redundant computation.It requires an index to find preserved cache and a modified inference engine and attention kernel to reuse it.Source: §2 · p. 2

Intra-Request Optimization

• Disaggregated inference separates prefill and decode to reduce their interference and let each scale independently, but requires KV-cache transfer between instances.Sequence parallelism distributes self-attention outputs across instances, which also must exchange data.Source: §2 · p. 2

3 MemServe Overview

• MemServe supports prefill-only, decode-only, and PD-colocated inference instances, with each engine running on one or more AI servers depending on parallelism configuration.Source: Figure 1 · p. 3
• The overview identifies a global scheduler, multiple inference instances, and an elastic memory pool as MemServe’s three main components.The MemPool APIs provide memory allocation, index management, and distributed transfer.Source: §3 · p. 3
• The global scheduler forwards user requests to suitable instances using locality-aware policies based on distributed prompt trees to maximize KV-cache reuse.MemServe builds context caching on both regular and disaggregated inference architectures using MemPool APIs.Source: §3 · p. 3

4 Elastic Memory Pool

• MemPool manages CPU DRAM and GPU HBM, retaining both active KV cache for ongoing requests and historical KV cache after completion.Its indexing layer maps prompt tokens to historical KV, while distributed transfer exchanges data between instances.Source: §4 · p. 3

4.1 API

• Memory-block, index, and transfer APIs allocate storage, map prompts to historical KV, and exchange KV across inference instances.Allocation can target HBM-only, DRAM-only, or mixed memory; transfer flags can control on-demand allocation.Source: §4.1 · p. 3, Table 1 · p. 3

4.2 Indexing

• MemPool adopts token indexing for shared-prefix generality and extends SGLang radix trees to reference data across memory and instances.Indexing granularity follows the inference engine’s configuration; in the vLLM tests, tree nodes point to 16-token KV blocks.Source: §4.2 · p. 4

4.3 Distributed Transfer

• Transfers allocate memory at the receiver, transmit KV, and can optionally insert the received prompt and cache into the local index.transfer_with_insert avoids an extra network round-trip for mapping; a specified destination address list skips initial allocation. Transmission handles parallelism, memory, and network differences, with hardware-dependent copy paths.Source: §4.3 · p. 4

4.4 Failure Handling and Scaling

• Failure and scaling are handled at instance granularity; the cluster manager monitors health, broadcasts configuration changes, and prompts instances to release memory allocated by failed instances.The manager detects failures through regular heartbeats.Source: §4.4 · p. 5

4.5 Use Cases

• MemPool APIs support context caching, disaggregated inference, and sequence parallelism in one system, while other combinations are left for future work.The described scenarios use index APIs for caching and transfer APIs for disaggregated or sequence-parallel inference.Source: §4.5 · p. 5

5.1 Design

• PD-Basic transfers active KV cache from prefill to decode; PD-Caching-1 then retains prefill-produced cache for future reuse.PD-Caching-1 suits shared long prefixes, but in multi-turn chat it repeatedly forwards the same active cache, wasting bandwidth and affecting time-to-second-token.Source: §5.1 · p. 6
• PD-Caching-2 inserts prefill-produced KV cache at decode and retains decode-produced cache there, reducing repeated transfer from prefill.It still lacks decode-phase historical cache at prefill, so caching benefits stay flat as prompts grow in multi-turn chat.Source: §5.1 · p. 6
• PD-Caching-3 returns decode-phase KV cache to prefill instances, so preserved historical cache grows and caching benefits increase linearly with turns.This completes the progression to full-fledged context caching for disaggregated inference.Source: §5.1 · p. 6
• MemPool supplies primitives for data transfer and local-data management; inference engines choose the memory layout and number of API calls.The section identifies memory and network design as remaining challenges.Source: §5.1 · p. 6

5.2 Memory and Network Optimization

• By-request aggregation combines smaller KV blocks into one block of 2 * L smaller blocks, reducing network API calls by 2 * L; this works only for by-request transfer.By-layer transfer needs at least L calls; at low load it achieves the lowest JCT, while under high load by-layer-agg outperforms by-layer through fewer network calls.Source: §5.2 · p. 7

5.3 Cost Model for Context Caching

• The global scheduler uses predicted execution time to route requests, while the cost model also compares transferring additional cached KV cache with recomputing it.The scheduler evaluates cached ratios across prefill-related instances; transfer is chosen when its estimated time is preferable to recomputation.Source: §5.3.1 · p. 7
• The cost model predicts prefill execution time from prompt length and cached ratio using either architecture-level fitting or summed operator-level functions.It models compute-bound, memory-bound, and constant operators; memory-bound costs are fitted from read/write operations because GPU cache levels complicate throughput limits.Source: §5.3.2 · pp. 7–8
• The operator-level cost model is more accurate than the architecture-level model, which has a 20% accuracy drop, and is more scalable across parallelism configurations.The operator-level model can be adjusted by multiplying constants for tensor or pipeline parallelism, whereas the architecture-level model needs recalibration when configuration changes.Source: §5.3.2 · pp. 7–8

6 Locality-Aware Global Scheduling

• Global prompt trees track prompt prefixes and the instances storing their KV cache across prefill-only, decode-only, and colocated instances.Each tree type supports insert and match operations, with an instance pointer stored at each node.Source: §6 · p. 8, Figure 6 · p. 8
• For each request, the scheduler tokenizes the prompt, matches it against global trees, and selects an instance with the longest cached common prefix using current load information.It can also identify other instances holding additional historical KV cache; tree updates occur when instances return responses.Source: §6 · p. 8
• Prompt-tree scheduling can exploit cache reuse across sessions, unlike session-ID routing, which enables reuse only within a session.The least-load policy does not account for cache locality.Source: Table 6 · p. 8, §6 · p. 9
• The policy is best-effort because the scheduler may miss local eviction events and hold outdated prompt-tree state; a time-to-live is configured, commonly in minutes.The scheduler updates its trees only when responses pass through it.Source: §6 · p. 9

7 MemServe Implementation

• MemServe comprises MemPool, context caching with disaggregated inference, and a global scheduler; MemPool is implemented in 5K SLOC of Python and 1.6K SLOC of C++.The vLLM modifications use 200 SLOC of Python and 400 SLOC of CUDA C++; the scheduler and cluster management use 600 SLOC of Python.Source: §7 · p. 9
• MemPool exposes a Python library and uses a C++ core for transmission, with NCCL send/recv for HBM and sockets when either side uses DRAM.Because NCCL lacks destination-address specification and gather/scatter APIs, MemPool uses one thread per communicator to preserve ordering and repeats send/recv pairs for heterogeneous parallelism.Source: §7 · p. 9
• The vLLM adaptation replaces its cache engine and hash-based prefix cache with MemPool, and modifies CUDA kernels to support block aggregation.Modified kernels include paged_attention, swap_blocks, and reshape_and_cache.Source: §7 · p. 9

8.1 Setup; 8.2 Workloads

• Tests use Llama2-13B with tensor parallelism 2 on one eight-H800 server, and workloads include ShareGPT conversations, LooGLE document QA, and ReAct traces.Prompt arrivals are simulated with a Poisson distribution while preserving session request dependencies.Source: §8.1 · p. 9, §8.2 · pp. 9–10

8.3 End-to-End Applications

• Across ShareGPT, LooGLE, and ReAct, caching further improves JCT; ShareGPT gains 17% average and 29% P99 JCT, plus 58% average and 45% P99 TTFT.Figure 8 reports end-to-end results across request rates per inference instance.Source: §8.3 · p. 10, Figure 8 · p. 11

8.4 Microbenchmarks

• MemPool memory API latency rises roughly 800 ns per block, while inserting a 4K-token prompt takes at most 0.7 ms; basic context caching adds minimal overhead.Figure 9 studies API latency and Figure 10 compares MemPool’s radix index with vanilla vLLM’s hash index.Source: §8.4 · p. 10, Figure 9 · p. 11, Figure 10 · p. 11
• For 2048-token KV transfers, block aggregation outperforms the original layout; smaller blocks benefit from multiple communicators, which consume extra HBM.Figure 11 compares performance and HBM usage across NCCL settings.Source: §8.4 · p. 10, Figure 11 · p. 11
• By-request aggregation outperforms both by-layer and by-request transfer in the 1024-prompt-32-decode workload across varied request rates.The comparison is shown in Figure 12.Source: §8.4 · p. 10, Figure 12 · p. 11
• Caching benefit increases with cached ratio and prompt length, and batch size acts like prompt length; TTFT improves once cached ratio passes a threshold, even for DRAM-resident cache.Figure 13 varies prompt length, batch size, block size, and cached location; swapping historical cache from DRAM into HBM adds movement cost, which reduced computation largely offsets.Source: §8.4 · pp. 10–11, Figure 13 · p. 12
• The operator-level cost model has lower prediction error and better scalability than the architecture-level model.Figure 14 compares model accuracy and the operator-level versus architecture-level approaches.Source: Figure 14 · p. 12, §8.4 · p. 12
• With 3P1D on LooGLE, prompt-tree scheduling improves P99 TTFT by 59% over intra-session scheduling.The study attributes the comparison to maximizing KV-cache reuse; Figure 15 plots results against share ratio.Source: Figure 15 · p. 12, §8.4 · p. 12

• MemServe proposes a standalone MemPool module and uses its APIs to build a holistic serving system.Source: §9 · p. 12

Disaggregated Inference

• MemServe first abstracts MemPool, then builds disaggregated inference as a use case, rather than modifying the inference engine ad hoc.Prior work includes Splitwise [23], TetriServe [12], DistServe, Dejavu, and LoongServe [29].Source: §9 · p. 12

Context Caching

• MemServe builds context caching atop disaggregated inference using MemPool APIs, whereas prior context-caching work used prefill-decode-colocated setups.The passage describes this as the first context-caching solution atop disaggregated inference.Source: §9 · p. 12

Scheduling

• Prior scheduling work addresses local execution efficiency, while MuxServe [7] addresses global placement and adaptive batch scheduling.Examples include Orca, Sarathi [1] [2], and FastServe [30] at the local layer.Source: §9 · p. 12

Generic Memory Optimization

• Other memory-optimization approaches include weight quantization, paging to reduce fragmentation, and low-level algorithm or kernel optimizations.Source: §9 · p. 12

• MemServe combines context caching, disaggregated inference, and their combined configuration using MemPool APIs.Source: §10 · pp. 12–13
• MemServe's distributed MemPool manages KV caches across distributed instances.Source: §10 · pp. 12–13
• End-to-end results show large-margin improvements in JCT, TTFT, and TPOT.Source: §10 · p. 13