PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang

arXiv:2310.07240v6cs.NIcs.LG

TL;DR

Reusing KV caches can reduce long-context processing delay, but fetching their large tensors over fluctuating networks remains costly. CacheGen compresses and adaptively streams KV caches, achieving 3.2-3.7× faster context transmission and processing than quantization baselines at similar generation quality.

  • Problem

    Reusing KV caches requires fetching large caches when local GPU memory is insufficient, while fluctuating bandwidth can violate context-fetching latency objectives.

  • Method

    CacheGen encodes KV caches into compact bitstreams and dynamically adjusts encoding levels or recomputes caches during streaming to meet latency targets while preserving response quality.

  • Results

    3.2-3.7× faster context transmission and processing than the quantization baseline at similar generation quality, while using 3.5-4.3× less bandwidth.

  • Takeaways & Limitations

    CacheGen reduces overall long-context loading delays while maintaining high task performance across models and datasets.

  • Takeaways & Limitations

    If bandwidth is too low to send most chunks at a high encoding level, CacheGen's response quality can suffer.

Abstract

from arXiv · show

As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.

1 Introduction

CacheGen addresses long-context latency by compressing reusable KV caches for network transmission and adapting compression to bandwidth conditions. Across LLMs and long-context datasets, it substantially reduces context-loading delay while preserving generation quality.

  • Motivation: Long contexts delay generation because the LLM must load and process the entire context before producing a response, with processing cost growing super-linearly with context length.Processing a 3K-token context can still take 2 seconds, according to the cited prior work.
  • Motivation: When reused KV caches are stored on other machines, fetching their large high-dimensional tensors can add 100s of milliseconds to over 10 seconds of network delay.This retrieval overhead can undermine efforts that optimize computation alone.
  • Positioning: CacheGen targets transmission-time KV-cache size rather than GPU-memory size, distinguishing it from prior token-dropping and KV-quantization methods.Because transmission is the target, CacheGen can encode the cache into a compact bitstream rather than retaining tensor format.
  • CacheGen: CacheGen encodes precomputed KV caches into compact bitstreams instead of preserving tensor shapes, reducing bandwidth and context-loading delay.Its encoder uses custom quantization and arithmetic coding.
  • Evaluation: The evaluation covers three LLMs ranging from 7B to 70B and four long-context datasets containing 662 contexts of 1.4K to 16K tokens.The baselines include KV quantization, text-context loading, and state-of-the-art context compression.
  • Key findings: 3.2–3.7× faster than KV quantization at similar generation quality, and 3.1–4.7× faster than loading text contexts with less than 2% accuracy drop.These results measure the delay of transmitting and processing contexts, or time-to-first-token.

2 Background and Motivation

Long, frequently reused contexts improve LLM responses but increase prefill computation and TTFT. Reusing cached KV tensors can avoid recomputation, while loading compressed KV or text can reduce both network and computation delays.

  • Background: During prefill, each transformer layer produces key and value tensors whose collection forms the KV cache used during autoregressive generation.The KV cache contains information essential for later context use and is typically kept in GPU memory during decoding because of its large footprint.
  • Motivation: Because long contexts are often reused across different inputs, caching their KV representations can reduce TTFT by avoiding repeated prefill computation.Prior systems have explored reusing intermediate KV-cache results across requests [82].
  • Motivation: Long contexts can improve response quality by supplying domain knowledge, but they make prefill increasingly expensive and delay the first generated token.Contexts commonly include thousands of tokens or more, and prefill overhead grows superlinearly with input length.
  • Motivation: Loading context as text reduces transfer data but increases computation delay, whereas loading compressed KV or text can save both network and computation delay.Figure 2 contrasts network and attention-computation delays for alternative context-loading methods.

3 The Hidden Network Bottleneck

Reusing KV caches can reduce TTFT, but the cache must first reside in local GPU memory and may need to be fetched or moved across machines. This network delay is substantial in ordinary deployments, unlike systems assuming high-speed GPU links such as direct NVLinks [111].

  • 3 The Hidden Network Bottleneck: Reusing KV caches can reduce TTFT, but the cache must be present in local GPU memory before reuse provides its benefit [82] [156].The cache may need to be fetched from another machine in practice.
  • 3 The Hidden Network Bottleneck: Longer contexts make storing KV caches on dedicated storage servers more practical and economical than storing them on CPUs or GPUs [51] [56] [63] [138].Requests reusing a cache may also reach different GPUs, requiring the cache to move between machines.
  • 3 The Hidden Network Bottleneck: Fetching a KV cache across machines causes substantial network delay, whereas prior multi-GPU systems assume high-speed links such as direct NVLinks, making fetch delay negligible [111].Direct NVLinks can provide bandwidth of up to several hundred Gbps.

4 CacheGen: KV Cache Encoding and Streaming

CacheGen is a KV cache streamer that encodes, streams, and decodes KV caches to reduce transmission-time size and network delay. Its custom codec exploits KV-cache tensor distributions, while dynamic streaming adapts encoding to changing bandwidth.

  • 4 CacheGen: KV Cache Encoding and Streaming: CacheGen encodes KV caches into compact bitstreams, streams them over variable-throughput networks, and decodes them back into KV caches.Encoding can be performed offline.
  • 4 CacheGen: KV Cache Encoding and Streaming: Unlike prior methods targeting runtime memory or input-window constraints, CacheGen targets transmission-time KV-cache size and network delay.Prior techniques preserve large floating-point tensor shapes for direct runtime consumption, whereas CacheGen compresses transmitted representations.
  • 4 CacheGen: KV Cache Encoding and Streaming: CacheGen’s custom KV-cache codec exploits distributional properties of KV tensors to minimize bitstream size and reduce bandwidth demand and time to first token.The codec comprises an encoder and decoder.
  • 4 CacheGen: KV Cache Encoding and Streaming: Under dynamic bandwidth, CacheGen switches between encoding levels while streaming KV bitstreams.This adaptation is intended to maintain efficient transmission as network throughput changes.

5 CacheGen Design

CacheGen designs a domain-specific KV-cache encoder around token locality, layer-dependent loss sensitivity, and channel/layer structure, then adapts encoded streaming to fluctuating bandwidth. Its encoder computes token deltas, applies layer-specific quantization, and uses separate lossless arithmetic coding by layer and channel.

  • 5.1 Insights on KV Cache: 2.4-2.9× lower delta variance than original-value variance motivates encoding token deltas rather than original K/V values.Within a layer and channel, nearby tokens have more similar K/V values than distant tokens.
  • 5.1 Insights on KV Cache: LLM output quality is more sensitive to losses in shallower KV-cache layers than deeper layers, motivating heterogeneous compression across layers.The design evaluates layer-group losses through rounding and average response accuracy across 100 contexts, while the supplied passage does not provide the resulting accuracy values.
  • 5.2 KV Cache Encoder: CacheGen’s encoder computes deltas between nearby-token K/V tensors, quantizes layers at different levels, and losslessly arithmetic-codes values separately by layer and channel.These choices target the KV-cache properties identified in the design: token-wise locality, heterogeneous layer sensitivity, and greater grouping information from channels and layers than token positions.
  • 5.3 Adapting to Bandwidth: CacheGen adapts chunk encoding during streaming because bandwidth can fluctuate over transmissions lasting hundreds of milliseconds to seconds, helping meet a specified fetching SLO.A chunk can use a smaller encoding level under low bandwidth without changing the compression loss of other chunks, but quality can suffer if most chunks cannot use high encoding levels.
  • 5.3 Adapting to Bandwidth: For concurrent requests arriving within T seconds, CacheGen batches and streams them together, up to B requests, dividing each request into equal-sized chunks.For each chunk index c, it tracks the number of requests N_c containing that chunk.

6 Implementation

CacheGen is implemented in Python and CUDA on PyTorch and exposes interfaces for computing KV caches and generating with them. It integrates with LangChain and manages encoded KV chunks through storage-server operations.

  • Implementation: CacheGen uses about 2K lines of Python and 1K lines of CUDA kernel code, built on PyTorch v2.0 and CUDA 12.0.
  • Implementation: CacheGen interfaces with LLMs through calculate_kv for KV-cache computation and generate_with_kv for token generation while skipping context prefill.
  • LangChain integration: CacheGen integrates with LangChain, reusing an existing context cache or creating one before generation through its BaseLLM _generate function.
  • KV cache management: CacheGen stores encoded K/V bitstreams by context chunk and retrieves them from a storage server for transmission to the inference server.

7 Evaluation

Across three long-context LLMs and four datasets, CacheGen substantially reduces TTFT and KV-cache bandwidth versus text prefill and quantization while preserving quality. It remains effective with context-condensing methods, varying bandwidth, and adaptive loading, with minimal decoding overhead.

  • Evaluation setup: Evaluation covers 662 contexts from four datasets and fine-tuned Mistral-7B, Llama-34B, and Llama-70B models supporting contexts up to 32K.TTFT includes KV-cache loading and question prefill, while KV-cache size measures post-compression bandwidth requirements.
  • Baseline comparison: 3.1–4.7× lower TTFT than text context and 3.2–3.7× lower than default quantization across three models and four datasets at 3 Gbps.CacheGen’s KV encoder also reduces KV-cache size by 3.5–4.3× versus default quantization while achieving similar quality.
  • Context compression: 3.5–4× smaller KV caches than H2O and 3.3–4.2× smaller than LLMlingua without losing quality.CacheGen is complementary to token-level context condensation because it further compresses the resulting floating-point KV caches into compact bitstreams.
  • Bandwidth adaptation: 60% lower TTFT SLO violation at 0.5s and a reduction from 81% to 8% at 1s versus the quantization baseline across bandwidth traces.CacheGen’s adaptive approach significantly outperforms both the quantization baseline and CacheGen without adaptation over 0.1–10 Gbps traces.
  • Decoding overhead: GPU-accelerated, pipelined decoding has minimal impact on end-to-end delay despite adding decompression relative to quantization.CacheGen pipelines context-chunk decoding with context transmission to reduce this overhead.

8 Related Work

Related work accelerates LLM serving through inference optimizations and KV-cache reuse, while long-context systems address memory constraints through offloading and related techniques. CacheGen instead compresses KV caches into query-independent bitstreams, preserving context length without retraining and improving delay–quality trade-offs.

  • Faster LLM serving: LLM-serving research also explores parallelized generation, edge inference, weight quantization, reduced memory I/O, and lower self-attention complexity.CacheGen focuses specifically on reducing time to first token (TTFT).
  • Faster LLM serving: KV-cache reuse is a common way to accelerate inference without modifying LLMs, and CacheGen enables reuse across multiple queries in existing frameworks.The cited frameworks include [82].
  • Longer LLM contexts: Long-context systems address limited GPU memory by offloading parts of the attention matrices and using external memory.These approaches target the large attention matrices created by longer contexts.
  • Longer LLM contexts: CacheGen compresses KV caches into bitstreams rather than shortening contexts, avoiding query dependence and quality risks from dropping important tokens.It leverages KV-cache distributional properties for better compression rates and does not require retraining the LLM.

9 Discussion and Limitations

CacheGen is compatible with emerging smart-quantization methods and can further compress quantized KV caches using delta encoding and arithmetic coding. Future work includes incremental KV-cache streaming, initially sending low-quality caches and progressively improving quality.

  • Compatibility with other KV-cache compression work: Smart quantization methods [62] [78] [97] are complementary to CacheGen, which can apply delta encoding and arithmetic coding after quantization.This combination is demonstrated in Figure 10.
  • Incremental KV cache streaming: Future work will extend CacheGen to incrementally stream KV caches, initially sending low-quality caches and then improving quality through additional transmissions.The proposed approach is analogous to Scalable Video Coding (SVC).

10 Conclusion

CacheGen minimizes delays in fetching and processing long LLM contexts by compressing KV caches into compact bitstreams, reducing bandwidth while maintaining high task performance.

  • 10 Conclusion: CacheGen reduces overall context-fetching and processing delays while maintaining high task performance across three models and four datasets.Its encoder is tailored to compress long-context KV caches into compact bitstreams, reducing the bandwidth required for transmission.

A Text Output Examples of CacheGen

Figure 17 presents a LongChat example in which CacheGen correctly answers a question about the first topic in a long, multi-round conversation, while matching the default quantization baseline in compressed KV-cache size.

  • Text output example: CacheGen correctly answers “What is the first topic we discussed?” by identifying the first topic as the role of art in society.The example uses a long, multi-round conversation history between the LLM and user from the LongChat dataset.
  • Text output example: CacheGen has a compressed KV-cache size similar to the default quantization baseline in this example.

B CacheGen vs. more intrusive methods … D CacheGen’s improvement under various workloads

CacheGen outperforms more intrusive context and model modifications on PIQA while adapting KV-cache streaming to bandwidth and workload conditions. Its evaluation spans smaller models, token reduction, context selection, gisting, and workloads varying in GPU capacity and network bandwidth.

  • B CacheGen vs. more intrusive methods: CacheGen outperforms smaller-model, token-dropping, context-selection, and gisting baselines on PIQA while preserving the original Llama-7B model.The comparison uses PIQA because the public pre-trained gisting model supports only contexts up to 512 tokens.
  • B CacheGen vs. more intrusive methods: Figure 18 compares CacheGen with methods that modify the LLM or context, including smaller models, token dropping, context selection, and gisting.The smaller-model comparison replaces Llama-7B with Llama-3B and applies different quantization levels.
  • C.1 KV Streamer Adaptation Logic: The KV streamer adapts transmission to bandwidth by estimating network throughput and remaining time against the serving SLO.The logic exposes a pseudo-code algorithm for bandwidth-aware streaming adaptation.
  • C.1 KV Streamer Adaptation Logic: When recomputation fits within the remaining time, the streamer raises compression level as needed so queued chunks can meet the deadline.It selects the maximum suitable level based on chunk size, throughput, and remaining time before encoding and sending the current chunk.
  • C.2 Default Encoding Level: By default, CacheGen partitions LLM layers into three equally spaced groups and uses quantization bins of 0.5, 1, and 1.5.These parameters define the default encoding configuration.
  • D CacheGen’s improvement under various workloads: Figure 19 evaluates CacheGen’s improvement over the best quantization-or-text-context baseline across workloads varying in GPU available cycles and logarithmic bandwidth.Figures 11 and 12 correspond to horizontal and vertical cross-sections of Figure 19.

E Cost of storing KV cache

CacheGen may reduce the cost of storing and repeatedly serving long-context KV caches: storing an 8.5K-token Llama-13B context costs about $0.05 per month, while recomputing it costs at least $0.00085 per request. With more than 150 monthly reuses, CacheGen may also reduce inference cost, though cost-focused context-loading design remains future work.

  • Cost comparison: Over 150 monthly reuses of an 8.5K-token Llama-13B context can make CacheGen cheaper than recomputing its KV cache.Recomputation costs at least $0.00085 per input-only request, whereas storing compressed versions costs $0.05 per month on AWS.
  • Limitations: The cost analysis is only a rough estimate, and designing a context-loading system specifically for cost savings is left to future work.The paper’s primary focus remains reducing TTFT while minimally affecting generation quality.
Loading 2310.07240v6…