PaperlayerView exact revision on arXiv

Source-linked AI summary

ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding

Yuetao Chen, Peng Wang, Hong Xu

arXiv:2610.33719v1cs.ARcs.DC

TL;DR

Long contexts and concurrent requests make KV-cache capacity and repeated attention reads costly, while prior lossless methods restore the cache before attention. ZipFusion co-designs compression and attention to consume compressed values directly and reassigns freed memory. On an NVIDIA H100, it reduces attention latency relative to FlashInfer and improves 8K-context decode throughput over uncompressed SGLang.

  • Problem

    The growing KV cache limits concurrent requests, and repeated reads during attention can limit decoding speed; prior lossless compression restores the cache before attention.

    Source: §I · p. 1

  • Method

    ZipFusion co-designs the lossless format and attention kernel to reconstruct compressed cache values in registers, while its manager makes freed memory available to the scheduler.

    Source: §I · pp. 1–2

  • Results

    On an NVIDIA H100, attention latency falls by 18.5%–20.2% relative to FlashInfer, and 8K-input decode throughput improves by up to 26.3% over uncompressed SGLang.

    Source: §VI · p. 6

  • Takeaways & Limitations

    ZipFusion’s evaluated results show that lossless compressed attention can reduce attention latency while improving decode throughput within the same GPU memory budget.

    Source: §VI · p. 6

Abstract

from arXiv · show

The key-value (KV) cache places growing demands on GPU memory capacity and bandwidth as large language models serve longer contexts and more concurrent requests. Its footprint limits concurrency, while repeated KV reads during attention can make decoding memory-bandwidth bound. Prior lossless KV cache compression methods reduce transfers between devices but restore the uncompressed KV cache in GPU memory before attention. They therefore leave these repeated KV reads unchanged and require additional work and memory for decompression. We present ZipFusion, which enables attention to consume the losslessly compressed KV cache directly. ZipFusion co-designs the compression format and attention kernel to reduce the bytes read from GPU memory during attention. The kernel reconstructs the original values directly in registers using simple bit operations, avoiding a separate decompression pass and easing the memory-bandwidth bound. ZipFusion also reassigns memory freed by compression across layers to store additional KV cache. Its memory manager reserves space for ongoing generation and makes the remaining capacity available to new requests. We implement ZipFusion on top of FlashInfer and integrate it with SGLang. On an NVIDIA H100, ZipFusion achieves about 1.25x speedup over FlashInfer in decode attention. With 8K context, it improves model decode throughput by up to 26.3% over uncompressed SGLang under the same GPU memory budget. At each method's largest supported batch, it also delivers up to 2.76x the throughput of the best-performing lossless KV cache compression baseline.

I. INTRODUCTION

Longer contexts and more concurrent requests increase KV-cache memory use, while repeated attention reads can limit decoding speed. ZipFusion keeps the cache compressed for attention and uses the saved memory to admit requests.

  • Prior lossless codecs reduce transfer costs but restore the cache before attention, leaving repeated GPU-memory reads unchanged and requiring decompression work and memory.In one instrumented configuration, decompression accounts for 32.7% to 73.8% of decode time.Source: §I · p. 1
  • ZipFusion lets attention consume losslessly compressed KV-cache blocks and reconstructs needed values directly in registers, without maintaining an uncompressed GPU-memory copy.The approach co-designs the format and attention kernel, arranging compressed keys and values for their respective matrix computations and using simple bit operations.Source: §I · pp. 1–2
  • Its KV-cache manager reports memory freed by compression to the scheduler, which admits additional requests while reserving space for ongoing generation.Source: §I · p. 2
  • ZipFusion improves 8K-input decode throughput by 26.3% for Llama and 25.8% for Qwen over uncompressed SGLang at each system’s largest supported batch under the same GPU memory budget.It delivers 2.76× throughput for Llama and 2.61× for Qwen relative to the best-performing lossless KV-cache compression baseline.Source: §I · p. 2
  • The paper evaluates BF16 ZipFusion on an NVIDIA H100, integrating it with FlashInfer attention and the SGLang KV-cache manager.Across evaluated configurations, decode attention is 1.23× to 1.25× faster than FlashInfer.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

LLM inference builds a KV cache during prefill, then extends it while generating one token per request at each decoding step. Dense attention repeatedly reads stored keys and values, and serving systems organize the cache into pages.

  • Prefill processes the prompt and builds the KV cache; decoding generates one token per request at each step and extends that cache.Source: A. KV Cache and BF16 · p. 2
  • Dense attention compares the current query with stored keys, then combines stored values, so portions of the KV cache are read repeatedly after being written.Longer contexts increase both these reads and the memory occupied by the cache.Source: A. KV Cache and BF16 · p. 2
  • The paper considers BF16, a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits; lossless compression preserves every value’s bit pattern.Source: A. KV Cache and BF16 · p. 2

B. Compression for Faster Decode Attention

Lossless compression can reduce the bytes read by attention, but restoring the cache before attention leaves repeated reads uncompressed. Measurements show substantial decompression time, motivating a format designed for inexpensive in-kernel reconstruction.

  • Compression can further reduce the bytes that efficient attention kernels read, beyond techniques such as tiling and improved work partitioning.Source: B. Compression for Faster Decode… · p. 2
  • Because completed cache blocks are consumed across subsequent decoding steps, keeping them compressed for attention lets one encoding operation reduce bytes read repeatedly.Source: B. Compression for Faster Decode… · p. 2
  • The measurement uses Llama 3.1 8B with 2048 input tokens and batch size 32 on one H100; prefill and initial compression are outside timing.Each codec restores one layer into a shared buffer before FlashInfer attention.Source: B. Compression for Faster Decode… · p. 2
  • Decompression takes 5.3 to 30.8 ms per step, while the remaining instrumented time stays near 10.9 ms across three codecs.Attention reads the restored cache, so separate reconstruction adds work without reducing repeated reads.Source: B. Compression for Faster Decode… · p. 2
  • The format and attention kernel must be designed together to make reconstruction inexpensive and arrange keys and values for their respective computations.Reconstruction within attention avoids an intermediate buffer, but its instruction cost still affects latency.Source: B. Compression for Faster Decode… · p. 2

C. Compression for Higher Serving Capacity

Compression creates memory for additional requests, but the usable capacity changes as pages are compressed and running requests grow. ZipFusion combines page compression, in-register reconstruction, and cross-layer memory reassignment.

C. Compression for Higher Serving Capacity

  • The scheduler needs physical space available for further KV-cache writes to admit requests while accounting for cache growth in ongoing generation.Fixed capacity based on uncompressed page sizes cannot use space released by compression.Source: C. Compression for Higher Serving… · p. 2
  • More concurrent requests increase the cache read at each decoding step, linking higher serving capacity to efficient attention over compressed KV cache.Source: C. Compression for Higher Serving… · p. 3

A. Overview

  • ZipFusion divides each page’s keys and values into independently encoded compression blocks, with GPU HBM allocated in larger chunks that can hold multiple pages.Source: A. Overview · p. 3
  • Blocks meeting the compression condition shrink to 75% of original size; attention restores compressed values in registers, and chunks freed by one layer can be reassigned to another.This reduces memory reads without writing restored values back to GPU memory.Source: A. Overview · p. 3

B. Lossless KV Cache Format

ZipFusion compresses eligible BF16 blocks by encoding exponent offsets while preserving the original sign and fraction information. Its format reduces eligible blocks to 75% of their original size and supports exact reconstruction.

B. Lossless KV Cache Format

  • A block is compressed when all its exponents fall within [b, b + 15], where b is the minimum exponent rounded down to an even integer.Otherwise, the block remains uncompressed.Source: B. Lossless KV Cache Format · p. 3
  • Each value retains its low byte and stores a four-bit code combining its sign bit with three exponent-offset bits, totaling 12 bits per value.Two four-bit codes are packed into one byte.Source: B. Lossless KV Cache Format · p. 3
  • Eligible blocks occupy 75% of their original size, excluding metadata.Blocks that do not meet the exponent-range condition remain uncompressed.Source: B. Lossless KV Cache Format · p. 3, Figure 2 · p. 3
  • Reconstruction restores the original exponent, sign, and fraction fields exactly, with no floating-point rounding.The retained low byte supplies the lowest exponent bit and fraction bits; the stored code supplies the offset and sign.Source: B. Lossless KV Cache Format · p. 3
  • For a 16-token page with head dimension 128, each eligible 512-value block occupies 768 rather than 1024 bytes, excluding metadata.The retained low bytes are stored contiguously, followed by the packed codes.Source: B. Lossless KV Cache Format · p. 3

C. Attention over the Compressed KV Cache

ZipFusion arranges compressed keys and values to match their attention computations and reconstructs each thread’s operands in registers. The attention kernel reads compressed blocks from shared memory and uses them without a separate decompression pass or GPU-memory output buffer.

C. Attention over the Compressed KV Cache

  • Each thread reads eight retained low bytes and four code bytes, then reconstructs eight BF16 elements in registers for a Tensor Core instruction.Key and value layouts differ because they serve as different operands in QKT and PV.Source: C. Attention over the Compressed KV… · p. 4
  • Packed codes let each thread restore four high bytes per 32-bit addition, then combine them with retained low bytes to form eight BF16 values.The additions restore high bytes without carrying across byte boundaries.Source: C. Attention over the Compressed KV… · p. 4
  • Compressed blocks stay compressed while copied from HBM to shared memory, and threads reconstruct values in registers immediately before use.Values follow the same path as keys before PV; uncompressed blocks need no reconstruction.Source: C. Attention over the Compressed KV… · p. 4
  • Attention consumes compressed blocks without a separate decompression pass or an uncompressed output buffer in GPU memory.The kernel overlaps fetching metadata for the next portion of the KV cache with attention on the current portion.Source: C. Attention over the Compressed KV… · pp. 4–5

D. KV Cache Management and Scheduling

ZipFusion coordinates page compression with allocation across layers so freed memory can hold additional KV cache. Its allocator reserves space for new pages needed by ongoing generation and updates capacity as compression releases memory.

D. KV Cache Management and Scheduling

  • Pages remain uncompressed while partially filled so decoding can append values directly; a full page can be compressed after current attention reads finish.The page identifier remains unchanged if compression moves its data, and compression can finish in place when no separate destination is available.Source: D. KV Cache Management and Scheduling · p. 5
  • Each new logical KV cache page needs physical space in every layer, so the number of additional pages depends on capacity across all layers.A shared pool lets an empty chunk released in one layer be reassigned to another layer after outstanding reads finish.Source: D. KV Cache Management and Scheduling · p. 5
  • The allocator estimates additional page capacity using uncompressed page size because new pages must accept writes before compression.It updates available capacity as compression releases memory.Source: D. KV Cache Management and Scheduling · p. 5

A. Experimental Setup

ZipFusion reduces decode-attention latency versus production FlashInfer across the tested models, contexts, and batch sizes. The evaluation uses fixed 8K-input workloads for framework throughput and separate attention timings across two context lengths and three batch sizes.

A. Experimental Setup

  • Framework experiments use 8192 input tokens and 256 output tokens per request, timing 255 decode steps after the first token and excluding prefill and initial compression.Comparisons use the largest supported batch under the same GPU memory budget; newly generated KV cache remains uncompressed for all methods.Source: A. Experimental Setup · p. 5
  • The evaluation uses Llama 3.1 8B Instruct and Qwen3-8B on one NVIDIA H100, with BF16 weights and KV cache.ZipFusion is integrated into SGLang with 16-token pages, and framework comparisons enable CUDA Graphs.Source: A. Experimental Setup · p. 5

B. Attention Kernel Performance

  • Attention benchmarks compare ZipFusion with production FlashInfer at 2048- and 8192-token contexts and batch sizes 32, 64, and 96.Measurements use captured queries and KV cache, clear L2 before timed calls, and include attention, fused reconstruction, and split-result merge.Source: B. Attention Kernel Performance · p. 5
  • 1.23–1.25× speedup over production FlashInfer, corresponding to 18.5%–20.2% lower attention latency, holds across both models, both context lengths, and all three batch sizes.At 8192 tokens and batch size 64, Llama attention falls from 704.6 to 562.7 µs and Qwen from 704.8 to 562.5 µs; timings include reconstruction.Source: B. Attention Kernel Performance · p. 5, Figure 4 · p. 5

C. Framework Decode Throughput

ZipFusion improves decode throughput over uncompressed SGLang and SplitZip at fixed batch size, with further gains at each method’s largest supported batch under the same memory budget.

C. Framework Decode Throughput

  • At batch size 32, ZipFusion reaches 2044 tokens/s for Llama and 1882 tokens/s for Qwen, 15.3% and 15.2% above uncompressed SGLang.ZipFusion achieves the highest throughput among evaluated methods at this batch size.Source: C. Framework Decode Throughput · p. 5
  • At batch size 32, ZipFusion delivers 2.51× throughput for Llama and 2.57× for Qwen relative to SplitZip, with the same request count per batch.SplitZip is the fastest compressed baseline at this batch size.Source: C. Framework Decode Throughput · p. 5
  • At each method’s largest batch, ZipFusion improves throughput over SGLang by 26.3% for Llama and 25.8% for Qwen under the same memory budget.Supported batches rise from 46 to 57 for Llama and from 40 to 50 for Qwen.Source: Figure 5 · p. 6, C. Framework Decode Throughput · p. 6
  • Register reconstruction within attention avoids a separate decompression pass and reads less KV cache data from GPU memory.Source: C. Framework Decode Throughput · pp. 5–6

V. RELATED WORK

Prior lossless KV-cache methods reduce transfers but reconstruct before attention, while ZipFusion keeps completed pages compressed during attention. The conclusion reports faster decoding and larger batches within the evaluated memory budget.

V. RELATED WORK

  • Unlike IBP and SplitZip, which reconstruct the KV cache before attention, ZipFusion keeps completed pages compressed during attention, reducing footprint and repeated GPU-memory reads.Source: §V · p. 6
  • ZipFusion targets the growing KV cache rather than weight decompression, demonstrating benefits on H100 with HBM3; for 8192-token inputs, throughput gains reach 15.3% at batch 32 and 26.3% at each system’s largest batch.Source: §V · p. 6
  • ZipFusion links memory reclaimed by compression to request admission while reserving space for ongoing generation.Source: §V · p. 6

VI. CONCLUSION

  • With 8K contexts and precompressed prefixes, ZipFusion improves decode throughput over uncompressed SGLang by up to 26.3% at each system’s largest batch under the same budget.The conclusion also reports 18.5% to 20.2% lower attention latency than FlashInfer on an NVIDIA H100.Source: §VI · p. 6

Related papers

Loading 2610.33719v1…