PaperlayerView exact revision on arXiv

Source-linked AI summary

Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention

Sergii Kozyrev, Davyd Maiboroda

arXiv:2608.23834v1cs.AI

TL;DR

Long-context LLM serving is constrained by KV-cache capacity and bandwidth as retained state grows with context. Minima-KV introduces a retention-preserving FP8/TQ3 hierarchy and mixed-format attention that keeps live pages addressable while merging heterogeneous partial states without a dense shadow. Across configuration-bound Qwen3.6-27B profiles, it reports compressed attention KV, matched 16K RULER quality, small LongBench v2 deltas through 64K, and a direct-decode canary with near-control throughput.

  • Problem

    KV-cache growth with long context creates a capacity and bandwidth bottleneck that limits request admission, batching, and GPU utilization.

  • Method

    Minima-KV retains Recent and Anchor pages in FP8, moves older non-anchor pages to TQ3, and combines format-specific partial attention states through a globally normalized online-softmax merge.

  • Results

    The profiles report 18.3 KiB/token, 3.497× compression relative to BF16, 1.749× relative to FP8, and a direct canary with 3.625× active-KV compression and 0.9821× throughput.

  • Takeaways & Limitations

    The evaluated system provides a practical mixed-format path for compressing long-context attention state while retaining every live-request page and routing all 16 full-attention layers without fallback.

  • Takeaways & Limitations

    Evidence is limited to Qwen3.6-27B on one RTX PRO 6000, and the direct mixed-format attention-score path and controller behavior were not validated.

Abstract

from arXiv · show

The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a retention-preserving hierarchy for mixed-format paged attention. Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3; every live-request page remains addressable. Format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax merge, enabling direct heterogeneous decode without a cache-sized dense shadow. Across separate, configuration-bound Qwen3.6-27B profiles on a single 96-GB NVIDIA RTX PRO 6000 Blackwell GPU, deployment accounting reports 18.3 KiB of attention KV per live token, corresponding to 3.50x compression relative to BF16 and 1.75x relative to FP8. A materializing quality profile matches its dense control on 16K RULER needle-in-a-haystack tasks. On the same 503-question LongBench v2 set, measured deltas are -0.80, -0.60, and -0.40 percentage points at 16K, 32K, and 64K. A separate single-pair direct-decode canary with two 59,008-token requests measures 3.625x active-KV compression and 0.9821x throughput relative to its control, routes all 16 full-attention layers without fallback, and retains no dense shadow. These results establish a practical mixed-format path for compressing long-context state without evicting live-request KV pages.

1 Introduction

Long-context decoding turns KV caching into a growing memory-capacity and bandwidth bottleneck that limits serving efficiency. Minima-KV addresses this with retention-preserving mixed-format storage that keeps important live pages addressable while reducing their representation cost.

  • Motivation: KV-cache size grows with context length, batch size, full-attention layers, KV heads, and head dimension, constraining request admission, batching, and GPU utilization.PagedAttention reduces allocator fragmentation and enables cache sharing, but does not reduce the information stored in each physical page.
  • Prior work: Prior approaches trade off uniform recoverable quantization, selective retention, and structural compression against unequal token importance, attention shifts, or reconstruction cost.These methods expose a central tension between memory savings, recoverability, and runtime overhead.
  • Minima-KV: Minima-KV uses a fidelity hierarchy with Recent and protected Anchor pages in high fidelity, while older non-anchor pages move to three-bit TQ3 without deleting live-request pages.The controller can promote repeatedly retrieved old pages toward Anchor, while completed prefixes are managed separately after their final live reference is released.
  • Minima-KV: Mixed-format paged-attention kernels combine normalized FP8 and TQ3 partial outputs through a stable global softmax merge without constructing a cache-sized dense shadow.The evaluated path uses format-specific storage and preserves logical page addressability during heterogeneous decode.
  • Accounting scope: Qwen3.6-27B uses four KV heads of dimension 256 on its full-attention path, while allocated memory also includes scales, metadata, padding, reserves, and temporary workspaces.The paper distinguishes attention KV from the separate Gated DeltaNet recurrent state when accounting for memory.
  • Design rationale: Minima-KV preserves separate key and value error semantics in its codec and kernel interfaces, even when both tensors use the same nominal tier.Key distortion affects attention routing, whereas value distortion affects the content read after routing.

3 The Minima-KV Hierarchy

Minima-KV organizes paged KV state into Recent, Anchor, and Stale tiers, combining FP8 and TQ3 pages while keeping logical pages addressable. Its controller and kernels support globally normalized mixed-format attention and an effective-rate accounting model.

  • Tiered lifecycle: The controller can use page age, attention statistics, structural metadata, layer/head sensitivity, K/V role, and memory pressure to manage tier assignment.Attention scoring is supported as an input, but the evaluated Qwen3.6 profiles disabled it and provide no scoring-efficacy claim.
  • Tiered lifecycle: Minima-KV assigns Recent and Anchor pages to FP8, while older non-anchor pages use packed TQ3.The lifecycle promotes stale pages by reconstructing them into FP8 before publication and retains every live-request logical page.
  • Effective rate: The effective-rate model combines 8-bit Recent and Anchor storage, the realized TQ3 rate for Stale pages, and metadata overhead.The model expresses compression relative to BF16 as CBF 16 = 16/beff.
  • Effective rate: 18.3 KiB per live token corresponds to approximately 4.58 bits per scalar, 3.50× compression relative to BF16, and 1.75× relative to FP8.This is an owner-reported workload aggregate; its tier fractions, metadata, scratch, and context-dependent occupancy are not identified.
  • Mixed-format attention: Mixed-format kernels compute per-format partial attention states and merge them with one stable global softmax normalization.The direct path unpacks TQ3 while loading, preserves logical ordering and causality, and avoids a cache-sized dense shadow.

4 Implementation

The implementation uses format-partitioned page tables, fused FP8/TQ3 decode, ownership-aware transitions, and separate evidence profiles. The measured direct path is constrained by disabled attention scoring and a homogeneous graph cohort, while prefix reuse maintains independent ownership of shared pages.

  • Runtime layout: The evaluated Qwen3.6 path uses 1,792-token pages, 24 query heads, four KV heads, and head dimension 256.The direct canary used two 59,008-token requests and represented each request with either 9 FP8 plus 24 TQ3 or 10 FP8 plus 23 TQ3 logical entries.
  • Runtime layout: Format-specific page-index arrays preserve logical order while separating FP8 and TQ3 physical storage.The fused decode path reuses each unpacked KV tile across the six query heads in its GQA group and writes normalized partial outputs with log-sum-exp.
  • Execution constraints: Graph execution is limited to homogeneous supported Qwen3.6 cache cohorts with live TQ3 metadata, disabled attention scoring, and a released dense shadow.Unsupported shapes, prefill, or inconsistent cache state fail closed to the generic path.
  • Execution constraints: Attention scoring remains outside the measured claims because the reference collector uses a retained dense view and the direct mixed-format score source is unvalidated.Both the materializing quality profile and direct canary disabled attention scoring.
  • Retiering: Conversion writes the compressed side store before releasing dense ownership, but throughput, queue lag, scratch usage, and overlap are unmeasured.Therefore the implementation makes no asynchronous-overlap claim.
  • Prefix reuse and safety: Prefix reuse gives cached FP8 or TQ3 pages independent ownership through immutable page identifiers, format tags, and reference counts.Only an incomplete suffix is request-private, while physical storage is freed after the last reference and any conversion event clear.
  • Evidence profiles: Table 1 separates evidence-bearing profiles, so no single profile establishes all reported properties.The implementation evidence spans direct decode, quality evaluation, graph behavior, and ownership validation.

5 Experimental Methodology

The evaluation uses configuration-bound Qwen3.6-27B profiles on one RTX PRO 6000 GPU, separating theoretical formats, controls, quality measurements, and direct-decode evidence. The reported profiles disable attention scoring, and the direct canary is a single pair rather than a repeated serving matrix.

  • Configurations: The three-tier policy uses FP8 for Recent and Anchor pages and TQ3 for Stale pages.Claims concern this hierarchy unless explicitly labeled analytical or external full-stack evidence.
  • Configurations: The experiments run Qwen3.6-27B on one 96-GB NVIDIA RTX PRO 6000 Blackwell GPU with tensor parallelism one.The evidence package does not bind immutable model, engine, CUDA, driver, PyTorch, or TileLang revisions.
  • Profiles: The quality profile enables dynamic retiering but disables attention scoring, while the direct canary uses a static page mix and disables both.The dense control’s KV dtype is not bound, so its throughput ratio is not labeled a BF16 or FP8 comparison.
  • Quality evaluation: The quality evaluation covers eight RULER NIAH tasks at 4K, 8K, and 16K plus 503 paired LongBench v2 questions at 16K, 32K, and 64K.The RULER subset is not generalized to all 13 RULER tasks.
  • Performance evaluation: The direct performance evidence is a single decode-only pair with two 59,008-token prompts and 161 counted decode tokens over 127 scheduler steps.It has no run-to-run dispersion, confidence interval, TTFT, or inter-token-latency percentiles.
  • Configurations: Prefix-cache measurements come from a separate Qwen3.6-27B FP8/TQ3 profile using 832-token blocks.This warm-prefix implementation validation is distinct from the 1,792-token quality and direct profiles.

6 Results

Minima-KV reports lower analytical attention-KV storage, near-control quality on the measured long-context tasks, and direct mixed-format decoding without fallback. The evidence remains configuration-specific, and several capacity and performance quantities are analytical or single-pair measurements rather than broad serving guarantees.

  • 6.1 Attention-KV memory: At 1 million live tokens, FP8 and the 18.3-KiB scenario require 30.52 and 17.45 GiB, respectively.These context rows are constant-rate analytical projections, not additional measurements.
  • 6.2 Long-context quality: At 4K RULER NIAH, the aggregate corresponds to 249/256 effective correct for dense control and 240/256 for Minima-KV.Losses concentrate in niah_multikey_3 and niah_multiquery; the 16K LongBench row has six changed predictions.
  • 6.3 Fused-decode performance and quality canary: The direct canary measures 29.804 versus 29.270 counted tok/s for dense control and compressed wall times of 5.401996 versus 5.500437 seconds.Both arms count 161 decode tokens over 127 scheduler steps.
  • 6.3 Fused-decode performance and quality canary: The fast arithmetic settings score 1.0, 0.875, 0.828125, and 0.78125 on four reported 16K RULER tasks, with five other tasks at 1.0.Because this quality run uses eager execution, the speed and quality claims cannot be combined into one profile.
  • 6.4 Concurrency capacity: At 32K, the historical analytical scenario predicts 53 resident FP8 contexts and 84 exact 3.50×-BF16 KV contexts.Resident capacity is distinct from active decode concurrency and does not guarantee unchanged latency.

7 Related Work

Prior KV-cache methods reduce storage through quantization, selective retention, dynamic policies, structural representations, or offload. Minima-KV instead combines retention-preserving page formats with direct heterogeneous decoding, while remaining compatible with external placement hierarchies.

  • Quantization and retention: Low-bit methods reduce precision, whereas eviction methods can make removed KV unavailable after an attention shift.Minima-KV keeps an approximate representation for every retained logical page.
  • Dynamic and hybrid management: Dynamic and hybrid systems vary budgets, precision, sparsity, or promotion policies across layers, tasks, channels, or segments.These include DynamicKV, DiffKV, HqeKV, QEvict, and MosaicKV.
  • Head- and layer-aware policies: Head- and layer-aware policies are complementary to Minima-KV’s page-format controller.The optional controller can use globally normalized attention mass to retain or promote FP8 anchors, but scoring is disabled in the reported Qwen3.6 results.
  • Structural compression: Structural approaches use low-rank, sparse, rotated, Tucker, or residual representations and are distinct from the evaluated three-tier FP8/TQ3 system.The related work rules out a “first tensor decomposition” claim.
  • Offload and placement: LMCache and Mooncake move state across GPU, host, storage, and network resources, whereas Minima-KV’s tiers are on-GPU fidelity states.The two approaches can compose because FP8 or TQ3 pages can still be moved externally.

8 Limitations •

The evaluation is bounded by a single model and GPU, incomplete accounting, separated configurations, and limited quality and direct-decode evidence.

  • The study evaluates Qwen3.6-27B on one RTX PRO 6000; other accelerators, non-hybrid models, multi-GPU, tensor-parallel, and vision paths are unevaluated.
  • The reported 18.3 KiB/token aggregate covers only full-attention KV, excluding Gated DeltaNet state and total process HBM.
  • The aggregate lacks per-tier, metadata, scratch, dense-shadow, occupancy, and manifest reconciliation, so it does not establish constant compression across lengths.
  • Quality results include a 0.90 regression at 4K RULER NIAH and LongBench v2 deltas of 0.80, 0.60, and 0.40 percentage points through 64K.
  • Quality, direct decode, deployment accounting, and prefix reuse use different configurations; the direct canary is one pair with an unbound control KV dtype and no uncertainty distribution.

Conclusion

Minima-KV implements a three-tier paged-KV runtime that retains live-request pages, executes heterogeneous FP8/TQ3 attention directly, and reports configuration-bound quality, compression, and throughput evidence.

  • Minima-KV uses FP8 for Recent and Anchor pages and TQ3 for Stale pages in a retention-preserving three-tier runtime.
  • Every live-request page remains retained through ownership-safe format changes, rather than being deleted during lifecycle transitions.
  • Mixed-format kernels combine FP8 and TQ3 partial outputs through a globally normalized online-softmax merge without a cache-sized dense shadow.
  • 3.497× compression relative to BF16 and 1.749× relative to FP8 are reported from 18.3 KiB/token deployment accounting on Qwen3.6-27B.
  • The materializing profile matches its dense control at 16K RULER NIAH, while LongBench v2 deltas are −0.80 pp, −0.60 pp, and −0.40 pp at 16K, 32K, and 64K.
  • The direct canary achieves 3.625× active-KV compression and 0.9821× within-pair throughput while routing all 16 full-attention layers without a dense shadow.
  • Stale2 tensor-network groups are proposed as a future fourth tier, and every quantitative claim here refers only to the measured three-tier system.

A Artifact Index and Publication Gate

The artifact index records configuration-linked evidence and states that the proprietary implementation and supporting audit materials are not publicly released.

  • Table 7 summarizes the configuration-linked evidence used by the paper.
  • No public artifact URL is assigned, and publication is gated on publishing the listed bundle.
  • The implementation is proprietary; manifests, telemetry, and raw logs remain internal despite reported model, hardware, software, protocol, and aggregate configuration details.

B Claim Ledger

The claim ledger separates evidence classes, coordinates, and deployment comparisons, emphasizing that reported results are configuration-bound and that external full-stack studies are not directly comparable.

  • Table 8 classifies the evidence supporting the paper’s quantitative claims.
  • The warm-prefix record uses 32,256 prompt tokens, 80 requested output tokens, 832-token blocks, and FP8/TQ3 storage, distinct from the 1,792-token quality/direct coordinates.
  • Table 9 reports Qwen3.6 warm-prefix results against dense FP8 automatic prefix caching.
  • 4.342× is the physical KV compression counter for the warm-prefix coordinate, while 1.021× is its paired steady-throughput ratio.
  • The deployment reports combine weight, kernel, scheduler, and KV changes rather than isolating KV compression alone.
  • Rafay and Qdrant provide deployment context, but their full-stack comparisons use different operating points and do not expose matching memory envelopes or block reservations.
Loading 2608.23834v1…