PaperlayerView exact revision on arXiv

Source-linked AI summary

Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention

Sergii Kozyrev, Davyd Maiboroda

arXiv:2608.23834v1cs.AI

TL;DR

Long-context LLM serving is constrained by KV-cache capacity and bandwidth; Minima-KV keeps recent and protected pages in FP8 while storing older non-anchor pages in TQ3 without evicting live-request pages. Its separate profiles report 18.3 KiB of attention KV per live token, while the materializing profile matches its dense control on 16K RULER NIAH and shows small LongBench v2 deltas through 64K.

  • Problem

    KV-cache size grows with context and other workload dimensions, while PagedAttention reduces allocator fragmentation but not the information stored per page, limiting request admission, batching, and GPU utilization at long context or high concurrency.

    Source: §1 · p. 1

  • Method

    Minima-KV retains Recent and protected Anchor pages in FP8, moves older non-anchor pages to TQ3, and combines format-specific partial attention states without deleting live-request pages.

    Source: §1 · p. 1, §9 · pp. 8–9

  • Results

    18.3 KiB/token is reported; the materializing profile matches its dense control at 16K RULER NIAH and records LongBench v2 deltas of −0.80 pp (16K), −0.60 pp (32K), and −0.40 pp (64K).

    Source: §9 · p. 9

  • Takeaways & Limitations

    The results support a mixed-format path for compressing long-context state while retaining all live-request KV pages.

    Source: §9 · pp. 8–9

    Evaluation is limited to Qwen3.6-27B on one RTX PRO 6000; other accelerators, non-hybrid models, and multi-GPU paths were not evaluated.

    Source: §8 · p. 8

Abstract

from arXiv · show

The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a retention-preserving hierarchy for mixed-format paged attention. Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3; every live-request page remains addressable. Format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax merge, enabling direct heterogeneous decode without a cache-sized dense shadow. Across separate, configuration-bound Qwen3.6-27B profiles on a single 96-GB NVIDIA RTX PRO 6000 Blackwell GPU, deployment accounting reports 18.3 KiB of attention KV per live token, corresponding to 3.50x compression relative to BF16 and 1.75x relative to FP8. A materializing quality profile matches its dense control on 16K RULER needle-in-a-haystack tasks. On the same 503-question LongBench v2 set, measured deltas are -0.80, -0.60, and -0.40 percentage points at 16K, 32K, and 64K. A separate single-pair direct-decode canary with two 59,008-token requests measures 3.625x active-KV compression and 0.9821x throughput relative to its control, routes all 16 full-attention layers without fallback, and retains no dense shadow. These results establish a practical mixed-format path for compressing long-context state without evicting live-request KV pages.

1 Introduction · 2 Background and Motivation

Minima-KV addresses long-context KV memory pressure with a retention-preserving fidelity hierarchy and mixed-format attention, keeping live-request pages addressable while reducing the precision used for older non-anchor pages. The background motivates this design by noting that cache demand scales with serving configuration and that tokens and keys versus values differ in importance and sensitivity.

1 Introduction

  • KV-cache size grows with context length, batch size, full-attention layers, KV heads, and head dimension, so long contexts and high concurrency constrain admission, batching, and GPU utilization.PagedAttention reduces allocator fragmentation and enables cache sharing, but does not reduce information stored per physical page [1].Source: §1 · p. 1
  • Uniform quantization spends equal bits on unequal tokens, eviction risks losing context that later matters, and structural compression requires runtime codesign to avoid reconstruction costs outweighing memory savings.Prior approaches include quantization [2] [4], eviction or sparse retention [6] [8] [9], and low-rank or structural representations [10] [11] [18].Source: §1 · p. 1
  • Minima-KV keeps live-request pages addressable through Recent and Anchor FP8 tiers and Stale TQ3, while mixed-format kernels merge partial attention without a cache-sized dense shadow.Its contributions also include a CUDA-graph-compatible heterogeneous decode path and ownership and instrumentation for routing, fallback, dense shadows, physical bytes, and immutable prefixes.Source: §1 · p. 1
  • Minima-KV reports Qwen3.6-27B quality evidence of equality on 16K RULER NIAH, within 1% regression on 4K RULER NIAH, and LongBench v2 through 64K.These are profile-separated results; the contribution description also notes direct heterogeneous decode and runtime instrumentation.Source: §1 · p. 1
  • The paper positions its contribution as integrating retention-preserving FP8/TQ3 side stores, direct heterogeneous decode, ownership, and instrumentation in a production-oriented paged runtime—not as the first dynamic, tiered, or mixed-precision KV method.DynamicKV, DiffKV, HqeKV, QEvict, and MosaicKV are identified as close precedents [13] [15] [16].Source: §1 · pp. 1–2

2.1 KV-cache scaling

  • A BF16 attention cache requires 2.00 GiB per 32K-token sequence and 8.00 GiB per 128K-token sequence; separate Gated DeltaNet recurrent state limits concurrency gains from attention-KV compression.For Qwen3.6-27B, 64 language layers comprise 16 groups of three Gated DeltaNet layers followed by one full-attention layer, whose path has four KV heads of dimension 256.Source: §2.1 · p. 2

2.2 Why uniform precision is suboptimal

  • Recent tokens support local coherence and reasoning, while older instructions, headings, identifiers, retrieved evidence, and attention sinks may remain globally important; older background can matter after attention shifts.This motivates allocating more bits to recent and globally important context, fewer to stale context, and avoiding irreversible deletion.Source: §2.2 · p. 2
  • Key distortion can alter attention scores and route attention to the wrong position, whereas value distortion changes the content read after routing; Minima-KV preserves this distinction in its codec and kernel interfaces.KIVI and KVQuant motivate asymmetric treatment of keys and values [2, 3], even when a deployment selects the same nominal tier for both tensors.Source: §2.2 · p. 2

3 The Minima-KV Hierarchy

Minima-KV keeps Recent and protected Anchor pages in FP8 and moves eligible old non-anchor pages to packed TQ3 while preserving logical-page addressability. It reports 18.3 KiB of attention KV per live token and computes mixed-format attention with a shared softmax normalization.

3.1 Paged state and lifecycle

  • At retier checkpoints, the controller can combine page age, structural features, and globally normalized attention mass to allocate Anchors, but evaluated profiles disabled attention scoring.The reference collector uses a retained dense view, and the direct FP8/TQ3 kernel has not been validated as its score source; no scoring-efficacy or scoring-overhead claim is made.Source: §3.1 · p. 2
  • Recent and Anchor pages use FP8, while eligible old non-anchor pages use packed TQ3; promotion reconstructs FP8 and every live logical page remains addressable.Recent pages occupy a sliding window; protected Anchors can include instructions, attention sinks, retrieved evidence, structural pages, or high-attention pages.Source: Figure 2 · p. 3, §3.1 · p. 3

3.2 Effective rate and realized compression

  • 18.3 KiB per live token corresponds to 3.50× BF16 compression and 1.75× FP8 compression in deployment accounting.The aggregate is an owner-reported workload figure; it does not specify tier fractions, metadata, scratch, or context-dependent occupancy.Source: §3.2 · p. 3

3.3 Mixed-format paged attention

  • Format-specific kernels compute partial attention states and merge them with one stable online-softmax normalization, equivalent to concatenated-cache attention up to codec error and finite precision.TQ3 tiles unpack while loading, and page tables preserve logical order and the causal mask; kernels are authored in TileLang [21] and compiled to CUDA.Source: Figure 1 · p. 3, §3.3 · p. 4

4 Implementation

The implementation indexes FP8 and TQ3 pages separately while preserving logical-page order, merges format-specific attention states under one softmax normalization, and supports independently owned reusable prefix pages. Attention-score feedback and asynchronous conversion overlap remain outside the validated claims.

4.1 Physical stores and decode dispatch

  • The 1,792-token-page decode path scans FP8 and TQ3 stores separately, then merges normalized partial outputs under one softmax normalization.Compact per-request format-specific page-index arrays preserve logical order; graph capture is restricted to homogeneous supported cohorts with live TQ3 metadata, scoring off, and no dense shadow.Source: §4.1 · p. 4

4.2 Attention-score feedback

  • Attention-score feedback is unvalidated: the reference collector reduces retained-dense-view summaries with Eq. 9, while mixed-format reduction should rescale local page mass using Eq. 8.Both evaluated profiles and the graph route keep attention scoring off, so controller behavior is outside the measured claims.Source: §4.2 · p. 4

4.3 Evaluated profiles

  • Retiering batches eligible layer pages through normalize–rotate–pack before releasing dense ownership, but conversion throughput, queue lag, peak scratch, and serving-work overlap were not measured.The records therefore do not establish asynchronous overlap.Source: §4.3 · p. 4

4.4 Memory safety and prefix reuse

  • Prefix reuse binds sealed-block hashes to immutable format-specific pages, which new requests reference directly while allocating only a private Recent FP8 suffix.Reference counts cover requests, cached hashes, and in-flight retiering; storage returns to a per-kind free list only after all references and conversion events clear. Validation used 832-token blocks, distinct from the 1,792-token quality/direct profiles.Source: §4.4 · pp. 4–5

5 Experimental Methodology

The methodology evaluates a three-tier Recent/Anchor/Stale cache, keeping Recent and Anchor pages in FP8 and storing Stale pages in TQ3. It distinguishes quality and performance profiles, controls, and evidence classes.

5.1 Hardware, model, and runtime

  • Quality and direct-decode experiments use one 96-GB RTX PRO 6000 Blackwell Server Edition with Qwen3.6-27B, excluding the vision encoder and using tensor parallelism one.The evidence package does not bind immutable model or tokenizer revisions, engine commit, or CUDA, driver, PyTorch, and TileLang versions; exact commands remain a publication gate.Source: §5.1 · p. 5

5.2 Baselines and causal attribution

  • The controls distinguish BF16 (64 KiB/token), FP8 (32 KiB/token), the implemented three-tier policy, and an 18.3 KiB/token deployment aggregate.The direct-canary dense control's KV dtype is unbound, and full-stack deployment changes weight precision and model-specific kernels, so their throughput or TTFT deltas are not attributed to KV compression.Source: §5.2 · p. 5

5.3 Quality evaluation

  • Quality evaluation covers eight RULER NIAH tasks at 4K–16K and 503 paired LongBench v2 questions at 16K, 32K, and 64K using materialized TQ3 pages.The RULER subset is not generalized to all 13 tasks; LongBench records lack paired-bootstrap intervals, a predeclared non-inferiority margin, and one fully specified truncation rule across lengths.Source: §5.3 · pp. 5–6

5.4 Performance evaluation

  • Performance evidence is one decode-only pair with two active requests, 59,008 prompt tokens each, and a static FP8/TQ3 mix using direct fused decode and CUDA-graph replay.The compressed arm has no dense shadow, dynamic retiering, or attention scoring; the canary has no repeated-run dispersion, confidence interval, TTFT, or inter-token latency percentiles, so its point estimate is not interpreted as statistical equivalence.Source: §5.4 · p. 6

5.5 Evidence classes

  • The evidence classes separate configuration-linked Qwen3.6 observations, the 18.3 KiB/token owner aggregate, warm-prefix evidence, analytical capacities, and external full-stack results.Projected review-plan targets are excluded from result tables.Source: §5.5 · p. 6

6 Results

Minima-KV reports lower attention-KV memory, small but length-dependent LongBench v2 regressions, and a direct mixed-format decode canary with compressed active KV and near-control throughput. Its concurrency and service-level implications remain analytical or unmeasured rather than guarantees of simultaneous decode capacity or tail-latency improvement.

6.1 Attention-KV memory

  • At one million live tokens, the 18.3-KiB/token scenario requires 17.45 GiB versus FP8’s 30.52 GiB, an analytical projection rather than a context-specific measurement.The aggregate uses 71.4% fewer bytes than BF16 and 42.8% fewer than FP8; Table 2 excludes allocator reserve, metadata not included in the aggregate, scratch, and recurrent state.Source: §6.1 · p. 6, Table 2 · p. 6

6.2 Long-context quality

  • -0.80, -0.60, and -0.40 percentage-point LongBench v2 deltas at 16K, 32K, and 64K accompany exact 16K RULER NIAH equality.The paired profile also has a small positive 8K task-macro delta and a 4K regression. No confidence intervals or predeclared parity margins are available.Source: Table 3 · p. 6, §6.2 · p. 6
  • 249/256 versus 240/256 effective correct at 4K favors the dense control; at capped 16K LongBench v2, six predictions change, with five losses and one gain.At 4K, losses concentrate in niah_multikey_3 (0.8125 →0.5625) and niah_multiquery (0.96875 →0.9375). Process peaks of 92,024 versus 60,704 MiB for NIAH and 92,076 versus 61,290 MiB for LongBench are not active-request KV measurements.Source: §6.2 · p. 6

6.3 Fused-decode performance and quality canary

  • 3.6251× active-KV compression and 0.9821× within-pair throughput accompany a direct route covering all 16 physical full-attention layers with no fallback or dense shadow.Two 59,008-token requests yield 29.804 versus 29.270 counted tok/s; the control KV dtype is unbound. Separate fast-arithmetic settings fail the 16K RULER NIAH canary, so speed and quality claims do not describe one profile.Source: §6.3 · p. 7, Table 4 · p. 7

6.4 Concurrency capacity

  • At 32K, the historical analytical scenario estimates 84 resident contexts at an exact 3.50×-BF16 KV rate versus 53 with FP8, a 1.58× gain.These are derived memory ceilings, not simultaneous-decode SLO guarantees; the separate report observes 98 admitted 32K context equivalents while the analytical prediction is 97, and neither means unchanged inter-token latency.Source: §6.4 · p. 7, Table 5 · p. 7

6.5 Service-level implications

  • The available data do not include an open-loop offered-load experiment, so they support no KV-only p95/p99 tail-latency claim.Such a claim requires an offered-load sweep reporting admission, queueing, and tail latency separately.Source: §6.5 · p. 7

7 Related Work

Related work spans low-bit and mixed-fidelity caches, eviction, dynamic tiering, structural compression, and placement hierarchies. Minima-KV is distinguished by retaining approximate representations for logical pages and directly decoding its FP8/TQ3 page lifecycle in a paged runtime.

Low-bit KV quantization

  • KIVI [2], KVQuant, ZipCache [4], MiKV [5], and TurboQuant [20] establish low-bit and mixed-fidelity KV prior art; Minima-KV couples TQ3 pages to retained FP8 ownership and direct mixed-format paged decode.Minima-KV differs by combining a retained FP8/TQ3 page lifecycle with direct mixed-format decode inside a paged serving runtime.Source: §7 · p. 8

Eviction and sparse retention

  • Scissorhands [6], H2O, SnapKV [8], and PyramidKV [9] save resources through eviction or sparse retention, but removed KV is unavailable after attention shifts; Minima-KV approximately represents every retained logical page.This preserves an approximate representation for every retained page rather than deleting KV.Source: §7 · p. 8

Dynamic and hybrid management

  • Dynamic tiering and hybrid cache strategies are established; Minima-KV's evaluated contribution is retention-preserving FP8/TQ3 page ownership with direct mixed-format implementation.DynamicKV reallocates budgets across layers and tasks; DiffKV combines key/value differentiation with dynamic sparsity, paging, and compaction [13]; HqeKV uses multiple precision levels, eviction, and reclassification; QEvict [15] has full, quantized, and deleted states; MosaicKV [16] selects segment-wise strategies with packed kernels; RDKV [17] frames allocation as rate-distortion bit allocation.Source: §7 · p. 8

Head- and layer-aware policies

  • Head- and layer-aware budget policies complement Minima-KV's page-format controller; optional EWMA attention-mass scoring can retain or promote FP8 anchors but was disabled in reported Qwen3.6 profiles.DuoAttention separates retrieval heads from streaming heads [22], while PyramidKV [9] and DynamicKV [12] vary budgets across layers or tasks.Source: §7 · p. 8

Structural compression and offload

  • Palu [10], ShadowKV [11], JoLT [18], and GEAR [23] use structural compression or residual-correction approaches, distinct from Minima-KV's evaluated three-tier FP8/TQ3 system.These approaches include low-rank latent states, GPU key retention with value offload, Tucker compression with rotated residuals, and low-bit quantization with low-rank and sparse corrections.Source: §7 · p. 8
  • LMCache and Mooncake [24] [25] tier by resource placement, unlike Minima-KV's on-GPU fidelity tiers; external cache layers can move either FP8 or TQ3 pages.The placement hierarchies span GPU memory, host DRAM, storage, and network resources, so they can compose with Minima-KV.Source: §7 · p. 8

8 Limitations •

The evaluation is limited to one model and GPU, and its compression, quality, and serving results come from separate profiles with incomplete accounting and uncertainty. Several mechanisms and equal-concurrency latency effects remain unevaluated.

Quality coverage

  • Quality regresses by 0.90 at 4K RULER NIAH and by 0.80, 0.60, and 0.40 percentage points on LongBench v2 at 16K, 32K, and 64K.The 32K and 64K records report aggregate paired deltas rather than absolute arm counts, and no paired confidence interval is available.Source: §8 · p. 8

Evaluation scope

  • Results cover Qwen3.6-27B on one RTX PRO 6000; other accelerators, non-hybrid models, tensor-parallel, multi-GPU, and vision paths are unevaluated.Only 16 of 64 language layers hold full-attention KV, so the 18.3 KiB/token aggregate excludes Gated DeltaNet state and total process HBM.Source: §8 · p. 8

Aggregate accounting

  • The 18.3 KiB/token aggregate lacks per-tier bytes, metadata, scratch, dense-shadow accounting, context-specific occupancy distributions, and a matching immutable manifest.It therefore does not establish a constant compression rate across lengths.Source: §8 · p. 8

Profile separation

  • Quality, direct decode, deployment accounting, and prefix reuse use different configurations; the direct canary is one pair with unbound dense-control KV dtype and no repeated-run uncertainty or latency distribution.Source: §8 · p. 8

Unevaluated mechanisms and confounding

  • Attention scoring is off in both Qwen3.6 profiles, while conversion throughput, scratch, and asynchronous overlap are unmeasured.Public TPS and TTFT improvements also use NVFP4 weights and model-specific kernels, without a fixed-weight factorial ablation.Source: §8 · p. 8

Concurrency

  • Higher admission capacity does not guarantee unchanged latency at equal active concurrency.Source: §8 · p. 8

9 Future Directions and Conclusion

Minima-KV retains every live-request page in a mixed-format paged runtime, with FP8 and TQ3 attention states combined through a globally normalized online-softmax merge. Its measured three-tier system reports compression and benchmark outcomes; a fourth-tier tensor-network extension remains proposed, not measured.

9 Future Directions and Conclusion

  • Recent and Anchor pages use FP8 while Stale pages use TQ3; ownership-safe format changes retain every live-request page, and both formats execute through a globally normalized online-softmax merge.The physical-compression path releases the cache-sized dense shadow, and evidence profiles are reported within their bound configurations.Source: §9 · pp. 8–9
  • On Qwen3.6-27B, deployment accounting reports 18.3 KiB/token: 3.497× versus BF16 and 1.749× versus FP8; its materializing profile matches 16K RULER NIAH but regresses at 4K.Per-tier reconciliation remains part of the publication gate.Source: §9 · p. 9
  • LongBench v2 measured deltas were −0.80 pp at 16K, −0.60 pp at 32K, and −0.40 pp at 64K.Source: §9 · p. 9
  • The separate direct canary achieved 3.625× active-KV compression and 0.9821× within-pair throughput, routing all 16 full-attention layers with no dense shadow.Source: §9 · p. 9
  • The proposed, unmeasured fourth tier would relabel Stale as Stale1 and add cold-page Stale2 tensor-network groups.Drawing on earlier Minima tensor-factorization work [19], it would amortize a shared Tucker or tensor-train core across compatible layers and groups with key/value- and group-specific adapters; tiles would reconstruct on demand and emit merge states [18].Source: §9 · p. 9
  • Evaluating Stale2 requires a frozen configuration binding its ranks, residuals, factor and metadata bytes, conversion and scratch costs, and tile-decode route to joint quality, throughput, and prefix-correctness evaluation.Only the Recent/Anchor/Stale FP8/TQ3 runtime is measured.Source: §9 · p. 9

A Artifact Index and Publication Gate

The paper’s implementation and audit materials are not publicly released, and the paper should not be submitted until the evidence bundle is published. It reports configuration-linked evaluation information while retaining detailed manifests, telemetry, and raw logs internally.

A Artifact Index and Publication Gate

  • The paper should not be submitted until the evidence bundle is published; no public artifact URL is assigned yet.Table 7 summarizes the evidence classes used and their scope.Source: A Artifact Index and Publication Gate · p. 10, Table 7 · p. 10
  • The implementation is proprietary and unreleased; reported evaluation context includes model and tokenizer versions, hardware and software stack, benchmark protocols, and configuration-linked aggregate results.Detailed manifests, telemetry, and raw logs are retained internally for audit but are not part of a public release.Source: A Artifact Index and Publication Gate · p. 10

B Claim Ledger

The claim ledger classifies evidence for quantitative claims.

B Claim Ledger

  • Table 8 classifies the evidence supporting quantitative claims.Source: Table 8 · p. 10

C Qwen3.6 Warm-Prefix Validation · D External Full-Stack Deployment Reports

The warm-prefix run reports 4.342× physical KV compression and 1.021× paired steady throughput against dense FP8 APC. Separate self-published full-stack reports offer deployment context, but their combined changes and unmatched operating points do not isolate KV compression effects.

C Qwen3.6 Warm-Prefix Validation

  • The warm-prefix coordinate uses Qwen3.6-27B, 16 target full-attention layers plus MTP attention, 832-token blocks, and FP8/TQ3 storage.It is N=32 with 32,256 prompt tokens, 80 requested output tokens, replay-one, and batched bound-prefix prefill; it differs from the 1,792-token quality/direct coordinates.Source: C Qwen3.6 Warm-Prefix Validation · p. 10
  • 4.342× physical KV compression and 1.021× paired steady-throughput ratio are reported for r14’s warm-prefix run against dense FP8 APC.The final run uses the shared-page TileLang cohort for three-row MTP calls; one-row cohort mode remains opt-in.Source: C Qwen3.6 Warm-Prefix Validation · p. 11, Table 9 · p. 11

D External Full-Stack Deployment Reports

  • Rafay compares FP8 weights and FP8 KV with an NVFP4 W4A4 stack that also enables model-specific kernels and Minima-KV.The report provides deployment context, not a causal evaluation of KV compression.Source: D External Full-Stack Deployment Reports · p. 11
  • Qdrant compares a BF16 full stack with a full Minima stack, so the report does not isolate the effect of KV compression.Weight, kernel, scheduler, and KV changes are combined in the self-published deployment evidence.Source: D External Full-Stack Deployment Reports · p. 11, Table 10 · p. 11
  • With W = 15.72 GiB, the historical capacity equation predicts 97 contexts rather than Rafay’s observed 98, making the fitted envelope approximate.The studies do not expose matching memory envelopes and block reservations.Source: D External Full-Stack Deployment Reports · p. 11
  • Qdrant’s BF16 row admits 11 sessions, while substituting its nominal weights into the Rafay envelope predicts roughly 17; these are incomparable operating points.The studies lack matching memory envelopes and block reservations.Source: D External Full-Stack Deployment Reports · p. 11

Related papers

Loading 2608.23834v1…