Source-linked AI summary

ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction

Zheyu Shen, Guanhua Wang, Dezhan Tu, Mengchi Zhang, Yanjia Li, Adnan Aziz, Chunqiang Tang, Ang Li

arXiv:2609.36835v1cs.AI

TL;DR

Long-context inference faces growing KV-cache memory costs, and iterative anchor search makes OMP-based reconstruction costly. ARC-KV amortizes selection with a reusable value-aware indexer while fitting each context’s compact cache. It leads reported methods in most tested settings and substantially reduces QuALITY compaction time at 10% retention.

  • Problem

    KV caches grow with sequence length and make long-context decoding memory-bound, while iterative anchor search dominates OMP-based Attention Matching compaction cost.

    Source: §1 · pp. 1–2
  • Method

    ARC-KV trains a reusable value-aware indexer for anchor selection and retains context-specific key merging and attention-mass and compact-value fitting.

    Source: §1 · p. 2, §3.4 · p. 5
  • Results

    ARC-KV led reported compacted methods in most settings; at 10% QuALITY retention, accuracy rose from 0.6409 to 0.6474 over OMP-based Attention Matching while compaction time fell from 959.8 to 37.3 s.

    Source: §6 · p. 10
  • Takeaways & Limitations

    The approach targets reusable static context prefixes by building each compact cache once and reusing it for subsequent queries.

    Source: §6 · p. 10

    ARC-KV currently targets one-shot compaction of static reusable context prefixes; generated-state recompression and reducing reference-query construction and dense key-assignment costs remain future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task performance with compact KV caches. However, iterative anchor search dominates the compaction cost of OMP-based Attention Matching. This motivates our selective amortization principle of learning a reusable anchor-selection policy across contexts while retaining context-specific reconstruction. In this work, we propose ARC-KV, a novel reconstruction-based KV cache compaction method that follows this principle. To this end, we first train a value-aware indexer to select real-key anchors in a single scoring pass. ARC-KV then applies convex-hull-constrained key merging and fits an attention-mass bias and compact values against the full cache. At inference time, ARC-KV builds the compact cache once per context using the frozen indexer and reuses it for all subsequent queries. Extensive experiments demonstrate that ARC-KV outperforms reported compaction methods in most settings across QuALITY, RULER, and LongBench on Llama-3.1-8B-Instruct. In particular, at 10% KV retention on QuALITY, ARC-KV improves accuracy from 0.6409 to 0.6474 over Attention Matching while reducing compaction time by a factor of 25.73, from 959.8 s to 37.3 s.

1 INTRODUCTION

Long-context inference is burdened by KV-cache memory use, while reconstruction-based compaction can preserve information but OMP-based anchor search is costly. ARC-KV amortizes anchor selection across contexts while retaining context-specific reconstruction.

1 INTRODUCTION

  • A 128K-token sequence requires 320GB for the LLaMA-65B KV cache alone in FP16, degrading serving throughput and latency.The cache grows with sequence length and must be repeatedly written to and read from memory during autoregressive decoding.Source: §1 · p. 1
  • Prior cache-reconstruction methods preserve rich information in compact caches; Attention Matching uses iterative OMP anchor search and fitting, while Still synthesizes a cache in one forward pass.Attention Matching is reported to outperform rule-based eviction at 20×–100× compaction ratios.Source: §1 · p. 2
  • ARC-KV amortizes anchor selection through a reusable policy while keeping cache reconstruction specific to each context.The decomposition reflects different generalization needs: anchor selection can reuse a prior across contexts, while reconstruction fits the current cache.Source: §1 · p. 2
  • ARC-KV trains a value-aware indexer to select real-key anchors in one pass, then merges keys and fits attention-mass bias and compact values.The indexer is trained to preserve held-out attention-output fidelity using disjoint scoring, fitting, and holdout queries, without OMP-generated anchor labels.Source: §1 · p. 2
  • ARC-KV ranks first among reported compacted methods in most evaluated settings across QuALITY, RULER, and LongBench on Llama-3.1-8B-Instruct.At 10% retention on QuALITY, accuracy rises from 0.6409 with OMP-based Attention Matching to 0.6474.Source: §1 · p. 2

2 MOTIVATION AND ANALYSIS

The analysis motivates reconstruction rather than unchanged token retention, identifies iterative anchor search as a compaction bottleneck, and finds value vectors informative for token ranking.

2 MOTIVATION AND ANALYSIS

  • The analysis identifies three findings: a hard KV subset cannot retain sufficient information, iterative anchor search dominates computation, and value vectors provide a consistent signal beyond attention-only scores.These findings motivate cache reconstruction and value-aware, amortized anchor selection.Source: §2 · p. 2

2.1 CACHE RECONSTRUCTION OUTPERFORMS TOKEN EVICTION IN FIDELITY

  • At 5% retention, merging keys before fitting compact values yields the lowest local output error, 0.3200, versus 0.3546 for value fitting alone and 0.8778 for the hard subset.The comparison fixes the same anchor set across cache constructions; fitting the attention-mass bias alone leaves the local output error essentially unchanged.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 ITERATIVE ANCHOR SEARCH DOMINATES COMPACTION TIME

  • At 20% retention, OMP-fast takes 3.70 minutes at 4K tokens and 15.36 hours at 64K, while every other profiled AM stage takes at most 1.17 minutes at 64K.The profiling identifies anchor search as the stage to target for amortization.Source: §2.2 · p. 3

2.3 VALUE VECTORS MATTER IN TOKEN SELECTION

  • Value-influence ranking lowers truncated-output error by 1.05–6.23% relative to attention-only ranking across four layer–KV-head pairs at every matched keep ratio.The comparison uses disjoint held-out queries and a matched token budget.Source: §2.3 · p. 3

3 METHOD

ARC-KV constructs compact caches from real-key anchors, combining a reusable value-aware selector with context-specific merging and fitting. Its training objective targets held-out attention outputs, while inference reuses the constructed cache for future queries.

3.1 PROBLEM FORMULATION

  • ARC-KV represents a context with real-key anchors, convex-hull-constrained compact keys, an attention-mass bias, and compact values fitted against the full cache.The goal is to preserve full-cache attention mass and outputs for unseen inference queries, rather than retain selected keys and values unchanged.Source: §3.1 · pp. 3–4, §3 · p. 3

3.2 METHOD OVERVIEW

  • The context-specific pipeline scores the full cache, selects top-t real-key anchors, merges keys, and analytically fits attention-mass bias and compact values.The frozen indexer is reused across contexts, while reference-query construction, scoring, merging, and fitting are performed for each context.Source: §3.2 · p. 4

3.3 VALUE-AWARE INDEXER

  • The value-aware indexer scores cache positions from reference activations, queries, and KV pairs, extending a query–key scorer with a value-attention head.RMS aggregation yields one context-level ranking; it gives extra weight to positions important to only a subset of reference queries, and the selected set is reused for future queries.Source: §3.3 · p. 5

3.4 CONVEX-HULL-CONSTRAINED KEY MERGING

  • Profile-based grouping assigns each key to its most similar anchor, then shifts that anchor toward a group mass-weighted centroid while keeping the merged key within the group’s convex hull.The merge uses keys and reference queries, not direct averaging of values; values influence selection through the indexer and are handled in subsequent compact-value fitting.Source: §3.4 · p. 5

3.5 TRAINING THE REUSABLE VALUE-AWARE INDEXER

  • Full-attention behavior trains one frozen indexer per base model to select anchors that reconstruct held-out attention outputs after analytical fitting, without OMP-generated labels.Training uses disjoint scoring, fitting, and holdout query subsets; hard top-t selection supplies the forward value while a differentiable soft surrogate supplies gradients.Source: §3.5 · p. 6
  • At inference, the compact cache is built once per context and reused for future queries; the stated scope is one-shot compaction of reusable context prefixes.The fitted attention-mass bias and compact values are determined with merged keys fixed, and the indexer itself remains frozen.Source: §3.4 · p. 6, §3.5 · p. 7

4 EXPERIMENTS AND RESULTS

ARC-KV led the reported compacted methods in most Llama-3.1-8B-Instruct benchmark settings and was faster than reconstruction-based Attention Matching on QuALITY. Cross-model results, efficiency profiles, and ablations further characterize its performance and costs.

4.1 EXPERIMENTAL SETUP

  • The main evaluation used Llama-3.1-8B-Instruct on QuALITY, RULER, and LongBench, comparing compactors at retention ratios from 1% to 20%.Indexer training used 600 context-only articles per model, plus 60 for development.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • ARC-KV ranked first in 13 of 15 benchmark–retention settings, improving over the strongest compacted baseline across QuALITY, RULER, and LongBench.It led all five QuALITY settings and four of five on each other benchmark.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 CROSS-MODEL RESULTS

  • At 5% retention, ARC-KV ranked first among compacted methods in three of four Qwen3-8B and Gemma-3-12B-IT benchmark–model settings.It ranked second on Qwen3-8B RULER.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 EFFICIENCY AND ABLATIONS

  • At 10% retention, ARC-KV reduced QuALITY compaction time from 959.83 to 37.3 s and improved accuracy by 0.65 points over AM.The corresponding speedup was 25.73×; lightweight selectors remained faster in absolute compaction time.Source: §4.4 · p. 8, Figure 3 · p. 9
  • At 64K tokens, OMP key selection took 29.49 min while the learned indexer took 7.273 s; reference-query construction remained ARC-KV’s dominant cost.At ρ = 0.05, repeat prefill accounted for 70.8 s of ARC-KV’s 99.8 s total.Source: §4.4 · p. 9
  • At 20% retention, compact attention with fitted bias ran 3.19–4.43× faster per layer than full-cache FlashAttention-2 at 4K–32K contexts.This isolates attention speed, not end-to-end decode latency.Source: §4.4 · p. 9
  • On QuALITY, RMS aggregation beat mean and max pooling by 4.59–8.95 points at ρ ≤ 0.05, while uniform budgeting cost 1.68–16.22 points.Every nonzero λm lowered unmerged-anchor attention-mass error by 43.9–56.9% at the best setting.Source: §4.4 · p. 9
  • The indexer and OMP-fast had limited anchor-set overlap on the first 30 QuALITY articles, with mean Jaccard scores of 0.144–0.201.The range covers 1–20% retention under matched per-head budgets.Source: §4.4 · p. 9

5 RELATED WORK

Prior KV compaction methods either evict tokens or reconstruct compact caches, with different trade-offs in information retention, optimization cost, and model-specific training. ARC-KV amortizes anchor discovery while retaining per-context reconstruction.

5 RELATED WORK

  • Token-subset eviction deploys hard KV subsets without value refitting or attention-mass correction, while compensated-retention methods add model components.ARC-KV instead leaves base context encoding unchanged and fits attention-mass bias and compact values for each context.Source: §5 · p. 9
  • Cartridges optimizes each context with gradients, Attention Matching uses iterative OMP search, and Still trains a separate Perceiver for each base-model checkpoint.These methods differ in per-context optimization, anchor search, or checkpoint-specific training costs.Source: §5 · p. 9
  • ARC-KV learns one scalar score per key and amortizes discrete anchor discovery while retaining context-specific reconstruction of keys, attention mass, and values.Its offline training is described as lightweight.Source: §5 · p. 10

6 CONCLUSION

ARC-KV amortizes anchor selection while retaining context-specific reconstruction, and the paper reports leading results in most tested settings. The appendices detail the value-aware indexer, inference procedure, long-context chunking, and remaining computational costs.

6 CONCLUSION

  • ARC-KV led reported compacted methods in most Llama-3.1-8B-Instruct settings and improved QuALITY accuracy at 10% retention while reducing compaction time.It targets one-shot compaction of static reusable context prefixes; extending to generated-state recompression and reducing reference-query construction and dense assignment costs remain future work.Source: §6 · p. 10

A.1 VALUE-AWARE INDEXER ARCHITECTURE

  • The value-aware pathway enriches query embeddings with value context and cached-key embeddings with their colocated values; only indexer parameters are learned.The frozen language-model weights remain unchanged, and zero-initialized value projections initially reproduce query–key-only logits.Source: §A.1 · p. 15

A.2 INFERENCE

  • At inference, the available reference-pair pool serves anchor scoring, key merging, and attention-mass and compact-value fitting; the resulting cache is reused for future queries.The indexer replaces iterative OMP anchor search with one scoring pass.Source: §A.2 · pp. 15–16

A.3 KV-BASED CHUNKING FOR LONG CONTEXTS

  • For article spans longer than 10,000 tokens, fixed chunking selects local anchors, then performs merging and fitting once against the full article cache.Chunking is bypassed when the article span is at most 10,000 tokens.Source: §A.3 · p. 16

A.4 COMPLEXITY ANALYSIS

  • The learned scorer removes repeated OMP support selection and coefficient refitting, but dense key assignment can still dominate at larger budgets.Reference-query construction and final mass and value solves also remain.Source: §A.4 · p. 16

A.5 METHOD OVERVIEW AND REFERENCE-QUERY CONSTRUCTION

ARC-KV reuses a frozen, value-aware anchor indexer across contexts while constructing reference queries and reconstructing each context’s compact cache separately. It selects real-key anchors, merges discarded keys, and fits attention mass and compact values against the full cache.

A.5 METHOD OVERVIEW AND REFERENCE-QUERY CONSTRUCTION

  • The reference-query pool comes from a repeated-context pass after the model has observed the complete context, while the keys and values being compacted remain from the original prefill.The retained activation-query pairs therefore probe the original cache with context-conditioned queries.Source: §A.5 · p. 16

A.6 VALUE-AWARE INDEXER: DESIGN WALK-THROUGH

  • The value-aware indexer uses key and value information to score cache positions in one pass, selecting real-key anchors that are reused for future queries within that context.The selected anchors define later key groups but need not be the final compact keys.Source: §A.6 · p. 17

A.7 JOINT PRESERVATION OBJECTIVE

  • The preservation target is full-cache attention behavior, with zero added bias for an uncompressed KV block in the joint objective.The passages identify full-cache probabilities and outputs as the target and describe appending zero bias for the uncompressed block.Source: §A.7 · p. 17

A.8 KEY-MERGING SCHEMATIC AND REMARKS

  • Each compact key interpolates between its selected real-key anchor and its group’s attention-mass-weighted centroid, with λm controlling the shift.At λm = 0 the anchor remains unmerged; at λm = 1 it is replaced by the group centroid.Source: §A.8 · p. 18, Figure 4 · p. 18

A.9 ATTENTION MASS AND COMPACT VALUE FITTING

  • With compact keys fixed, context-specific fitting approximates attention mass using least squares followed by projection and fits compact values with a ridge coefficient.Together, the fits approximate the original cache’s unnormalized attention mass and attention outputs without adding learned parameters.Source: §A.9 · p. 18

A.10 INDEXER TRAINING: ADDITIONAL DERIVATIONS

The indexer is trained per frozen base model using full-attention self-supervision rather than anchor labels from iterative search. A hard-selection forward pass and differentiable soft surrogate support training toward held-out attention-output reconstruction.

A.10 INDEXER TRAINING: ADDITIONAL DERIVATIONS

  • Training uses full-attention behavior as self-supervision, without anchor labels from iterative search, and applies deterministic grouping and merging only after anchor quality is trained.A KL warm-up precedes joint training on held-out attention-output reconstruction.Source: §A.10 · pp. 18–19
  • The hard-selection branch evaluates real keys, while a soft surrogate relaxes top-t membership over a candidate set containing the selected anchors so gradients can reach the indexer.The relaxation operates on candidates from TopK(s, min{T, 2t}).Source: §A.10 · p. 19
  • The soft gate relaxes membership near the top-t boundary, and its temperature is annealed linearly from 1.0 to 0.1 during joint training.The gate is not normalized to make its memberships sum to t.Source: §A.10 · p. 19
  • For the soft branch, compact values are obtained with a differentiable ridge solve and evaluated on holdout queries using relative output error.The fitting queries and holdout queries serve separate roles in the reconstruction objective.Source: §A.10 · p. 19

B.1 ANCHOR SELECTOR DEFINITIONS

The selector comparison finds lower held-out truncated-output error for value-influence ranking than for attention-only ranking across the reported keep ratios. The section also distinguishes unchanged token retention from reconstruction and profiles iterative anchor search as the dominant Attention Matching compaction stage.

B.2 HARD-SUBSET AND RECONSTRUCTED ATTENTION OPERATORS

  • Reconstruction-based methods use selected keys as anchors to construct compact keys, an attention-mass bias, and compact values, unlike eviction methods that retain selected KV pairs unchanged.Each compact key corresponds to an anchor; β calibrates its attention mass and the corresponding compact value supplies its output value.Source: §B.2 · p. 20

B.3 ATTENTION MATCHING PIPELINE PROFILE

  • 15.36 hours at 64K tokens: OMP-fast anchor search dominates the Attention Matching profile, while each other profiled stage takes at most 1.17 minutes.At 4K tokens, OMP-fast search takes 3.70 minutes; the reported runtime comparison is at 20% retention.Source: §B.3 · p. 20, §B.4 · p. 21

B.1 ANCHOR SELECTOR DEFINITIONS

  • 1.05–6.23% lower held-out truncated-output error: VI outperforms HA at every keep ratio across four selected layer–KV-head pairs.The comparison uses matched anchor budgets and disjoint scoring and evaluation queries.Source: §B.1 · pp. 19–20, Figure 5 · p. 21

C.1 MODEL AND IMPLEMENTATION DETAILS

The implementation compacts only Gemma-3-12B-IT’s full-attention layers, leaving its sliding-window layers unchanged. ARC-KV uses frozen base-model weights and a trained value-aware indexer to build reusable per-context compact caches.

C.1 MODEL AND IMPLEMENTATION DETAILS

  • For Gemma-3-12B-IT, only the eight full-attention layers are compacted; the 40 sliding-window layers remain unchanged because they cache at most 1,024 tokens.The full-attention layers account for 44% of the KV cache at 4K tokens and 93% at 64K.Source: §C.1 · p. 21
  • Experiments use FP16 on an NVIDIA B200 GPU, with base-model weights frozen and only the value-aware indexer trained.At inference, repeat-prefill reference queries are used to build a reusable compact cache for each context.Source: §C.1 · p. 21

C.2 INDEXER TRAINING DETAILS

The indexers are trained once per model on context-only data with repeat-prefill supervision, then frozen; the accompanying setup defines the baselines and nominal retention budgets.

C.2 INDEXER TRAINING DETAILS

  • Training uses 600 context-only articles from 11 datasets, plus 60 development articles, with repeat-prefill supervision and no QA labels.The sources cover multihop reasoning, summarization, code, and question answering.Source: §C.2 · p. 21
  • A 10-epoch KL-only warm-up precedes five joint epochs; training costs 147 GPU-hours for Llama, 151 for Qwen, and 34 for Gemma, with each indexer frozen for reuse.The training schedule is shared across the three models.Source: §C.2 · pp. 21–22
  • Table 3 reports indexer parameter counts across compacted layer–KV-head pairs and ratios relative to each model’s backbone; Gemma includes only eight full-attention layers.Source: Table 3 · p. 22

C.3 BASELINES AND CACHE BUDGET

  • The baselines include full-context and no-context references plus SnapKV, Attention Matching, KV Policy, KVzip, and Expected Attention.The passages describe the distinct selection or scoring approach used by each compacted baseline.Source: §C.3 · p. 22
  • Retention is defined as ρ = t/T, with compacted methods tested at 0.20, 0.10, 0.05, 0.02, and 0.01; ARC-KV uses nominal budgets distributed by a fixed model-specific allocation.Full context is ρ = 1 and is not compacted.Source: §C.3 · p. 22

D.1 PER-BENCHMARK DISCUSSION OF THE MAIN RESULTS

ARC-KV ranked first among compacted methods in most benchmark–retention settings. The section also reports compaction and attention timing, anchor overlap, and a key-merging ablation.

D.1 PER-BENCHMARK DISCUSSION OF THE MAIN RESULTS

  • On QuALITY, ARC-KV exceeded the strongest compacted baseline by 0.23–3.63 percentage points; at ρ = 0.05, it exceeded AM by 3.63 points.From ρ = 0.05 to 0.20, its accuracy was 0.6437–0.6567 versus 0.6387 for full context.Source: §D.1 · p. 22
  • On RULER, ARC-KV beat all compacted baselines from ρ = 0.01 to 0.10 by 3.34–25.94 points; at ρ = 0.20, it scored 0.9515 versus 0.9655 for full context.At ρ = 0.05, ARC-KV scored 0.7394 versus 0.4800 for AM.Source: §D.1 · pp. 22–23
  • On LongBench, ARC-KV had the highest aggregate score from ρ = 0.02 to 0.20, but at ρ = 0.01 scored 0.2355 versus 0.2451 for AM-HA.The OMP-based AM variant was impractically slow over the 1,050 LongBench instances, so the comparison used AM-HA.Source: §D.1 · p. 23
  • ARC-KV ranked first among compacted methods in all five QuALITY settings and four of five settings each on RULER and LongBench.Source: §D.1 · p. 23

D.2 QUALITY–EFFICIENCY TRADE-OFF: ADDITIONAL REMARKS

  • At 4K and 64K, ARC-KV’s component totals were 2.35 s and 1.66 min versus AM’s 16.47 s and 30.91 min; these are compaction, not end-to-end, times.Figure 6 profiles all 256 layer–KV-head pairs; at 64K, repeat prefill was ARC-KV’s largest remaining cost.Source: §D.3 · p. 23, §D.2 · p. 23, Figure 6 · p. 24

D.4 DECODE-TIME ATTENTION SPEEDUP

  • At ρ = 0.20, attention latency across 32 layers fell from 5.58 to 1.75 ms per token at 4K and from 47.12 to 10.63 ms at 32K; this isolates attention, not end-to-end decoding.The full-length bias-aware path was 1.06–1.09× slower than FlashAttention-2.Source: §D.4 · p. 24, Table 4 · p. 24

D.5 ANCHOR-SELECTION OVERLAP WITH OMP

  • Across five retention ratios, mean Jaccard overlap ranged from 0.144 to 0.201, indicating different selected supports at matched cardinality; the diagnostic does not establish a quality advantage.Figure 7 compares anchors before key merging and cache-reconstruction fits on 30 QuALITY development articles.Source: Figure 7 · p. 25, §D.5 · p. 25

E.1 KEY-MERGING COEFFICIENT

  • Key merging reduced relative attention-mass error by 43.9–56.9% at the best setting, whose coefficient rose from 0.4 at ρ = 0.01 to 0.7 at ρ = 0.20.Every nonzero coefficient improved on unmerged anchors at every retention ratio.Source: §E.1 · p. 26, Table 5 · p. 26

E.2 AGGREGATION METHOD

The ablations favor root-mean-square score aggregation and the default per-head budget allocation. The related-work discussion distinguishes ARC-KV’s fitted compact cache from hard-subset eviction and other reconstruction approaches.

E.2 AGGREGATION METHOD

  • Root-mean-square aggregation achieved the highest QuALITY accuracy at every retention ratio; at ρ ≤ 0.05, mean and max aggregation were worse by 4.59–6.60 and 7.05–8.95 points.It normalizes scores across keys per query before pooling across queries, preserving evidence from some queries without letting one peak determine the ranking.Source: §E.2 · pp. 26–27, Table 6 · p. 26

E.3 PER-HEAD BUDGET ALLOCATION

  • Uniform per-head allocation reduced QuALITY accuracy by 1.68–16.22 points, with losses exceeding 10 points below 10% retention and the largest loss at ρ = 0.02.Both allocation variants used the same total nominal cache budget.Source: §E.3 · p. 27, Table 7 · p. 27

F EXTENDED RELATED WORK

  • Many eviction methods retain hard subsets without refitting discarded mass or value contributions; ARC-KV instead fits β and Cv for each context while leaving base context encoding unchanged.The related-work discussion also describes methods that preserve discarded information through context adaptation or latent memory.Source: F EXTENDED RELATED WORK · p. 27
  • Compared with AM’s iterative greedy selection and refitting, ARC-KV selects original-key anchors in one indexer pass, then merges keys and fits β and Cv per context.Still avoids per-context optimization with a trained Perceiver compactor, while Cartridges optimizes a new cache for each context.Source: F EXTENDED RELATED WORK · p. 28
Loading 2609.36835v1…