Source-linked AI summary
When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
Haeyong Kang, Chang D. Yoo
TL;DR
Training-free KV-cache compression evicts at prefill end to preserve expected future attention mass, but that target can diverge from task quality. DGE drafts the first two answer tokens with the full cache and delays eviction; it scores 44.2 on LongBench, near FullKV’s 44.3, while timing-only DGE-W reaches the same score.
Problem
Prefill-time methods optimize attention-mass preservation, but restoring mass or capturing more of it does not necessarily preserve task quality.
Source: §1 · p. 2Method
DGE drafts the first k=2 answer tokens using the full cache and evicts afterward, changing when eviction occurs while retaining the per-head budget.
Source: §1 · p. 2Results
DGE leads prior methods at every evaluated budget on five of six backbones, scoring 44.2 on LongBench versus FullKV’s 44.3; timing-only DGE-W reaches 44.2.
Source: §1 · p. 3, §8 · p. 10Takeaways & Limitations
The results support deferring eviction until the answer’s initial trajectory is available, rather than relying only on improved prefill-time selection.
Source: §1 · p. 2, §8 · p. 10Trajectory anchoring is inferred from ablations rather than isolated, decoding is greedy throughout, and sampling robustness remains open.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Training-free KV-cache compression methods such as SnapKV, H2O, and PyramidKV evict tokens at the end of prefill, aiming to preserve the attention mass that future queries are expected to use -optimizing what to keep. We show that this objective fails in two distinct ways. (1) Compensation: restoring the evicted attention mass can recover the attention-level target without recovering task quality. (2) Selection: covering more of the true decode-query mass can hurt quality when the recovered mass is fragmented rather than concentrated in coherent spans. These failures share a common cause: eviction occurs before the queries that determine the answer trajectory exist. We propose Draft-Guided Eviction (DGE), which defers eviction until after drafting the first k=2 answer tokens using the full cache - just one decode step beyond prefill. Because the draft is generated from the answer's own prefix, no cache entries are discarded before this trajectory signal becomes available. The per-head cache budget remains unchanged, and DGE can be applied directly to SnapKV, PyramidKV, H2O, and StreamingLLM without modifying their eviction scores. Unlike extra-pass methods, DGE changes when eviction occurs rather than what cache entries are selected. Extensive experiments demonstrate that DGE outperforms prior methods at every evaluated budget on five of six instruct-tuned backbones, achieving 44.2 on LongBench, nearly matching FullKV at 44.3. The timing-only control DGE-W achieves the same score, demonstrating that the gain comes from when eviction occurs rather than what is selected - an effect we term trajectory anchoring.
1 INTRODUCTION
Prefill-time eviction preserves estimated future attention mass, but that objective can fail to preserve task quality. DGE drafts answer tokens with the full cache and delays eviction, changing when the cut occurs rather than what is selected.
1 INTRODUCTION
- Long-context KV caches grow with context length and batch size, making memory use and repeated cache reads central serving costs.An 8B model at 128K tokens requires roughly 16 GiB for the cache, comparable to its weights.Source: §1 · p. 1
Draft-guided eviction
- Training-free eviction regimes differ in which queries score the cache and when the irreversible cut occurs.DGE drafts with the full cache before eviction, whereas prefill-time methods cut before generation and LAQ uses synthesized pseudo-queries.Source: §1 · pp. 1–2
Controlled analysis of attention-based eviction
- Restoring evicted attention mass does not necessarily recover task quality, and greater attention coverage can hurt when retained content is fragmented rather than coherent.Both failures arise because eviction precedes the decode queries that consume the cache.Source: §1 · p. 2
Draft-guided eviction
- DGE drafts the first k answer tokens on the full cache, then evicts using real decode queries rather than prefill-time estimates.The draft is the answer’s own prefix, leaves the per-head budget unchanged, and moves eviction off the prefill critical path.Source: §1 · p. 2
Empirical validation
- Across six instruct-tuned backbones, DGE beats prior methods at every evaluated budget on five and scores 44.2 on LongBench, versus 40.1 for LAQ and 35.3 for SnapKV.DGE is within 0.2 of uncompressed FullKV; the timing-only DGE-W also reaches 44.2.Source: §1 · p. 3
2 RELATED WORK
Prior training-free methods generally choose which tokens to keep or repair the consequences of eviction, but still cut at prefill end. DGE instead defers that decision until after initial answer tokens.
Selection and allocation: which tokens to keep
- Selection methods vary token scoring and budget allocation, but commit at prefill end before the first answer token exists.Examples include SnapKV’s observation-window scores, H2O’s accumulated attention, and PyramidKV’s layer-wise budgets.Source: §2 · p. 3
- DGE changes eviction timing while leaving the per-head budget unchanged and allowing the base scoring rule to remain unchanged.On SnapKV’s scores, the timing change alone outperforms the selection rules evaluated in the paper.Source: §2 · p. 3
Compensation: restoring what selection discarded
- Compensation methods can restore targeted attention mass but cannot recover original token-level content once compression discards it.DGE avoids this repair problem by generating steering tokens while the cache remains intact.Source: §2 · p. 3
Extra passes: sharpening the prefill-time signal
- Extra-pass methods sharpen the prefill-time eviction signal, yet still cut before the queries that actually consume the cache exist.The passage contrasts repeated prefill, pseudo-query synthesis, and learned score prediction with DGE’s use of the answer’s own prefix.Source: §2 · pp. 3–4
3 PRELIMINARIES
The paper defines eviction as keeping a fixed per-head token budget and evaluates cache behavior with held-out real decode queries. Its diagnostics measure missed attention mass, coverage, output error, retained spans, and attention-matching loss.
Notation
- Each layer keeps B tokens per head: B−w selected past tokens plus the last w window tokens, with w=8 throughout.The prompt has length L before eviction.Source: §3 · p. 4
Held-out instrumentation
- Held-out coverage measures relative reduction in decode missed mass βdec versus SnapKV, while other diagnostics track output error, retained-mask run length, and attention-matching loss.The instrumentation captures real decode queries using a full-cache run and replays them against candidate caches offline.Source: §3 · p. 4
Evaluation protocol
- The evaluation uses LongBench’s 16 English datasets on Llama-3.1-8B-Instruct at budgets 32, 64, and 128, with greedy decoding and matched examples.Backbone checks include Mistral-7B, Qwen2.5-7B/14B, Llama-3.2-3B, and Llama-3-8B; comparisons use paired bootstrap intervals.Source: §3 · p. 4
Headline diagnostic
- At B=128, SnapKV misses 16.7% of decode attention mass on average, with a 9.4% median and 43% 90th percentile across examples, layers, and heads.The window-query missed mass correlates 0.72 with decode missed mass.Source: §3 · p. 4
4 ATTENTION-MASS ANALYSIS
Controlled dosage tests show that improved attention-level objectives do not reliably translate into LongBench gains. Compensation can restore mass without content, while selection trades coverage against contiguous retained spans.
Attention-mass objective tests
- Four what-axis mechanisms improve their attention objectives by 8–57% yet gain at most +0.6 on LongBench; LAQ’s 16 pseudo queries gain +2.0, and equal 16-token deferral adds a further +2.0.The reported pattern is that meeting each mechanism’s attention objective does not make its gains transfer to task score.Source: §4 · p. 4
Compensation restores mass, not content
- Compensation restores attention mass without necessarily restoring task quality because mass and original token content are not interchangeable.The compared approaches include merging, value absorption, and per-token bias correction.Source: §4 · pp. 4–5
Selection trades coverage for contiguity
- SnapKV’s max-pooling gives up 9–22% held-out coverage across B=64–512 yet wins scores by preserving span contiguity; at B=128, no pooling reverses both outcomes.Oracle bounds cap practical selection rules, while real decode queries provide the only signal materially above the window in coverage.Source: §4 · p. 5
- DGE16 gains +3.98 where no pooling loses 1.73, while timing-only DGE-W16 gains +3.95 using SnapKV’s long runs.The passage attributes the contrast to deferring the cut until real queries exist, rather than to a better prefill-time rule; SKV+DGE2 recovers FullKV in both short-answer and long-form regimes.Source: §4 · p. 5
5 DRAFT-GUIDED EVICTION
DGE moves eviction until after the model drafts the opening answer tokens on the full cache, so the cut follows real decode queries rather than preceding them. The draft anchors the generation trajectory; the cost is a one-time cache edit.
5 DRAFT-GUIDED EVICTION
- DGE delays the cut until the real decode queries exist, addressing the shared cause of compensation and selection failures.With the real queries available before eviction, the paper argues nothing remains to compensate and the coverage–contiguity trade-off from pre-generation scoring dissolves.Source: §5 · p. 5
- The policy drafts k answer tokens with the full cache, scores past tokens, keeps the per-head budget, and edits the cache once.The default scorer uses mean draft-query attention; the timing-only variant keeps the base evictor’s scoring rule unchanged.Source: §5 · p. 5
- The draft is the answer’s own prefix, and the cache surgery is a one-time in-place edit before generation resumes.The draft tokens remain ordinary output, and the procedure uses no extra pass or training.Source: §5 · p. 5
- 1.45× cache-surgery latency on short answers amortizes to 1.02× on long answers, while TTFT is 0.93× SnapKV’s.If the answer ends within the draft, no eviction occurs and output is bit-identical to FullKV’s.Source: §5 · pp. 5–6
- The authors hypothesize that the full-cache draft anchors a trajectory that the compressed cache can then carry.They connect this hypothesis to DGE-W matching DGE’s 44.2 average and carrying nearly all of the recovery in Figure 3(b).Source: §5 · p. 6
6 EXPERIMENTS
Across the reported experiments, one-step deferral leads prior methods at every evaluated budget and remains near FullKV. Results and controls attribute most of the gain to eviction timing, with a one-time cache-surgery cost.
6.1 EXPERIMENTAL SETUP
- The evaluation covers LongBench’s 16 English datasets, with backbone replications and matched budgets, sample sets, and greedy decoding.The primary setup uses Llama-3.1-8B-Instruct; reported budgets are 32, 64, and 128.Source: §6.1 · pp. 6–7
6.2 MAIN RESULTS
- 44.17 average LongBench score for SKV+DGE2 versus 40.13 for SKV+LAQ and 35.34 for SnapKV across fifteen backbone–budget cells.Both DGE variants beat every prior method in all fifteen cells and stay within 0.77 of FullKV; the result is summarized in Table 2.Source: §6.2 · p. 7, Table 2 · p. 7
- As B shrinks from 128 to 32, average SKV+DGE2 moves from 44.30 to 44.04 while SnapKV falls from 39.06 to 31.06.The paper also reports that deferral composes with PyramidKV, H2O, and StreamingLLM in the settings tested.Source: §6.2 · p. 7, §6.1 · p. 7
- On gov_report, SKV+DGE2 raises ROUGE-L from 22.3 to 32.6 against FullKV’s 33.8, while about 99% of tokens decode after eviction.The passage also reports 11.2% of samples terminating inside the draft at k=2 and no such terminations on long-generation categories.Source: §6.2 · p. 8, Figure 4 · p. 8
6.3 ATTRIBUTION ABLATION
- On the matched gov_report split, timing contributes +11.0 and the signal change contributes +0.7; timing accounts for 94% of recovery.Across fifteen cells, deferral alone raises SnapKV’s average from 35.34 to 44.17, while adding real decode-query scores leaves the average at 44.17.Source: §6.3 · p. 8
6.4 DOSE-RESPONSE IN k
- At B=128 on Llama-3.1-8B, DGE scores 46.37 at k=2 and remains flat through k=32; the dose is also flat at B=32.The passage reports a +4.2 jump from SnapKV at k=2; Figure 4 shows four of five swept backbones saturating at k=2, with Llama-3-8B as the exception.Source: §6.4 · p. 8, Figure 4 · p. 8
6.5 LOOKAHEAD-QUERY EVICTION
- DGE-W2 clears LAQ’s best dose by +2.1 at B=128 and +5.8 at B=32; its average margin across fifteen cells is +4.0.The captioned per-dataset view marks DGE’s lead over LAQ on the labelled sets.Source: Figure 4 · p. 8, §6.5 · p. 9
6.6 COMPLEXITY AND OVERHEADS
- TTFT is 0.93× SnapKV’s, while cache surgery costs 1.45× on 32-token answers and amortizes to 1.02× by 512 tokens.The reported H2O accumulator cost on 32-token answers is 2.15×.Source: §6.6 · p. 9, Table 3 · p. 9
7 DISCUSSION
The discussion reframes KV-cache evaluation around task quality and generation trajectory, not attention-output fidelity alone. It also identifies open questions about sampling, workload length, and memory overhead.
7 DISCUSSION
- A timing-only deferral closes almost all of the SnapKV-to-FullKV gap, separating eviction timing from selection quality in benchmarking.The authors describe the reported penalty as timing rather than selection quality.Source: §7 · p. 9
- Attention-output fidelity is geometric, while task quality depends on retrievable content and a stable trajectory.Source: §7 · p. 9
- The evidence uses greedy decoding, and LongBench’s short answers limit dose probing; sampling and longer-form, agentic, and multilingual workloads remain open.At k=16, a third of Llama-3.1-8B’s answers terminate inside the draft; the draft also holds the full cache, making the prototype working set 2× FullKV’s, or 1.13× in native GQA layout.Source: §7 · p. 9
8 CONCLUSION
The conclusion argues that prefill-time attention-mass preservation can miss task quality because eviction precedes the queries that shape the answer. DGE defers the cut until after a short full-cache draft and achieves near-FullKV LongBench performance.
8 CONCLUSION
- Restoring evicted attention mass may not restore task quality, and greater query-mass coverage can hurt when recovered mass is fragmented.Source: §8 · p. 9
- DGE drafts the first k=2 answer tokens on the full cache before eviction, then applies the timing change across SnapKV, PyramidKV, H2O, and StreamingLLM.Source: §8 · pp. 9–10
- DGE leads prior methods at every evaluated budget on five of six backbones, scoring 44.2 on LongBench versus FullKV’s 44.3.The timing-only control DGE-W reaches the same score.Source: §8 · p. 10
A EXPERIMENTAL DETAILS
The experiments span six instruct-tuned backbones and LongBench’s 16 English datasets, with matched decoding and per-head cache budgets. Controlled diagnostics find that restoring attention mass need not restore task quality, while selection trades held-out coverage against span contiguity.
A EXPERIMENTAL DETAILS
- The evaluation uses six instruct backbones, all 16 English LongBench datasets, greedy decoding, and per-head budgets B ∈ {32, 64, 128, 512}, with B=128 by default.The primary model is Llama-3.1-8B-Instruct; DGE uses k=2 in the main tables, the smallest dose admitted by the method.Source: A EXPERIMENTAL DETAILS · p. 14
B DIAGNOSTIC DETAILS
- Held-out decode coverage scores each selection rule by replaying decode queries against the full uncompressed keys, separately from the in-window quantities available to the rule.The diagnostic reproduces SnapKV’s scoring path, including softmax, score summation, max-pooling, and per-head top-B selection.Source: B DIAGNOSTIC DETAILS · p. 15
C COMPENSATION AND SELECTION DETAILS
- Compensation mechanisms meet their attention-level objective yet move LongBench by ∼0, while selection rules trade held-out coverage against span contiguity.SnapKV keeps the least raw mass but the longest contiguous runs and wins scores; raw-mass rules and oracles gain coverage by fragmenting spans.Source: C COMPENSATION AND SELECTION DETAILS · p. 15, Table 4 · p. 16, Table 5 · p. 16
- Renormalized mass inflow correlates 0.97 with a retained token’s own attention, and an unstructured value correction transfers poorly to held-out queries, with residual error 1.08× the uncorrected error.Post-RoPE arithmetic key averaging loses 30% of key norm to phase interference; repeat-prefill queries score selection 13% below the window in held-out coverage.Source: C COMPENSATION AND SELECTION DETAILS · p. 16
- A closed-form per-token bias adds log(1 + m_i/s_i) to the decode attention mask, while value absorption transports evicted values into retained ones.At B=128–512, offline replay measured 11–14% attention-output error reduction for absorption, versus −1.5% for bias alone.Source: C COMPENSATION AND SELECTION DETAILS · pp. 15, 17
D THE SIXTH BACKBONE: LLAMA-3-8B AND THE MINIMAL STEERING DOSE
On Llama-3-8B, the minimal useful draft length depends on the backbone: longer drafts progressively improve results. DGE defers the cache cut until after drafting, and the timing-only control tracks full DGE closely.
D THE SIXTH BACKBONE: LLAMA-3-8B AND THE MINIMAL STEERING DOSE
- 42.73 is the SKV+DGE32 LongBench score, above SnapKV 40.50 and recovering 72% of the gap to FullKV 43.58 on Llama-3-8B.The score rises from 39.73 at k=2 to 42.73 at k=32; the headline k=2 result is specific to decisive-opening backbones.Source: D THE SIXTH BACKBONE: LLAMA-3-8B AND… · p. 17
- Longer drafts help Llama-3 despite median gold-answer onset at 0 tokens and selection overlap with k=32 that is similar to Llama-3.1 at k=2 and k=16.The timing-only DGE-W climbs from 39.81 to 42.61 across k=2/16/32, remaining within 0.08, 0.19, and 0.12 of DGE at those doses.Source: D THE SIXTH BACKBONE: LLAMA-3-8B AND… · p. 18
E DRAFT-GUIDED EVICTION IMPLEMENTATION
- The implementation drafts before editing the cache, then retains top-B entries in temporal order and resumes generation; if the draft ends early, eviction does not fire.The prototype’s end-to-end latency is +9.9% over SnapKV, with TTFT 0.93×.Source: E DRAFT-GUIDED EVICTION IMPLEMENTATION · p. 20
E.1 WHY DEFERRAL CAN RECOVER FULLKV: AN EVICTION-MASS ARGUMENT
- At k=2, 11.2% of answers finish before eviction, while drafting on the full cache keeps recent draft tokens and is associated with reduced later evicted mass.The passage reports 42.2 < 46.17≈46.19 < 46.24 for SnapKV, DGE-W16, DGE16, and FullKV; at k=2, both deferred rows edge past FullKV by margins unresolved from zero.Source: §E.1 · p. 20
- Once eviction is deferred, DGE-W and DGE differ only in the evicted set, and their losses remain close because deferred mass is already small and their selected sets largely overlap.At k=16, DGE-W16 scores 46.17 versus 46.19 for DGE16.Source: §E.1 · p. 20
F DRAFT-LIKE PASSES COMPARED
- DGE uses the answer’s first k tokens and moves the cut later, whereas repeat-prefill, self-study, and Lookahead Q-Cache discard their pass and still cut at prefill end.With SnapKV’s window scores, DGE-W stays within 0.44 of full DGE in every Table 2 cell; LAQ saturates at least 2 points below the timing change at its best dose.Source: F DRAFT-LIKE PASSES COMPARED · p. 21
G COMPLEXITY AND OVERHEADS
These sections measure DGE’s overhead and test whether attention-matching scores predict downstream quality. They also assess deferred eviction under paired uncertainty analysis and tight-budget retrieval.
G COMPLEXITY AND OVERHEADS
- DGE’s total latency is flat across k=2, 4, and 8 (10.88s, 10.49s, and 10.49s), while its overhead over SnapKV is +0.96s (+9.9%).TTFT is 0.93× SnapKV’s, essentially FullKV’s 0.91×; the measured overhead is attributed to one-off per-layer cache surgery, which amortizes in long generations.Source: G COMPLEXITY AND OVERHEADS · pp. 21–22, Table 9 · p. 23
- DGE2’s working set is 13.13 GiB versus FullKV’s 6.58 GiB, but native GQA layout reduces DGE2 to 7.44 GiB without a material score change.The measured layout change removes query-head expansion from the cache and compute path; the corresponding spot-check score differences remain within the ±0.5 no-effect band.Source: G COMPLEXITY AND OVERHEADS · p. 22, Figure 7 · p. 22
H ATTENTION-MATCHING LOSS
- SnapKV’s attention-matching loss ranges 0.14–0.28 while its LongBench deficit ranges −1.5 to 13.2 points; DGE-W2 reduces the deficit to −0.09 on average.Across 16 Llama-3.1-8B sets, the loss and deficit are essentially uncorrelated (Pearson r≈0.2); DGE-W2’s worst-case deficit is 1.5 points.Source: H ATTENTION-MATCHING LOSS · p. 23, Figure 8 · p. 24
I CONFIDENCE INTERVALS AND SIGNIFICANCE
- Paired bootstrap intervals distinguish both deferred variants from SnapKV by ∼4.1 on Llama-3.1-8B, but do not distinguish DGE2 from DGE-W2.On Qwen2.5-14B at B≤64, the timing-only control leads by 0.4; both variants remain indistinguishable from FullKV at every budget.Source: I CONFIDENCE INTERVALS AND SIGNIFICANCE · pp. 24–25, Table 10 · p. 24
J NEEDLE-IN-A-HAYSTACK RETRIEVAL
- At B=32, DGE2 recovers 100.0 against SnapKV’s 74.7 and FullKV’s 98.9 on Needle-in-a-Haystack retrieval.The probe uses Qwen2.5-7B and 150 probes per method; deferral restores retrieval across all tested depths and context lengths.Source: J NEEDLE-IN-A-HAYSTACK RETRIEVAL · p. 25, Figure 9 · p. 25
K FULL RESULT TABLES
The full result tables report LongBench outcomes across baselines and deferred variants, including per-dataset, per-category, and dose comparisons. Results use official dataset sizes, subject to each backbone’s context limit.
K FULL RESULT TABLES
- The per-dataset tables cover prefill-time baselines and deferred rows across six backbones, with FullKV as a budget-independent reference and official LongBench sample sizes.The only stated truncation is each backbone’s standard context limit.Source: K FULL RESULT TABLES · pp. 25, 28, Table 11 · p. 26, Table 12 · p. 27
- On Llama-3.1-8B, both DGE and timing-only DGE-W stay at least 3.9 points above SnapKV and within 0.16 of FullKV across the tested k values.Table 13 covers B=128’s full dose grid and a sampled B=32 grid; both signals are flat from k=2.Source: Table 13 · p. 27
- Table 14 compares prefill-time LAQ’s pseudo-query scoring with DGE-W’s unchanged SnapKV scores and DGE’s real decode queries, across matched LongBench budgets.LAQ sharpens selection while evicting at prefill; both DGE variants defer eviction.Source: Table 14 · p. 28
- At B=32, one deferred decode step restores every LongBench category to FullKV levels, while SnapKV is 10.4 points lower overall and up to 16.3 points lower on Few-shot Learning.SnapKV’s losses concentrate in Few-shot Learning, Code, Single-Doc QA, and Summarization; Synthetic already matches FullKV.Source: Table 15 · p. 28
L ADDITIONAL ANALYSIS
Additional analyses test draft termination, composition with other eviction rules, and the dose needed across backbones. Held-out diagnostics report which selection, transport, allocation, and correction signals align with decode behavior.
L ADDITIONAL ANALYSIS
- At k=2, only 11.2% of answers finish in the draft, while DGE2 still matches FullKV at 46.37 versus 46.24; on gov_report, ROUGE rises from 22.3 to 32.6.On gov_report, 0% of answers terminate in the draft and ∼99% of tokens decode on the compressed cache.Source: L ADDITIONAL ANALYSIS · p. 28, Table 16 · p. 29
- At B=128, deferral lifts PyramidKV to 46.21, H2O to 45.73, and StreamingLLM to 45.83; timing-only compositions remain within 0.35 of draft-scored variants.The reported gains also hold at B=32 and on Qwen2.5-7B.Source: L ADDITIONAL ANALYSIS · p. 29
- LAQ’s B=128 averages peak at 44.20 with m=16, below FullKV’s 46.24 and DGE-W2’s 46.33; the gap to timing-only grows as budgets shrink.The reported gap is 1.0–2.1 points at B=128, 3.0–3.9 at B=64, and 5.8–9.1 at B=32.Source: L ADDITIONAL ANALYSIS · p. 29
- The dose reaching FullKV is backbone-dependent: several tested backbones saturate at k=2, while Llama-3-8B needs approximately k=8.Qwen2.5-14B at B=32 recovers to 45.74 with k=2, versus FullKV’s 46.05.Source: L ADDITIONAL ANALYSIS · p. 30
M WHAT THE DIAGNOSTICS ESTABLISHED
- On held-out diagnostics, repeat-prefill selection has −13% coverage and mean run length 1.2, while decode replay favors contiguous spans relevant to future decoding.Key-similarity transport has approximately zero correlation with true decode-time flow; empirical co-attention transport is supported by the offline replay.Source: M WHAT THE DIAGNOSTICS ESTABLISHED · p. 30
- Per-head allocation offers +8.0% held-out coverage headroom versus +2.7% for layer-only allocation; value absorption recovers 11–14% of missing contribution.The diagnostics also report +1.5% attention-output error from the closed-form correction without value absorption, and similar performance for blockwise and token-level allocation.Source: M WHAT THE DIAGNOSTICS ESTABLISHED · p. 31
Related papers
This paper cites
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018
- Efficient Streaming Language Models with Attention Sinks2023
- MuSiQue: Multihop Questions via Single-hop Question Composition2021
- SnapKV: LLM Knows What You are Looking for Before Generation2024
- Learning to Compress Prompts with Gist Tokens2023