PaperlayerView exact revision on arXiv

Source-linked AI summary

Value-Aware Stochastic KV Cache Eviction for Reasoning Models

Ting-Yun Chang, Harvey Yiyun Fu, Deqing Fu, Chenghao Yang, Jesse Thomason, Robin Jia

arXiv:2606.03928v1cs.LGcs.CL

TL;DR

Reasoning models’ long outputs create memory and compute bottlenecks, while KV cache eviction can reduce efficiency at an accuracy cost. VaSE protects large-magnitude value states and adds stochastic eviction, outperforming prior eviction methods by 4.4%–4.9% on average across Qwen3 models and six tasks.

  • Problem

    Reasoning models’ extended outputs create substantial memory and computational overhead, motivating accurate KV cache eviction methods.

  • Method

    VaSE is a training-free recipe that protects large-magnitude value states and introduces stochasticity to diversify retained KV pairs.

  • Results

    VaSE achieves the best average accuracy, surpassing R-KV by 4.4% on Qwen3-4B and 4.9% on Qwen3-14B across six tasks.

  • Takeaways & Limitations

    VaSE bridges the efficiency–accuracy gap in KV cache eviction and highlights value-state magnitude and stochasticity as important cache-management axes.

  • Takeaways & Limitations

    The mechanism by which large-magnitude value states sustain reasoning progression remains unresolved.

Abstract

from arXiv · show

Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.

1 Introduction

Reasoning models improve complex-task accuracy through extended chains of thought, but unnecessarily long outputs create a test-time efficiency bottleneck. VaSE addresses KV-cache efficiency by protecting large-magnitude value states and adding stochastic eviction, while generalizing the value-magnitude insight to quantization.

  • Motivation: Extended chains of thought improve accuracy on complex tasks but create a test-time efficiency bottleneck through unnecessarily long outputs.Models may overthink simple arithmetic questions, using more than 900.
  • Motivation: KV cache selection reduces compute and memory movement by activating only a sparse subset of KV pairs while retaining the full cache in memory.Sparse attention methods attend to only a subset of previous tokens and include KV cache selection methods.
  • Key findings: Large-magnitude value states are crucial for accuracy: evicting them can trigger repetitive reasoning loops or nonsensical outputs.Value-state magnitudes have strongly skewed distributions, with a small fraction of tokens having abnormally large vector magnitudes.
  • Method and evaluation: VaSE is a training-free eviction recipe that prioritizes large-magnitude value states and introduces stochasticity during decoding for Qwen3 models evaluated across six math, code, and science tasks.The provided passage states that VaSE targets the decoding step and evaluates Qwen3 models on six tasks spanning math, code generation, and science question answering.
  • Generalization: Large-magnitude value states also produce disproportionately large reconstruction errors under per-token KV-cache quantization, indicating value magnitude and stochasticity are fundamental cache-management axes.The paper presents VaSE as bridging the efficiency-accuracy gap left by prior sparse attention methods.

2 Problem Setup

This section defines decode-phase KV cache eviction, contrasting irreversible eviction with selection-based sparse attention, and formalizes periodic eviction under a fixed cache budget. It introduces SnapKV, R-KV, and CurDKV as representative baselines differing in attention scoring, redundancy handling, and value-state awareness.

  • Selection- and eviction-based methods: Selection methods retain the full KV cache while activating sparse KV pairs, whereas eviction methods permanently discard low-importance pairs under a memory budget.SeerAttention-R predicts sparsity patterns with a trained attention gate; eviction risks irrecoverable information loss.
  • Selection- and eviction-based methods: Over 10,000 generated tokens can arise from fewer than 200 prompt tokens in a math question, motivating decode-phase KV cache compression.The section focuses on autoregressive reasoning models whose long outputs stress memory capacity.
  • Periodic eviction framework: Periodic eviction maintains K persistent tokens and a B-token recent buffer, capping the cache at N = K + B and evicting B pairs whenever the buffer fills.The buffer satisfies B ≪ K, and each decoding step adds one KV pair.
  • Representative eviction baselines: SnapKV ranks candidates by recent-query attention, R-KV adds a redundancy penalty, and CurDKV combines key and value leverage scores.SnapKV ignores values, R-KV discourages selecting cosine-similar tokens, and CurDKV does not promote diversity.
  • Value-state motivation: 14.3% accuracy results when large-magnitude value outliers are evicted, greatly below random eviction at the same token budget.The result indicates that extreme value-state magnitudes can be crucial to model accuracy.

3 Methodology

The methodology identifies large-magnitude value states as critical to eviction accuracy and introduces VASE, a training-free recipe combining value awareness with stochastic retention for diversity. VASE instantiates these principles through SnapKV- and CurDKV-based variants.

  • Value-state analysis: Large-Range value outliers disproportionately influence attention outputs and are hypothesized to be critical for model accuracy.Attention outputs are weighted sums of values, so larger-magnitude values have greater influence.
  • Value-state analysis: 14.3% GSM8K accuracy results when evicting the B largest-Range KV pairs, underperforming random eviction by 38.9% at the same 512-token budget.The failure mode includes endless self-reflection loops and non-grammatical sentences.
  • VASE framework: VASE combines upweighting large-magnitude value states with stochasticity to promote diversity in the retained cache.The recipe is training-free and is applied to SnapKV and CurDKV through two variants.
  • VASE variants: VASE-ATTNV reserves Nv of K token slots for the largest-Range value states, then applies stochastic sampling on top of SnapKV.The reserved candidates are unconditionally retained, while stochastic sampling replaces hard topk selection for the remaining candidates.
  • VASE variants: VASE-DKV introduces independently resampled Gaussian projections at each eviction step to promote diversity while retaining CurDKV’s value-aware product score.This contrasts with CurDKV’s fixed projection, which can repeatedly assign low scores to tokens with certain representations.

4 Experiments

Experiments show that value-awareness and stochasticity each improve KV-cache eviction accuracy, with complementary gains when combined. Across reasoning tasks and constrained budgets, VASE matches or exceeds strong selection and eviction baselines while retaining eviction’s memory efficiency.

  • Value Awareness: Increasing the value-reservation budget from 16 to 256 raises GSM8K accuracy from 73.3% (+9.0%) to 80.5% (+16.The case study isolates value-awareness by using VASE-ATTNV without sampling and filling the remaining budget with SnapKV-ranked pairs.
  • Stochasticity: Stochastic sampling improves SnapKV accuracy from 64.3% to 70.9%, while resampled Gaussian projections improve CurDKV from 78.6% to 87.6%.The results indicate that stochasticity promotes diversity in retained KV pairs, and its combination with value scoring achieves the highest accuracies.
  • Overall Evaluation: 59.09% average accuracy makes VASE-ATTNV best on Qwen3-4B, exceeding SeerAttention-R at 58.81% and R-KV at 54.69%.VASE-DKV reaches 57.48%, improving over CurDKV at 49.78%; both VASE variants outperform prior eviction baselines on per-model averages.
  • Code Generation: 63.5% pass@1 for VASE-ATTNV on LiveCodeBench-v6-Medium is comparable to R-KV at 62.6% and VASE-DKV at 61.9%, while CurDKV reaches 34.6%.SeerAttention-R records 45.3% under the same 2048-token budget, below the reported VASE and R-KV results.
  • Token Budgets: At a 2048-token budget, both VASE variants clearly improve over R-KV and CurDKV on Qwen3-14B AIME26 and HMMT25, with gaps diminishing as budgets increase.The 2048-token budget corresponds to roughly 7.5× compression on AIME26 and 8.7× on HMMT25; all methods recover accuracy at larger budgets.
  • Quantization Connection: Empirical quantization experiments validate a relationship between Range(v) and per-token value-cache reconstruction error across bit-widths and group sizes.The study uses HQQ quantization and measures mean-squared reconstruction error between original and reconstructed value states.

5 Benchmark Throughput and Memory

On Qwen3-14B with FlashAttention2, VaSE eviction methods substantially improve decode throughput and reduce peak memory versus Full across KV-cache budgets and output lengths. VASE-DKV is fastest, while memory follows the same efficiency ordering in reverse.

  • Benchmark setup: Benchmarking uses Qwen3-14B on one A100-80G GPU with FlashAttention2, batch size 16, 256-token prompts, KV budgets of 2048–6144, and outputs of 16K–32K tokens.The setup excludes PagedAttention.
  • Decode throughput: 3.1× faster: VASE-DKV reaches 411 tokens per second versus 133 for Full at 16K output tokens and a 2048-token KV budget.Across settings, throughput ranks VASE-DKV > VASE-ATTNV > R-KV > Full.
  • Decode throughput: VASE-DKV is the fastest eviction method because it avoids attention-score computation during eviction, while VASE-ATTNV outperforms R-KV by avoiding redundancy-score computation.Throughput decreases as the token budget increases for all eviction methods.
  • Peak memory: Memory ordering reverses throughput ordering: VASE-DKV < VASE-ATTNV < R-KV < Full, consistently across different token budgets.The figure reports peak GPU memory at 16K output tokens and a 4096-token budget; Full is OOM at 32K.

6 Related Work

Related work compresses KV caches by reducing token count, precision, or hidden dimension, while reasoning-model decoding remains especially memory-bound because long generated sequences grow the cache. Prior scoring methods also incorporate value-state magnitude into KV-pair importance, motivating deeper investigation of large-magnitude values.

  • KV Cache Compression Methods: KV cache compression reduces token count through sparse attention, precision through quantization, or hidden dimension through low-rank decomposition.These approaches represent three distinct compression directions discussed in prior work.
  • Decoding Phase Compression: Decode-phase KV pairs dominate reasoning-model cache costs because token-by-token generation appends pairs to a growing, memory-bound cache.The prefill phase computes prompt KV pairs in parallel, whereas decoding generates one token at a time.
  • Leveraging Value-State Magnitude for KV Cache Scoring: Sparse-attention scoring methods weight attention scores by corresponding value norms because attention outputs combine value states.This work investigates the importance of large-magnitude values beyond simply multiplying attention and value scores.

7 Discussion and Conclusion · Appendix

The discussion identifies large-magnitude value states as essential for maintaining reasoning progression and eviction accuracy, while the conclusion emphasizes protecting them and adding stochasticity to improve cache diversity. It also proposes mixed-precision quantization as a future direction and outlines the appendix’s supplementary analyses.

  • 7 Discussion and Conclusion: Large-Range value states are also a major source of quantization error, motivating mixed-precision KV caches that preserve their precision.A proposed design reserves high precision for these states and quantizes them into the lower-precision cache once the reserved budget is exhausted.
  • 7 Discussion and Conclusion: Large-magnitude value states are crucial for reasoning progression because evicting them often causes repetitive reasoning loops without progress.The paper hypothesizes that these states help transition between latent reasoning steps and prevent collapse into self-reinforcing repetition.
  • 7 Discussion and Conclusion: VASE protects large-magnitude value states and introduces stochasticity to improve KV cache diversity during compression.The framework is training-free and is designed to improve existing eviction methods while addressing reasoning models’ memory bottleneck.
  • Appendix: The appendix reports additional analyses on value-state magnitude and variety, token statistics, hyperparameters, statistical significance, results, limitations, and compute resources.These topics are organized into appendices A through F.

A Variants for Capturing Value-State Magnitude and Variety

The section evaluates L2 norm, range, and variance as value-state scoring signals for protecting outliers during KV cache eviction. Because L2(v) and Range(v) are highly correlated, the method adopts Range(v), which directly captures extreme values linked to quantization outliers.

  • Outlier characterization: Value-state distributions contain prominent outliers with large L2(v) and Range(v) across different layers.These distributions were measured from Qwen3-4B’s full KV cache during GSM8K generation.
  • Outlier characterization: L2(v) and Range(v) exhibit Pearson correlation above 0.8 across layers.This indicates that the two magnitude measures track similar layer-wise value-state behavior.
  • Scoring variants: The eviction framework reserves dedicated value slots and tests keeping states with the largest L2(v), Range(v), or Var(v).The remaining token budget uses stochastic sampling on top of attention scores.
  • Scoring variants: The three value-scoring variants achieve comparable performance gains over the SnapKV baseline at 4× KV cache compression.Table 3 reports results for Qwen3-4B and describes the variants as capturing different aspects of value-state magnitude and variety.
  • Main scoring function: Range(v) is selected for the main approach because it captures each vector’s extreme max v and min v values, connecting it to quantization outliers.The section links large-Range states to the outlier challenge in LLM quantization and notes that evicting them can trigger nonsensical reasoning loops.

B Token Statistics and Hyperparameters · C Statistical Significance

Reasoning-model decoding can exceed 10,000 tokens despite fewer than 200 prefill tokens, motivating fixed KV-cache budgets. Statistical evaluation samples multiple generations per problem, and VASE outperforms all baselines across every task–model combination despite overlapping confidence intervals.

  • B Token Statistics and Hyperparameters: Over 10,000 decode tokens can follow fewer than 200 prefill tokens for a math question, intensifying decode-phase memory constraints.This motivates bounding KV-cache memory usage to a fixed cost.
  • B Token Statistics and Hyperparameters: Evicting large-Range value states can make model outputs enter repetitive reasoning loops instead of reaching a correct conclusion.Examples include repeated reconsideration of a file-download problem and endless self-reflection after generating the answer 18 once.
  • C Statistical Significance: Table 5 reports reasoning-task accuracy (%) with standard errors for Qwen3-14B eviction methods under ∼4× cache compression.The evaluation compares KV-cache eviction methods using average accuracy ± standard errors.
  • C Statistical Significance: For each problem, the evaluation samples R independent generations and computes per-problem pass@1 accuracy.R = 16 for datasets with fewer than 100 examples and R = 8 otherwise.
  • C Statistical Significance: Standard-error variability reflects model-sampling stochasticity and, for VASE methods, randomness from stochastic eviction.Larger standard errors on AIME25 and AIME26 primarily result from their 30-problem test sets.
  • C Statistical Significance: Despite overlapping individual confidence intervals, VASE outperforms all baselines across every task–model combination.The passage attributes overlapping intervals to small test sets.

D Additional Results

Additional results confirm that reserving large-range value states is important for GSM8K accuracy, while VaSE methods reduce peak memory more than R-KV and remain feasible for long outputs. Value-state range distributions remain stable across context positions after excluding sink tokens.

  • Value-state reservation: Randomly reserving 16, 32, or 64 KV pairs yields GSM8K accuracies of 65.6%, 63.4%, and 65.3%, respectively, underperforming Figure 3’s large-range reservation.These results are comparable to SnapKV (64.3%).
  • Value-state distributions: Except for Layer 0, sink tokens have lower median Range(v) than other token chunks, consistent with value-state drain phenomena.The first four sink tokens are analyzed as a separate chunk.
  • Memory footprint: VASE-DKV < VASE-ATTNV < R-KV < Full in peak memory footprint across KV-cache budgets of {2048, 4096, 6144} at 16K output tokens.VASE-DKV is the most memory-efficient method at every budget.
  • Memory footprint: At 32K output tokens, eviction methods retain the same memory footprints under identical token budgets, while Full no longer fits on a single 80GB GPU.This demonstrates the memory advantage of eviction for longer generations.

E Limitations … H Broader Impacts

The paper identifies evaluation, hardware, licensing, and scope limitations while framing VaSE as a training-free inference optimization with potential efficiency and access benefits. Future work includes extending compression beyond decode and integrating eviction with outlier-aware quantization.

  • E Limitations: The evaluation covers reasoning-model decode, not prefill-phase compression benchmarks, although the authors believe the methodology may extend to long-prompt compression.The method is built on observations of value-state distributions and stochastic diversity.
  • E Limitations: The methods are implemented only on Qwen3 models, limiting the reported model coverage.
  • E Limitations: The paper links value-state magnitude to quantization errors but focuses on sparse attention rather than developing quantization methods.Future work could combine KV cache eviction with outlier-aware quantization to improve compression ratios.
  • F Compute Resources: Experiments ran on NVIDIA A100 and H100 GPUs, with every experiment runnable on a single 80GB GPU.
  • F Compute Resources: Without PagedAttention and continuous batching, each task requires 10-72 GPU hours depending on dataset size and average generated-token count.
  • G Licenses: Table 7 lists licenses for artifacts used in the work, including copyrighted AIME problems used only for evaluation.Copyright © MAA identifies the Mathematical Association of America as the copyright holder for those problems.
  • H Broader Impacts: VaSE reduces reasoning-model inference memory footprints, potentially lowering hardware and energy requirements and broadening access for resource-constrained practitioners and organizations.The work describes VaSE as a general-purpose inference optimization that does not introduce new model capabilities or alter model output.
Loading 2606.03928v1…