PaperlayerView exact revision on arXiv

Source-linked AI summary

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang, Weihua Luo, Yazheng Yang, Jinsong Su

arXiv:2608.22704v1cs.CLcs.SD

TL;DR

Long-form audio makes KV-cache memory a dominant cost, and prefill-only compression can miss positions needed during decoding because attention shifts across the audio. WnW uses calibrated head roles and decode-time CPU recall to adapt retention, remaining near Full-Cache accuracy with only 20% of audio tokens on GPU. The evaluation reports generalization across languages, tasks, and domains, with limited recall overhead.

  • Problem

    Prefill-only audio KV compression assumes prefill attention predicts decode-time importance, despite evidence of weak overlap between the two signals on long-form audio.

  • Method

    WnW assigns KV-heads to anchor, tidal, and fixed roles offline, then uses anchor-head decode-time attention to recall CPU-resident audio chunks for tidal heads.

  • Results

    WnW remains within ∼1.6 WER points of Full Cache with only 20% of audio tokens on GPU across Voxtral-mini-3b and Qwen2.5-Omni-3B.

  • Takeaways & Limitations

    WnW stays close to Full Cache across GPU budgets, generalizes across language, task, and domain shifts, and adds limited CPU–GPU recall overhead.

  • Takeaways & Limitations

    The evaluation targets offline decoding, and WnW does not transfer to cross-attention, transducer, or state-space architectures.

Abstract

from arXiv · show

Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads remain on GPU and serve as a decode-time importance observer; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.

1 Introduction

Long-form audio makes KV-cache memory a central deployment bottleneck, while prefill-only retention is unreliable because prefill and decode attend to different audio positions. WnW addresses this mismatch with decode-time recall and achieves near-Full-Cache accuracy at a substantially reduced GPU footprint.

  • Motivation: At 12.5–25 audio tokens per second, a ten-minute clip occupies 7 500–15 000 KV positions, with audio comprising 70–80% of the total cache.Reducing the on-GPU audio KV footprint supports larger batch sizes and higher serving throughput under a fixed GPU budget.
  • Motivation: Prefill-only methods score positions once using prompt attention and retain the top-ranked positions throughout decoding.AudioKV adapts this paradigm with FFT-smoothed scoring and an audio-tuned per-head budget.
  • Motivation: Prefill attention concentrates near the audio start, whereas decode-cumulative attention spreads across the full clip, producing weak overlap between their top positions.This mismatch makes fixed retention vulnerable because discarded positions cannot be recovered during decoding.
  • Method: WnW classifies KV-heads as anchor, tidal, or fixed through offline calibration and retains CPU-resident complements for decode-time recall.Anchor heads remain fully on GPU as importance observers; tidal heads recall high-scoring chunks, while fixed heads retain a static GPU subset.
  • Results: WnW remains within ∼1.6 WER points of Full Cache with only 20% of audio tokens on GPU across two 3B backbones.At this budget, prefill-only baselines fail to terminate; results generalize across languages, tasks, and domains, with limited CPU–GPU recall overhead.

2 Method

WnW addresses long-form audio KV-cache pressure by combining offline KV-head triage with decode-time, CPU-backed chunk recall. It targets a prefill–decode attention mismatch in which prefill rankings poorly predict the audio positions used during decoding.

  • Head Functional Triage: WnW classifies KV-heads into anchor, tidal, and fixed roles using offline voice-score and head-sensitivity calibration.Anchor heads retain full audio KV on GPU, tidal heads offload a CPU-recallable complement, and fixed heads retain only an on-GPU subset.
  • Decode-Time Chunk Swap: During decoding, anchor-head attention identifies important audio chunks, which WnW recalls into tidal heads while releasing chunks that remain unselected.The CPU copy persists, allowing previously released tidal-head segments to be fetched again.
  • Prefill–Decode Attention Mismatch: Prefill-only compression is vulnerable because prefill attention concentrates near the audio start, whereas decode-cumulative attention spreads broadly across the clip.On LibriSpeech-Long, prefill assigns 47.9% of mass to the first 10% of audio, compared with 9.8% for decode-cumulative attention.
  • Prefill–Decode Attention Mismatch: 0.187 and 0.240 are the prefill/decode top-K Jaccard overlaps at K=100 and K=500, respectively, indicating weak ranking agreement.The corresponding random baseline at K=100 is approximately 0.013.

3 Experiments

Experiments evaluate WnW against compressed-cache baselines across backbones, tasks, languages, domains, retention levels, and runtime settings. WnW remains near Full-Cache quality at low GPU retention, while recall and head triage address failure modes of irreversible prefill compression.

  • Evaluation setup: Experiments use Voxtral-mini-3b as the main backbone, Qwen2.5-Omni-3B for cross-model evaluation, and Voxtral-Small-24B for a larger-scale check.LibriSpeech-Long provides the main English ASR benchmark; LongSpeech and PriMock57 test cross-lingual, cross-task, and domain transfer.
  • Evaluation setup: All KV-management baselines compress only audio tokens, while prompts and generated tokens remain uncompressed for a matched comparison.AffPool instead merges audio tokens before KV construction, so its retention is reported as the remaining audio-token ratio.
  • Main results: WnW stays within approximately 1 WER point of Full Cache on Voxtral and approximately 1.6 WER points on Qwen across retention levels.At low retention, Ada-KV and AudioKV fail to terminate, while ArkVale avoids non-termination but does not consistently match WnW.
  • Main results: At 20% GPU retention, WnW remains near Full-Cache accuracy, whereas AffPool collapses at 20%–40% retention and prefill-only baselines inherit decode-time mismatch.WnW preserves token identities and defers recall decisions to decoding; the ordering also holds on Voxtral-Small-24B.
  • Generalization: WnW transfers across language, task, and domain shifts, losing 2.3 WER points on French ASR, 0.27 BLEU on translation, and 0.76 WER points on medical dialogue relative to Full Cache.Replacing English calibration with French calibration changes English WER by at most 0.04 points and French WER by 1.12 points, with 76/85 tidal heads preserved.
  • Ablations: A larger tidal pool reduces WER from 51.92% to 6.31% at rGPU=0.1, while combining voice score and sensitivity avoids failures of either signal alone.At rGPU=0.1, sensitivity alone trails the combined signal by 26.63 WER points, and voice score alone fails to terminate at 124.66%.
  • Efficiency and runtime: CPU-to-GPU recall transfers at most 1.04 MB per step, and median decode time changes by less than 5% across a 6.7× range of tidal-head counts.These measurements indicate that recall is not the dominant decoding bottleneck in this setting.

4 Related Work

Prior KV-management methods largely make one-way retention decisions from prefill or early-decode signals, whereas WnW introduces speech-specific head triage and recallable decode-time chunk swapping.

  • Static KV Cache Compression: Static compression methods retain prefill- or early-decode-ranked positions and never re-admit evicted positions.This one-way eviction paradigm includes Scissorhands, H2O, SnapKV, Ada-KV, PyramidKV, and ChunkKV.
  • Head-Level KV Management: Tidal heads retain an exact CPU-resident complement that can be recalled mid-decode, unlike positions summarized away in non-retrieval heads.Anchor heads additionally observe decode-time importance.
  • Audio LLM KV Compression: AudioKV adapts static eviction to speech LLMs with FFT-smoothed scoring and audio-tuned per-head budgets.Its voice score derives from SparseMM’s visual score.
  • Audio LLM KV Compression: WnW drives recallable decode-time chunk swaps instead of static prefill budgets, distinguishing it from prior audio KV compression.It combines a reused voice score with gradient-based head sensitivity.

5 Conclusion

WnW addresses the mismatch between prefill and decode attention by deferring retention decisions and recalling audio chunks during decoding. It remains close to Full Cache across GPU retention levels and generalizes across several evaluation axes.

  • 5 Conclusion: WnW defers the retention decision to decode time through offline anchor, tidal, and fixed head roles.Anchor heads observe decode-time importance, while tidal heads recall audio chunks from CPU on demand.
  • 5 Conclusion: WnW stays close to Full Cache at every r_GPU and generalizes across language, task, and domain.The conclusion also reports scaling to a 24B backbone.

Limitations

The authors identify implementation and deployment boundaries for WnW, including naïve recall scheduling, high-retention behavior, offline decoding, and architectural scope.

  • Limitations: CPU–GPU chunk-swap traffic is scheduled naïvely, although measurements show limited decode-time overhead.Overlapping recall with attention via CUDA streams is proposed as a next optimization.
  • Limitations: The recall pathway is dormant in the high-r_GPU regime, motivating decoupling n_voice and λ.This is presented as a direction for further gains.
  • Limitations: The evaluation targets offline decoding, while extension to streaming recognition is described as natural for architectures with a growing shared KV cache.The same demand is noted for simultaneous machine translation.
  • Limitations: WnW does not transfer to cross-attention, transducer, or state-space architectures that lack a length-dependent audio KV cache.The stated limitation concerns the mechanism’s required cache structure.
  • Limitations: Application to non-transcription audio tasks is deferred until underlying speech LLMs reach reliable quality on those tasks.Examples include question answering, dialogue, and summarisation.

A Full Main Results

Table 5 reports the full LibriSpeech-Long results associated with Figure 3, including test-other scores and WnW’s measured retention ratios on both backbones.

  • A Full Main Results: WnW’s measured r_GPU and r_GPU+CPU values are reported at every retention level for both backbones.The supplied passage does not state the individual retention values.

B Scaling to a Larger Speech LLM

WnW scales to the Voxtral-Small-24B backbone while retaining its advantage over other KV-management methods at approximately 20% GPU retention.

  • Table 6 reports truncated WER results for scaling to Voxtral-Small-24B on LibriSpeech-Long.
  • At ∼20% GPU retention, WnW remains the best KV-management method on the Voxtral-Small-24B backbone.Head roles are recalibrated for the 24B model, while chunk size and anchor-head count are transferred from Voxtral-mini-3b.
  • On Voxtral-Small-24B, prefill-only baselines collapse, while ArkVale remains substantially worse than WnW.The absolute gap to Full Cache is larger than on 3B models, reflecting the stronger base model and lower Full Cache WER.

C Hyperparameter Sensitivity

WnW remains stable across broad sweeps of chunk size and anchor-head count at approximately 20% GPU retention.

  • Table 7 reports hyperparameter sensitivity using truncated WER at ∼20% GPU retention.
  • Across an 8× range of chunk sizes and a 20× range of anchor-head counts, WER fluctuates by no more than roughly 0.3 points.The sweep uses Voxtral-mini-3b on LibriSpeech-Long test-clean at approximately 20% GPU retention.
  • The default WnW configuration is not the result of narrow-range tuning.

D Cross-Dataset Calibration Robustness

Cross-dataset calibration produces similar English performance and bounded French variation, while a much lower confidence threshold leaves the calibration behavior highly correlated.

  • Table 8 reports cross-dataset calibration robustness with WhisperX threshold 0.85 and target r_GPU=0.2.
  • Switching the calibration language and dataset changes English WER by at most 0.04 points and French WER by 1.12 points.The matched rerun compares English LibriSpeech-Long and French LongSpeech ASR at n_voice=90, n_anchor=5, and target r_GPU=0.2.
  • Lowering the WhisperX confidence threshold from 0.85 to 0.44 increases retained calibration tokens by 45×, while per-head voice-score correlation remains 0.974.WER changes by at most 0.31 points relative to the French 0.85 calibration.

E Calibration Score Details

WnW’s voice-score calibration uses forced alignment to measure whether each KV-head’s top audio positions cover the audio interval corresponding to each answer token.

  • For each answer token, WhisperX forced alignment maps a word-level interval to audio-token indices.The interval [a_t, b_t] is converted using r_tok and flooring at the endpoints.
  • The hit ratio measures whether a KV-head’s top-K audio positions fall within the aligned audio interval.K is chosen to cover roughly one spoken word: K=4 for Voxtral and K=8 for Qwen2.5-Omni.
  • Voice scores are averaged over answer tokens with alignment confidence ≥0.85 and 50 calibration samples, then aggregated over GQA groups.
  • The reported calibration setup uses author-recommended baseline hyperparameters, with min_word_score=0.85 shared symmetrically by AudioKV and WnW.
Loading 2608.22704v1…