Source-linked AI summary
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Yanqi Yu, Pingwei Sun, Jianchao Tan, Tao Zhang, Yuchen Xie, Xunliang Cai, Yao Liu
TL;DR
Hybrid linear-attention prefix caching must preserve overwritten recurrent states alongside full-attention KV, creating a memory and replay trade-off. DASC selects long-horizon units from model-weight decay, stores them in balanced ragged checkpoints, and optionally refreshes omitted units from a bounded suffix. Conservative configurations stay near full-cache quality while increasing checkpoint capacity and improving serving efficiency; the approach remains limited to tested hybrid architectures and TP8.
Problem
Overwritten recurrent states require checkpoints for prefix reuse, but full checkpointing increases HBM pressure while sparse checkpointing increases replay or repeated prefill.
Method
DASC derives input-independent retention horizons from weights, retains long-horizon GDN heads or KDA channels, packs balanced ragged checkpoints, and optionally suffix-refreshes omitted units.
Results
Conservative DASC remains close to full-cache quality while fitting 2.63× more KDA checkpoints and reducing mean TTFT by 42.6% and increasing input throughput by 68.4% under matched state-memory budgets.
Takeaways & Limitations
Suffix replay recovers much of the accuracy lost at aggressive compression, and Qwen with GDN shows a similar quality–efficiency trend.
Takeaways & Limitations
Evaluation covers hybrid architectures at TP8; larger TP degrees remain untested, and GDN yields smaller capacity and serving gains than channel-grained KDA.
Abstract
from arXiv · showhide
Hybrid linear-attention architectures have recently scaled to large open-weight models, offering quality competitive with full attention while substantially reducing key/value (KV) cache growth. However, their in-place recurrent-state updates complicate cache management: prefix reuse requires state checkpoints alongside full-attention KV, while storing state checkpoints in full increases memory pressure, leading to more evictions and repeated prefill. By analyzing the decay structure of Gated DeltaNet (GDN) and Kimi Delta Attention (KDA), we find that different heads and channels retain prefix information over markedly different timescales, which we term \emph{retention horizons}. This variation suggests substantial compression potential in persistent state checkpoints. Building on this observation, we introduce \emph{Decay-Aware State Compression} (DASC), which derives retention horizons from model weights, selects long-horizon state units, and packs them into a ragged state checkpoint layout. To integrate efficiently with tensor-parallel inference engines, DASC furtherly balances compressed state checkpoints across TP ranks. On reuse, DASC either zero-fills omitted units or refreshes them from a bounded suffix with additional compute cost. Across retrieval and end-to-end reasoning benchmarks on Kimi-Linear, conservative DASC configurations remain close to full caching while compressing KDA recurrent state checkpoints by $2.63\times$. Under fixed state checkpoint memory budgets, the resulting capacity gains reduce mean Time to First Token (TTFT) by 42.6\% and improve input throughput by 68.4\%. At larger compression ratio, suffix refresh recovers much of the accuracy lost to more aggressive omission, at the cost of additional replay computation. Qwen with GDN exhibits a similar quality--efficiency trend, showing that DASC extends from channel-wise KDA to head-wise GDN.
DASC: DECAY-AWARE STATE COMPRESSION FOR HYBRID LINEAR-ATTENTION SERVING
The supplied passages list the paper’s authors: Yanqi Yu, Pingwei Sun, Jianchao Tan, Tao Zhang, Yuchen Xie, Xunliang Cai, and Yao Liu.
- The author list begins with Yanqi Yu, Pingwei Sun, Jianchao Tan, and Tao Zhang.
- The supplied author listing includes affiliation markers and contribution symbols.
- Yuchen Xie, Xunliang Cai, and Yao Liu are also listed as authors.
1 INTRODUCTION
Hybrid linear-attention models reduce KV-cache growth but make prefix caching difficult because recurrent states are overwritten and must be checkpointed. DASC uses static decay-derived selection and compressed, balanced checkpoints, with suffix refresh available when omission is more aggressive.
- Hybrid architectures combine recurrent summaries with full-attention layers, retaining strong quality while slowing KV-cache growth.
- Full recurrent-state checkpointing can exhaust HBM, while sparse checkpointing increases replay or repeated prefill.
- DASC derives an input-independent plan that retains long-horizon GDN heads or KDA channels, packs them into ragged checkpoints, and balances them across TP ranks.
- On reuse, DASC-NR zero-fills omitted units, while DASC-WR refreshes them from a bounded suffix with additional computation.
- 2.63× more recurrent state checkpoints fit the same state-memory budget, with 42.6% lower mean TTFT and 68.4% higher input throughput on Kimi-Linear.
2 RELATED WORK
Prior work addresses linear-attention states, prefix-cache organization, and recurrent-state replay, whereas this paper studies selecting and compressing recurrent states for hybrid serving.
- Linear-attention and state-space models replace context-length-dependent KV caches with fixed-size recurrent states, while hybrids interleave recurrent and full-attention layers.
- Prefix-cache systems organize, allocate, extend, quantize, evict, or sparsify reusable full-attention KV storage.
- Related hybrid-serving methods persist recurrent states, optimize prefix admission and eviction, or reconstruct states from cached recent inputs.
3 RETENTION HETEROGENEITY IN HYBRID RECURRENT STATES
Recurrent-state units exhibit heterogeneous decay, motivating weight-derived retention horizons as an input-independent selection signal. Interventions and correlations show compression opportunity and support the signal’s relation to observed decay and activity.
- Recurrent prefix caching requires storing each layer’s boundary state or recomputing it through prefill when the checkpoint is absent.
- Removing full-attention KV reduces NIAH-S1 accuracy to zero, whereas removing recurrent-state checkpoints changes accuracy by at most one point.
- The static horizon fixes the input gate to a=-0.3 and measures tokens until the decay factor falls to ϵ, producing an input-independent metric from learned decay parameters.
- KDA exposes channel-wise decay while GDN uses head-wise decay; KDA horizons vary broadly, and 90.6% of KDA heads span both sides of a 16-token cutoff.
- For KDA, readout magnitude measures per-channel activity rather than each channel’s net contribution to the summed output.
- Hs correlates with empirical horizons at 0.801 for GDN and 0.856 for KDA, while associations with state and readout magnitudes are weaker.
4 DECAY-AWARE STATE COMPRESSION
DASC compresses recurrent state checkpoints by retaining long-horizon units, packing them raggedly, and balancing storage across tensor-parallel ranks. On reuse, it loads retained units exactly while either zero-filling or suffix-refreshing omitted units.
- Decay-aware selection: DASC uses static, weight-derived horizons to retain long-horizon units and omit short-horizon units from persistent checkpoints.Selection is performed offline without calibration prompts or online profiling; granularity is head-wise for GDN and channel-wise for KDA.
- Ragged storage: Ragged storage keeps only retained units per layer rather than reserving the largest retained count for every layer.Its layout gain over padded storage is Rmax/r, with larger gains expected for KDA’s imbalanced retained counts.
- Load policies: On cache hits, DASC loads retained global units exactly, while DASC-NR zero-fills omitted units and DASC-WR refreshes them from a bounded suffix.The stored global checkpoint and cached full-attention KV remain unchanged during suffix refresh.
- Load policies: Increasing Wmax improves compression but can increase DASC-NR approximation error and DASC-WR replay cost.DASC-WR remains approximate because it cannot recover local-state history preceding the replay window.
- TP-balanced placement: DASC balances compressed checkpoint payloads because tensor parallelism limits resident checkpoints by the most heavily loaded rank.It first permutes head bundles at model load, then redistributes packed units during checkpoint storage and loading.
5 EXPERIMENTS
Experiments evaluate DASC across retrieval, end-task quality, state-checkpoint capacity, matched-HBM serving, and suffix-refresh trade-offs on Kimi-KDA and Qwen-GDN. Conservative compression stays close to dense quality while substantially improving checkpoint capacity and serving efficiency.
- Setup: Experiments use Qwen-GDN and Kimi-KDA on Hopper GPUs, with TP8, unquantized BF16 execution, and mixed-precision recurrent states.The models expose decay at head and channel granularity, respectively.
- Retrieval quality: At Wmax = 16, every displayed RULER aggregate remains within 0.01 of dense for both Kimi-KDA and Qwen-GDN.At Wmax = 1024, Kimi-KDA loses 0.04/0.06/0.07 across 4k/8k/16k, while Qwen-GDN aggregates remain within 0.01 despite a 16k MK3 drop from 1.00 to 0.87.
- End-task quality: At Wmax = 16, DASC stays within 1.3 percentage points of dense on every end-task row and beats count-matched Random on all 15 Kimi-KDA reasoning comparisons.Seven Kimi-KDA differences remain significant after Holm correction; Qwen-GDN stays within 1.7 points of dense.
- State checkpoint capacity: 8.11× state checkpoint capacity results from combining Wmax = 16 channel selection, INT8 checkpoint storage, and DASC-NR on five Kimi-KDA end tasks.DASC reduces retained units, while quantization reduces bits per retained value, making the techniques complementary.
- State checkpoint capacity: 2.63–28.04× KDA compression and 1.10–2.48× GDN compression are achieved across Wmax = 16–512 with TP-balanced placement.The larger KDA range reflects channel-wise selection, whereas GDN drops whole heads; balancing reduces the worst-rank footprint.
- Matched-HBM serving: 42.6% lower TTFT and 68.4% higher input-token throughput are achieved for KDA under matched workload and HBM controls.At Wmax = 16, TTFT falls from 567.6 to 326.0 ms and throughput rises from 33.25 to 55.98 k tokens/s; aggressive compression can incur replay and communication costs.
6 LIMITATIONS
The evaluation is limited to hybrid architectures and TP8 serving, leaving transfer to other model classes and larger tensor-parallel degrees untested. Benefits also depend on architecture, with GDN yielding smaller gains than channel-grained KDA.
- Hybrid architectures are evaluated exclusively, so transfer to purely linear-attention or state-space models remains open.
- All serving experiments use TP8, leaving scaling to larger tensor-parallel degrees untested.
- GDN provides smaller state-checkpoint capacity and serving gains than channel-grained KDA because it exposes head-grained decay.
7 CONCLUSION
DASC converts retention heterogeneity into compressed, TP-balanced state checkpoints for hybrid prefix caches. On Kimi-KDA, it preserves near-dense quality while improving checkpoint capacity and serving efficiency, with layout and workload protocols explicitly controlling these comparisons.
- DASC combines decay-aware selection, ragged storage, and tensor-parallel-balanced placement for hybrid prefix caches.
- 2.63× state checkpoint capacity, 42.6% lower mean TTFT, and 68.4% higher input-token throughput are achieved on Kimi-KDA under matched HBM.
- Retention maps vary across KDA channels and GDN heads, supporting architecture-specific selection granularities and motivating ragged storage for layer-dependent widths.
- Buffer layout: Padded layouts allocate every layer against the widest retained-unit count, whereas ragged layouts store only each layer’s retained units.
- Buffer layout: KDA’s imbalanced per-channel retention makes ragged temporal-layout gains larger, while GDN’s per-head fractions are comparatively even.
- Serving evaluation: The serving study uses TP8, mixed-precision recurrent checkpoints, matched state-memory budgets, controlled cache-fill/replay phases, and excludes long-decode throughput.
C.3 QUALITY-EVALUATION PROTOCOLS
Quality evaluation separates strict warm/replay accuracy from normal concurrent serving and pairs cache-arm comparisons using shared instances, prompts, and seeds. Statistical units and benchmark-specific protocols are defined to avoid treating repeated generations as independent evidence.
- RULER: RULER uses N = 30 unique instances per subtask–length–arm setting, with one warm-up pass and three measured replay rounds averaged per instance.
- Protocol separation: Strict warm/replay accuracy in Figure 3(g–i) is separate from Table 3’s normal concurrent-serving stream and matched-HBM TTFT workload.
- Paired evaluation: Paired inference uses shared seeds and the unique question as the statistical unit, reporting replay accuracy and dense-relative paired deltas on intersecting loaded pairs.
- End-task suite: The five-benchmark end-task suite evaluates paired prompts and sampling seeds through the normal concurrent serving path.
- LoCoMo: LoCoMo is evaluated separately with one greedy decode per question because it is a conversational-memory QA benchmark.
- Random-selection comparison: DASC is compared with five random masks using paired problem-level differences and 10,000 bootstrap resamples, with Holm adjustment over 15 comparisons.
D COMPLETE QUALITY RESULTS
The complete quality results are reported as replay-only RULER sweeps for Kimi-KDA and Qwen-GDN, alongside the state-quantization procedure. Quantized checkpoints are restored once on reuse, while active recurrence remains in native precision.
- Tables 6 and 7 provide the complete replay-only RULER sweeps for Kimi-KDA and Qwen-GDN, respectively.
- State quantization: Cached FP32 temporal recurrent state is quantized symmetrically to INT8, while convolution state remains BF16 and active temporal recurrence remains FP32.
- State quantization: On cache reuse, each quantized checkpoint is dequantized once into a fresh active FP32 temporal-state slot before recurrence and decoding.
E.2 END-TASK COMPOSITION PROTOCOL
The protocol compares dense mixed-precision state checkpoints with ragged DASC-NR and isolates suffix-refresh overhead using fixed-capacity, paired workloads.
- E.2 END-TASK COMPOSITION PROTOCOL: 8.11× state checkpoint capacity is achieved by ragged DASC-NR at Wmax = 16 relative to the dense mixed-precision reference.Selected temporal state uses symmetric INT8 storage and is dequantized once into active FP32 state on cache hits; full-attention KV remains unchanged.
- E.2 END-TASK COMPOSITION PROTOCOL: 899 state checkpoint slots are fixed across all Wmax values in the matched Kimi-KDA suffix-refresh diagnostic.DASC-NR and DASC-WR use the same 200-conversation, 1,051-prefix workload with five paired request-order seeds.
- E.2 END-TASK COMPOSITION PROTOCOL: Approximately 0.80 hit rates are maintained across arms, while Table 8 reports DASC-WR minus DASC-NR latency differences and per-hit normalization.The fixed-slot design separates reconstruction cost from cache-capacity effects.