PaperlayerView exact revision on arXiv

Source-linked AI summary

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zixu Jiang, Yuchen Xie, Xunliang Cai, Ziqian Zeng

arXiv:2608.27513v1cs.LGcs.AI

TL;DR

Recurrent states in GDN- and KDA-based language models consume memory and decoding bandwidth, while uniform quantization harms accuracy. DAMP selects high-risk channels using quantization-error energy and decay persistence, preserving near-FP32 accuracy at 9.9 bits per state value while reducing storage and latency.

  • Problem

    GDN- and KDA-based models use fixed-size recurrent states, but FP32 storage and full-matrix updates consume substantial GPU memory and decoding bandwidth.

  • Method

    DAMP ranks recurrent-state key channels by calibration quantization-error energy and decay-based persistence, storing selected channels at higher precision and the remainder in INT8.

  • Results

    At 9.9 bits per state value, DAMP largely preserves FP32 accuracy across six benchmarks while reducing state storage by 69.1%, recurrent-update latency by up to 2.01×, and full-model TPOT by up to 10.9%.

  • Takeaways & Limitations

    Effective recurrent-state compression requires jointly designing numerical precision allocation and systems layout.

  • Takeaways & Limitations

    Accuracy and systems gains may vary across architectures, hardware, and serving regimes, and INT4 or NVFP4 low-precision tiers do not yet recover FP32 accuracy.

Abstract

from arXiv · show

Softmax attention stores key and value vectors for every preceding token, causing inference memory to grow with sequence length. Recent language models incorporating Gated DeltaNet (GDN) or Kimi Delta Attention (KDA) reduce this cost by replacing the KV cache in most layers with fixed-size recurrent states. However, these recurrent states are commonly stored in FP32 and consume substantial GPU memory; their updates are memory-bandwidth bound and contribute significantly to decoding latency. To our knowledge, we are the first to study post-training quantization of recurrent states in GDN and KDA based language models. We find that uniform quantization provides a poor accuracy--storage trade-off: INT8 and FP8 already degrade accuracy on complex reasoning tasks, while INT4 and NVFP4 reduce it to near zero. We further find that most quantization-error energy is concentrated in a small subset of channels and that the relative decay strength of state channels remains stable across prompts and tasks. Motivated by these findings, DAMP uses both quantization-error energy and decay-based persistence to identify high-risk channels during offline calibration. It stores these channels at higher precision and the remainder in INT8. We evaluate DAMP on Qwen3.6-35B and Kimi-Linear-48B across six benchmarks covering mathematical reasoning, general reasoning, and code generation. At 9.9 bits per state value, DAMP maintains average accuracy close to the FP32 baseline. DAMP reduces recurrent-state storage by 69.1%, accelerates the recurrent-state update kernel by up to 2.01x, and lowers full-model TPOT by up to 10.9%.

1 INTRODUCTION

DAMP addresses the memory and decoding costs of FP32 recurrent states in GDN- and KDA-based language models by assigning higher precision to channels with greater quantization-error and decay-based risk. Across Qwen3.6-35B and Kimi-Linear-48B, it preserves accuracy near FP32 while reducing state storage and accelerating inference.

  • Motivation: Softmax attention makes KV-cache memory grow with sequence length, while FP32 recurrent states in Qwen3.6-35B occupy 15 GB and account for 24.3% of decoding cost.Recurrent-state updates read and write the full matrix at every decoding step, making them memory-bandwidth bound.
  • Quantization findings: Uniform quantization has a poor accuracy–storage trade-off: INT8 and FP8 degrade complex-reasoning accuracy, while INT4 and NVFP4 reduce it to near zero.The paper reports these degradations relative to FP32-state inference.
  • Method: DAMP ranks channels by quantization-error energy and decay-based persistence, stores high-risk channels at higher precision, and stores the remainder in low precision.The layout is calibrated once and reused during inference without retraining or token-wise selection.
  • Results: At 9.9 bits per state value, DAMP remains close to the FP32 baseline across mathematical, general-reasoning, and coding benchmarks on both evaluated models.The evaluated models are Qwen3.6-35B-A3B and Kimi-Linear-48B-A3B-Instruct.
  • Results: 69.1% storage reduction, up to 2.01× recurrent-update acceleration, and up to 10.9% lower full-model TPOT are achieved relative to FP32-state inference.These results are reported across the evaluated Qwen3.6-35B and Kimi-Linear-48B models.

2 RELATED WORK

Prior quantization work compresses transformer weights, activations, and KV caches, while recurrent-state methods mainly target Mamba’s selective SSM or reduce linear-attention state costs structurally. Unlike write-once KV entries, recurrent states are repeatedly quantized, motivating distinct treatment.

  • Transformer and KV-cache quantization: Transformer quantization methods compress weights, activations, and KV caches using scaling, outlier handling, and rotations.These approaches address fixed weights and write-once KV entries, whereas recurrent states are repeatedly quantized.
  • Quantization and compression of recurrent states: Quamba and MambaQuant quantize Mamba weights and transient activations, while Quamba2 and Q-Mamba additionally quantize cached Mamba states.These methods are developed for Mamba’s selective SSM.
  • Quantization and compression of recurrent states: Related linear-attention work reduces state size through structural channel pruning or accelerates low-precision triangular inversion in chunkwise GDN execution.These approaches address linear-attention state compression or execution efficiency rather than recurrent-state quantization generally.

3 RECURRENT-STATE QUANTIZATION IN GDN AND KDA

GDN and KDA replace prefix-dependent KV storage with fixed-size recurrent state matrices updated by decay and key-specific delta corrections. Quantization introduces state-write errors whose accumulation depends on both injected error and retention through subsequent recurrent updates.

  • Recurrent-state formulation: Each recurrent head summarizes the processed prefix in a fixed-size d_k × d_v state matrix indexed by key channel.The state associated with key channel u is d_v-dimensional and is read by the current query.
  • Recurrent-state formulation: The recurrent update first applies learned decay, then writes a key-specific delta correction toward the current value.The delta-rule step size β_t controls the strength of this correction.
  • Decay structure: KDA uses per-key-channel decay, whereas GDN shares one scalar decay within each head.KDA sets Λ_t = diag(exp(g_t)), while GDN sets Λ_t = α_tI, making decay a channel-specific retention signal only in KDA.
  • State quantization: States are computed in the model’s default precision, then quantized before persistent GPU storage using per-block asymmetric affine quantization.The evaluated integer formats include INT8 with b = 8 and INT4 with b = 4.
  • Quantization-error propagation: Accumulated state error combines error introduced at each low-precision write with retention of earlier errors through later recurrent transitions.The transition contains both the rank-one correction, which can redistribute error across key channels, and decay, which controls retention; retention is channel-specific in KDA and shared within each GDN head.

4 STRUCTURE IN QUANTIZATION ERROR AND DECAY

Quantization risk is structured: residual error concentrates in a small subset of key channels, while decay reveals stable, channel- or head-level differences in error persistence. These orderings remain consistent across tasks and calibration samples despite token-level fluctuations.

  • Quantization error: Quantization error remains concentrated across key channels after Hadamard transforming, with a small subset accounting for most residual INT8 error.Hadamard transformation reduces within-channel range imbalance but does not equalize error across key channels.
  • Decay persistence: KDA retention varies sharply token-to-token but differs strongly across key channels, motivating geometric-mean retention as a static persistence signal.The geometric mean is defined as aeff,u = exp(Et[log at,u]).
  • Cross-task stability: KDA decay ordering is reproducible across tasks, with Spearman correlations of 0.994 and 0.999 for Code and General relative to Math.The top-16 KDA key-channel selections from disjoint, domain-balanced calibration splits overlap by 92.0% on average.
  • Architecture differences: GDN exhibits a broad and stable decay spectrum at head level, because its single decay scalar is shared across all key channels within a head.Unlike KDA, GDN decay cannot distinguish individual key channels.
  • Summary finding: KDA key channels and GDN heads span broad retention strengths, with ordering consistent across samples and tasks despite token-level fluctuations.This supports estimating error persistence during offline calibration.

5 DAMP: DECAY-AWARE MIXED-PRECISION STATE QUANTIZATION

DAMP constructs a static mixed-precision recurrent-state layout by ranking key channels using quantization-error risk and decay-based persistence during offline calibration. It packs protected high-precision channels with low-precision channels and reuses the fixed layout in fused recurrent updates.

  • DAMP workflow: DAMP ranks key channels by accumulated-error risk during offline calibration, selects a protected set under a fixed storage budget, and packs precision tiers for fused recurrent updates.The risk combines error introduced at state writes with estimated persistence along the diagonal decay path.
  • Key-channel budget: For each layer and head, DAMP protects Khi key channels in high precision and stores the remaining channels in low precision.The shared Khi budget controls storage cost across layers and heads.
  • Accumulated-error risk: DAMP scores each channel as Cu = EuPu, combining quantization-error energy with a capped persistence estimate based on geometric-mean decay.The persistence model uses per-step retention aeff,u and maximum persistence τ.
  • Precision configuration: 9.9 bits per state value is achieved with FP16 for protected channels and INT8+Hadamard for the remaining channels, using Khi = 16.The configuration uses bhi = 16, blo = 9.0, and dk = 128.
  • Packed execution: DAMP applies a fixed per-layer, per-head permutation that places protected channels before their complement, enabling contiguous FP16 and INT8 regions reused for every input.A fused executor reconstructs INT8 states, performs the recurrent update in FP32, recomputes quantization parameters, and writes the packed state.

6 EXPERIMENTS

Experiments evaluate DAMP on Qwen3.6 and Kimi-Linear across mathematical reasoning, general reasoning, and code-generation benchmarks. At matched storage budgets, DAMP improves accuracy over uniform quantization and accelerates recurrent updates and end-to-end decoding.

  • Models: DAMP is evaluated on Qwen3.6-35B-A3B and Kimi-Linear-48B-A3B, covering GDN and KDA hybrid mixture-of-experts models at comparable activated scale.Qwen3.6 contains 30 GDN and 10 full-attention layers, while Kimi-Linear contains 20 KDA and 7 MLA layers.
  • Accuracy: 21.60 points: DAMP improves Qwen3.6 mean accuracy over uniform INT8+Hadamard across six benchmarks.On Kimi-Linear, DAMP improves AIME 2026 accuracy by 8.00 points, from 55.72 to 63.72, and raises the six-benchmark average by 3.26 points.
  • Accuracy: FP16 remains closest to FP32, INT8+Hadamard outperforms FP8 at 9.0 bits, and INT4 and NVFP4 nearly collapse mathematical reasoning and code generation.The precision ordering is consistent across both GDN and KDA checkpoints, while Hadamard range equalization improves INT8.
  • Efficiency: 1.46×–1.65×: DAMP accelerates GDN state updates across batch sizes 32–256, while KDA updates improve by 1.81×–2.01×.At batch size 256, TPOT reductions reach 10.9% for GDN and 9.6% for KDA.
  • Ablation: 63.72, 64.64, and 61.02: DAMP reaches these KDA accuracies on AIME 2026, GPQA-Diamond, and LiveCodeBench-v6 in the key-channel selector ablation.All selectors use the same INT8+Hadamard base quantizer, isolating the key-channel selection rule at 9.9 bits per state value.

Selector AIME GPQA LCB

The selector’s accuracy rises steeply up to Khi = 16 and largely saturates thereafter, making Khi = 16 the shared operating point. This protects 12.5% of key channels at 9.9 bits per state value while reaching 83.65 AIME 2026 accuracy for Qwen3.6 and 63.72 for Kimi-Linear.

  • Selector AIME GPQA LCB: Accuracy rises steeply through Khi = 16 and largely saturates thereafter for both models.The precision-budget sweep varies the fraction of key channels retained in FP16 while keeping the selector fixed.
  • Selector AIME GPQA LCB: Khi = 16 is the shared operating point, protecting 12.5% of key channels at 9.875 bits per state value, reported as 9.9.This operating point is used in the main experiments.
  • Selector AIME GPQA LCB: 83.65 AIME 2026 accuracy is reached for Qwen3.6, versus 63.72 for Kimi-Linear.Both results use the shared Khi = 16 operating point.

7 CONCLUSION

DAMP is a post-training method that compresses recurrent states in GDN- and KDA-based language models using calibration error energy and decay-based persistence to assign static mixed precision. At 9.9 bits per state value, it largely preserves FP32 accuracy while reducing storage and accelerating recurrent-state updates.

  • Method: DAMP ranks key channels by calibration quantization-error energy and decay-based persistence, then assigns them a static mixed-precision layout.The method targets recurrent states in GDN- and KDA-based language models.
  • Accuracy: At 9.9 bits per state value, DAMP largely preserves FP32 accuracy across both models and all six benchmarks.The benchmarks span the evaluated GDN- and KDA-based models.

A ADDITIONAL METHOD DETAILS … A.5 PACKED LAYOUT AND FUSED UPDATE

The appendix specifies DAMP’s quantizers, calibration and persistence scoring, offline protected-channel layout, and packed fused execution. Together, these details define a fixed mixed-precision representation and an update path that reconstructs, requantizes, and stores recurrent states efficiently.

  • A.1 STORAGE QUANTIZERS: DAMP’s storage quantizers use contiguous value-dimension groups, with parameters recomputed on every state write; INT8 and INT4 use groups of 32 values.INT8 and INT4 use asymmetric affine mapping with FP16 scale and zero-point metadata, while Hadamard variants transform each block before quantization.
  • A.1 STORAGE QUANTIZERS: DAMP retains 16 of 128 key channels in FP16 and stores the remainder with the 9-bit INT8+Hadamard format.The evaluated state dimensions require no group padding, and the fixed permutation is stored once with the checkpoint.
  • A.2 CALIBRATION PROTOCOL: Calibration samples 32 Pile validation documents across DM Mathematics, PubMed Abstracts, GitHub, and StackExchange, producing 1,024 state samples per recurrent layer.Documents are truncated to 256 tokens, evaluated once with teacher forcing, and states are sampled every eight tokens.
  • A.2 CALIBRATION PROTOCOL: Calibration uses only state and decay statistics from the corpus, applies estimates independently per layer and head, and fixes key-channel indices after calibration.The resulting layout is reused for all inference requests, without downstream evaluation data.
  • A.3 DECAY-BASED PERSISTENCE: DAMP estimates decay-based persistence by replacing future log-retention with its calibration expectation, forming an effective multiplier and summing its squared-energy profile with cap 1/τ.The resulting persistence score is denoted P_u in Equation 9.
  • A.4 OFFLINE LAYOUT CONSTRUCTION: Offline layout construction computes E and P, combines them as C = E ⊙P, and selects the top K_hi channels independently for every layer and head.At the main operating point, K_hi is 16; a stable partition places protected channels before the remaining channels.
  • A.5 PACKED LAYOUT AND FUSED UPDATE: The packed layout applies the calibrated permutation to states and key-channel-indexed operands, placing K_hi = 16 protected channels before low-precision channels in contiguous precision regions.The KDA executor uses value-block-major organization, co-locating each 32-value code tile with its scale and zero point.
  • A.5 PACKED LAYOUT AND FUSED UPDATE: The fused CUDA executor performs each state transition in one kernel, evaluates the recurrence in FP32, then derives affine parameters, requantizes the low-precision tier, and writes both regions directly.The executor uses coalesced transactions, UINT8 unpacking and saturating packing, register-retained tiles, and shared-memory staging.

B LONG-CONTEXT EVALUATION … C.2 GDN DECAY STRUCTURE

DAMP remains nearly indistinguishable from FP32 on RULER across 4K–128K contexts and selects key channels consistently across calibration splits. GDN decay is stable at the head level across tasks, but does not order key channels.

  • B LONG-CONTEXT EVALUATION: Across 4K–128K, DAMP’s maximum absolute gap from FP32 is 0.04 percentage points on Qwen3.6 and 0.02 on Kimi-Linear.RULER macro accuracy is compared across FP32, FP16, uniform INT8+Hadamard, and DAMP, with each format applied to all state writes during prefill and decoding.
  • B LONG-CONTEXT EVALUATION: Uniform INT8+Hadamard has larger maximum absolute gaps from FP32: 0.13 percentage points on Qwen3.6 and 0.15 on Kimi-Linear.These values are reported over the same 4K–128K RULER context-length range.
  • C.1 CALIBRATION-SPLIT AGREEMENT: The calibration-split analysis compares top-16 key-channel overlap and complete rankings for layouts built from two disjoint, domain-balanced subsets.The comparison is performed for every layer and head, with top-16 overlap defined as |H1 ∩ H2|/16.
  • C.1 CALIBRATION-SPLIT AGREEMENT: KDA and GDN retain 92.0% and 93.2% of selected key channels across calibration splits, versus 12.5% expected from random selection.Their complete rankings are also described as consistent across the two splits.
  • C.2 GDN DECAY STRUCTURE: GDN’s effective-decay distribution across 960 layer–head units has p10, p50, and p90 values of 0.376, 0.967, and 0.9996.Because each GDN head shares one scalar decay, effective decay describes a head rather than an individual key channel.
  • C.2 GDN DECAY STRUCTURE: Across tasks, GDN head-decay ordering agrees with Math at Spearman correlations of 0.998 for Code and 0.999 for General.The result indicates stable head-level timescales, while GDN provides no decay-based ordering of key channels.

D GDN SELECTOR ABLATION · E COMMON PER-HEAD BUDGET FOR GDN · F LIMITATIONS

The GDN ablation favors the shared E/EP selector over persistence alone, while common per-head budgets enable fixed-shape execution and avoid decay-based reallocation that does not improve AIME 2026 accuracy. Limitations include architecture- and serving-dependent gains, incomplete modeling by persistence, and unrecovered FP32 accuracy for INT4 or NVFP4 tiers.

  • D GDN SELECTOR ABLATION: On GPQA-D, the selectors span 80.98–81.50, with the shared E/EP selector highest at 81.50; persistence alone performs near randomly on AIME 2026.Because persistence is shared within each head, E and EP induce the same within-head key-channel ranking.
  • D GDN SELECTOR ABLATION: At 9.9 bits per state value, Table 6 reports the matched-budget GDN selector ablation.The comparison evaluates state-energy and persistence-informed selection under the same storage budget.
  • E COMMON PER-HEAD BUDGET FOR GDN: Common per-head budgets place each head’s Khi protected key channels in FP16 and its remaining dk − Khi channels in INT8, enabling uniform fused-executor strides and fixed request-slot storage.The protected indices remain layer- and head-specific, but the count is shared across heads.
  • E COMMON PER-HEAD BUDGET FOR GDN: Because GDN shares one decay factor across a head, decay cannot rank key channels within that head; quantization-error energy determines the within-head ranking.The head-budget ablation instead tests decay-guided allocation across fast- and slow-decay head groups.
  • E COMMON PER-HEAD BUDGET FOR GDN: Allocating more key channels to slow-decay heads reduces held-out state and output RRMSE but does not improve AIME 2026 accuracy.Head-level decay captures error retention duration, not injected magnitude or the complete effect on generation, so the main configuration uses a common per-head count.
  • F LIMITATIONS: The evaluation uses two released GDN and KDA checkpoints with an SGLang implementation, so accuracy and systems gains may vary across architectures, hardware, and serving regimes.This limitation concerns the scope of the reported evaluation rather than the quantization method itself.
  • F LIMITATIONS: The persistence score summarizes only the diagonal decay path and does not model the complete time-varying, key-dependent transition.Thus, persistence is an incomplete description of recurrent-state dynamics.
  • F LIMITATIONS: Although DAMP-INT8 largely preserves FP32 accuracy, low-precision tiers using INT4 or NVFP4 do not yet recover FP32 accuracy.The limitation applies specifically to the INT4 and NVFP4 variants.
Loading 2608.27513v1…