PaperlayerView exact revision on arXiv

Source-linked AI summary

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer, Rishabh Iyer, Ion Stoica

arXiv:2609.38166v1cs.LGcs.AI

TL;DR

Linear-attention states reduce long-context costs, but repeatedly transferring them burdens inference, and low-bit storage can impair accuracy through recurrent error and outliers. LeapQuant combines per-window quantization with high-precision outlier compensation and residual smoothing. Across model families, it reports accuracy comparable to FP32 with lower memory costs and faster inference.

  • Problem

    Repeated recurrent-state transfers impose memory bandwidth and capacity costs, while low-bit quantization can degrade accuracy through accumulated rounding errors and state outliers.

    Source: §1 · pp. 1–2

  • Method

    LeapQuant quantizes the recurrent state once per window and uses high-precision Compensator Tokens for outliers before smoothing the residual.

    Source: §1 · p. 2

  • Results

    Across 12 model–task pairs, LeapQuant achieves accuracy comparable to FP32 while reducing state memory traffic 3.4× and end-to-end memory footprint up to 56%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue LeapQuant can make low-precision recurrent states a practical serving default as hybrid models devote more layers to linear attention.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70$\times$ at the kernel level and 1.47$\times$ for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

1 INTRODUCTION

Hybrid LLMs use linear attention to compress token history, but recurrent-state transfers remain a decoding and memory cost, while naive low-bit quantization can degrade accuracy. LeapQuant combines per-window quantization with high-precision outlier compensation and residual smoothing; experiments report accuracy comparable to FP32 alongside reduced memory costs and faster inference.

1 INTRODUCTION

  • Hybrid LLMs use recurrent linear-attention states to compress token history, but each decode step transfers the full state, making HBM bandwidth and prefix-cache capacity costly.The state transfer repeats at every decoding step, and cached prefixes retain separate states.Source: §1 · p. 1
  • Naive per-token low-bit quantization can degrade accuracy as recurrent rounding errors accumulate and state outliers widen the quantization range.Outliers concentrated in a few rows and columns force smaller values onto coarser quantization levels.Source: §1 · p. 2
  • LeapQuant quantizes the state once per token window, computing outputs from a fixed low-precision boundary state and buffered higher-precision updates.It reconstructs and quantizes the full updated state only at the window's end.Source: §1 · p. 2
  • High-precision Compensator Tokens capture large state outliers, while residual smoothing further reduces quantization error.The compensators share the real-token update path and require no separate recurrent state update.Source: §1 · p. 2
  • Across 12 model–task pairs, LeapQuant matches FP32 accuracy, cuts state memory traffic 3.4× and footprint up to 56%, and averages 2.05–3.70× kernel and 1.47× end-to-end speedups.The evaluation spans the Qwen, Kimi, and GLM families and NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.Source: §1 · p. 2

2 RELATED WORK

Related work covers recurrent linear attention and hybrid architectures, quantization methods for weights, activations, caches, and SSM states, and serving mechanisms for state management. LeapQuant's per-window approach resembles checkpoint-and-replay methods but applies that structure to quantization accuracy.

2 RELATED WORK

  • Linear attention replaces softmax attention with a recurrence over a fixed-size state; later gating and delta-rule methods support hybrid LLM architectures, whose prior work focused on architecture and full-precision kernels.The cited discussion notes that recurrent-state storage precision was not the focus of those works.Source: §2 · pp. 2–3
  • Prior quantization work targets weights, activations, KV caches, and SSM states, but Mamba-specific state methods address a different update mechanism from other linear-attention architectures.Examples include outlier isolation or high-precision components for KV caches and state quantization for Mamba.Source: §2 · p. 3
  • Serving systems use checkpoints because recurrent states are overwritten at each token; ReplaySSM enables speculative rollback, while Marconi stores cached-prefix states at a memory cost.ReplaySSM recomputes recent tokens from a checkpoint; Marconi saves prefix states for hybrid-model caching.Source: §2 · p. 3
  • Per-window quantization also starts from a stored boundary state and replays updates, but uses this structure to improve quantization accuracy rather than speculative rollback.It buffers the window's higher-precision updates and quantizes the reconstructed full state once at the boundary.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD

LeapQuant quantizes recurrent states at window boundaries, preserves dominant outliers in high precision, and smooths the residual before quantization.

3.1 PRELIMINARIES

  • Each head’s recurrent state is a d_k × d_v matrix updated by diagonal decay and a rank-one delta-rule term; β is zero without the delta rule.The read vector β and key vector k are part of the update description.Source: §3.1 · pp. 3–4

3.2 PER-WINDOW QUANTIZATION

  • 100× lower 64K state error for BF16 and 39× for INT8 with 16-token windows versus per-step quantization.Within a window, LeapQuant holds the quantized boundary state fixed, buffers higher-precision token updates, and quantizes the materialized state only at the window end.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 COMPENSATOR TOKENS

  • Compensator Tokens preserve dominant state components in high precision as rank-one updates, leaving a residual to quantize and reusing the real-token update path.They are processed before the next window’s real-token updates without requiring an extra kernel.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4 RESIDUAL SMOOTHING

  • Residual smoothing rescales key rows by square-root mean-absolute-value scales with a positive floor, held fixed within each window and recomputed at each boundary.The inverse transform restores the original key coordinates after dequantization.Source: §3.4 · p. 6

4 EVALUATION

Across Qwen, Kimi, and GLM evaluations, LeapQuant preserves downstream accuracy while reducing recurrent-state memory costs and improving inference throughput. The evaluation covers accuracy, kernel and serving efficiency, and ablations of its design choices.

4.1 SETUPS

  • The evaluation covers five hybrid models using GDN or KDA, downstream accuracy datasets, and inference measurements on B200, RTX PRO 6000, and RTX 5090 GPUs.Eight-layer Qwen3.8-Flash and GLM-5.3-Flash versions are used only for efficiency measurements because they do not preserve model output.Source: §4.1 · pp. 6–7

4.2 ACCURACY RESULTS

  • 8-bit LeapQuant matches FP32 accuracy across 12 model–task pairs, while per-step FP8 scores 14.6% on AIME versus FP32’s 87.9% for Qwen3.5-9B.The downstream-performance table reports the accuracy comparisons; the AIME passage highlights the long-generation gap.Source: Table 1 · p. 7, §4.2 · p. 7
  • At 6 bits, LeapQuant averages 72.4% versus FP32’s 75.5%; at 4 bits, it averages 60.4%, while MXFP4 and TurboQuant average 9.7% and 22.9%.At 4 bits, LeapQuant remains within 1.1% of FP32 on every Kimi-Linear-48B-A3B task; NVFP6 and TurboQuant average 29.6% and 31.9% at 6 bits.Source: Table 1 · p. 7, §4.2 · p. 8

4.3 EFFICIENCY RESULTS

  • At batch size 512, LeapQuant achieves GDN speedups of 2.68×/3.95×/4.25× and KDA speedups of 2.41×/3.38×/3.64× on B200/RTX PRO 6000/RTX 5090.The comparison is against the FP32 FLA kernel for one linear-attention layer.Source: Figure 7 · p. 8, §4.3 · p. 8
  • At 4K, decode throughput rises 1.22–1.37× on B200 and 1.22–1.57× on RTX PRO 6000; output throughput rises 1.23–1.60× and 1.31–1.65×, respectively.The decode measurements use batch size 512 on B200 and the largest fitting batch on RTX PRO 6000.Source: Figure 8 · p. 8, §4.3 · pp. 8–9, Figure 9 · p. 9
  • 1.19 bytes per element instead of 4 reduces boundary-state storage 3.4×; with prefix caching, end-to-end memory falls by 41%, 51%, and 56% on the evaluated models.The reported memory reductions are for the default prefix-caching mode in vLLM and SGLang.Source: §4.3 · pp. 8–9

4.4 ABLATION STUDY

  • Per-step INT8 scores 7.1% on AIME and 9.2% on LiveCodeBench; per-window quantization raises them to 82.4% and 60.6%, and smoothing closes the FP32 gap at 2.52× kernel speedup.Compensator Tokens further raise the INT8 scores to 86.6% and 61.5% before smoothing.Source: Table 2 · p. 9, §4.4 · p. 9
  • p=16 is fastest, with no further accuracy gain at p=32; accuracy saturates at r=4, while r=16 makes the kernel slower than FP32.Longer windows increase record-buffer traffic, and larger compensator counts expose reconstruction overhead.Source: Table 3 · p. 9, §4.4 · p. 9

5 CONCLUSION

LeapQuant is a training-free method for near-lossless recurrent-state quantization in linear attention. It quantizes once per token window, captures state outliers in high-precision compensator tokens, and smooths the residual before quantization. With an 8-bit state, it matches FP32 accuracy on long reasoning and code generation while reducing state memory and accelerating decoding on GDN and KDA models.

5 CONCLUSION

  • LeapQuant combines per-window quantization, high-precision compensator tokens for state outliers, and residual smoothing to quantize recurrent states.The conclusion presents the method as training-free and near-lossless.Source: §5 · p. 10

A INSTANCES OF THE GENERAL UPDATE

The section expresses recurrent updates using common decay, write-key, and read-vector notation, then shows how models with different update structures fit that form. Some updates use one rank-one record per token, while a separate erase direction can be represented with two sub-steps.

A INSTANCES OF THE GENERAL UPDATE

  • The general update uses diagonal decay D_t, write key k̃_t, and read vector b_t to express the model-specific recurrence coefficients.The notation maps D_t to Diag(α_t), k̃_t to k_t, and b_t to β_t.Source: A INSTANCES OF THE GENERAL UPDATE · p. 13
  • The delta rule reads the previous state and subtracts that read from v_t along the write-key direction; plain and gated linear attention set b_t to zero.Source: A INSTANCES OF THE GENERAL UPDATE · p. 13
  • When the erase direction matches the write key, each Gated DeltaNet or KDA token contributes one rank-one record.The step size β_t is carried by the write key in this convention.Source: A INSTANCES OF THE GENERAL UPDATE · p. 13
  • RWKV7, whose erase direction differs from its write key, is represented by splitting each token into an erase and a pure-write sub-step.This gives each token two rank-one records, as for a DeltaProduct token with n = 2.Source: A INSTANCES OF THE GENERAL UPDATE · p. 13

B.1 PROPAGATION OF THE QUANTIZATION-INDUCED DEVIATION

The appendix analyzes how quantization errors propagate and how windowed state reads can be computed. It also specifies how reads combine the quantized residual with real and Compensator Token records, and how state coordinates are restored at window boundaries.

B.1 PROPAGATION OF THE QUANTIZATION-INDUCED DEVIATION

  • Per-window quantization makes quantization-induced error nonzero only at window boundaries, avoiding the coherent accumulation caused by rounding small updates at every step.Rounding the accumulated updates once per window reduces error by far more than the reduction in quantization frequency alone.Source: §B.1 · p. 14

B.2 WINDOW UNROLLING AND STATE READOUT

  • Unrolling the state lets each read avoid forming a d_k × d_v intermediate state, with the base term costing O(d_k d_v) and each record costing O(d_k + d_v).Cumulative diagonal decays can be maintained incrementally.Source: §B.2 · p. 14

B.3 READOUT WITH A SMOOTHED QUANTIZED RESIDUAL

  • The readout adds real-token and Compensator Token records to the quantized-residual read, while applying the residual scales only to the residual contribution.At a window boundary, the state is reconstructed in its original coordinates before fitting new factors and recomputing residual scales.Source: §B.3 · pp. 14–15

C FULL DOWNSTREAM RESULTS

The section describes Table 5 as a full downstream performance comparison across quantization formats and adapted methods at multiple bit widths. The supplied passages specify the table's scope and baseline setup but do not provide its score values.

C FULL DOWNSTREAM RESULTS

  • The comparison includes GSM8K and direct quantization formats alongside adapted methods at 8, 6, and 4 bits.Direct formats appear before adapted methods in each group.Source: C FULL DOWNSTREAM RESULTS · p. 15
  • All baselines re-quantize the state at every decode step, while LeapQuant uses the configurations specified for each bit width.Source: C FULL DOWNSTREAM RESULTS · p. 15
  • Table 5 reports full downstream performance as percentages, with higher scores better and best scores bolded within each bit-width column.Source: Table 5 · p. 15

D ADDITIONAL EFFICIENCY EVALUATION

The efficiency evaluation reports decode-step throughput gains for dense- and sparse-attention hybrid models at longer contexts. The reported gains remain substantial at 8K for a dense-attention model and at the longest tested contexts for two sparse-attention models.

D ADDITIONAL EFFICIENCY EVALUATION

E EVALUATION DETAILS

The downstream evaluation uses shared settings, prompts, and samples across methods of each model, with model-specific generation settings and a context length sized to avoid premature truncation.

E EVALUATION DETAILS

  • All methods of a model share the same generation settings, prompts, and sample set.Source: E EVALUATION DETAILS · p. 16
  • Qwen3.5 uses official model-card benchmark settings with thinking enabled, while Kimi-Linear-48B-A3B-Instruct has no thinking mode.Source: E EVALUATION DETAILS · p. 16
  • The maximum model length is the output limit plus 8,192 tokens, preventing context-window truncation before the output limit.Source: E EVALUATION DETAILS · p. 16
  • Table 6 lists the generation settings used for downstream evaluation.Source: Table 6 · p. 16

Related papers

Loading 2609.38166v1…