PaperlayerView exact revision on arXiv

Source-linked AI summary

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei

arXiv:2609.38169v1cs.CLcs.AIcs.LG

TL;DR

Persistent recurrent states can become a serving-memory bottleneck, and uniform low-bit quantization can sharply degrade accuracy. STEPQuant allocates precision by error magnitude and lifetime and fits spatial scales to row impact and state distributions. Across two models, it closely matches FP32-state accuracy at a nominal 6-bit budget and reduces total serving memory by up to 68.7%.

  • Problem

    Persistent recurrent states are stored for each concurrent request and can exceed model-weight memory, while uniform low-bit quantization sharply degrades model performance.

    Source: §1 · p. 1

  • Method

    STEPQuant combines lifetime-aware bit allocation with key-row-aware dual-axis fitting to quantize Delta-rule recurrent states.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    At a nominal 6-bit budget, STEPQuant closely matches FP32-state accuracy on both models, while its 4-bit configuration outperforms uniform INT8 on both.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With optimized SGLang kernels, 6-bit STEPQuant compresses recurrent-state memory by 5.03× on Qwen and 5.08× on Kimi, reducing total memory by 68.7% and 53.7%, respectively.

    Source: §1 · p. 2

    The lifetime weight approximates error persistence without fully modeling time-varying, key-dependent transitions, and gains beyond the two evaluated models and fixed workload settings remain unverified.

    Source: H LIMITATIONS · p. 27

Abstract

from arXiv · show

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

1 INTRODUCTION

Concurrent serving can make persistent recurrent states a larger memory bottleneck than model weights, while uniform low-bit quantization sharply reduces accuracy. STEPQuant allocates precision by error lifetime and spatial impact, closely matching FP32-state accuracy at 6 bits while compressing recurrent-state memory.

1 INTRODUCTION

  • Recurrent-state memory grows with concurrent requests and can exceed model-weight memory, motivating compression for serving.In official SGLang deployment, Qwen’s FP32 state pool exceeds its BF16 weight memory at 70 supported concurrent requests.Source: §1 · p. 1
  • Uniform recurrent-state quantization sharply reduces mean accuracy at 4 and 6 bits on both models, with a gap remaining at 8 bits.Quantization errors propagate because each Delta update uses an approximate state and produces a newly quantized one.Source: §1 · p. 1
  • Longer-lived state units tend to accumulate larger errors, so Lifetime-aware Bit Allocation assigns more precision to units with larger, longer-lived errors under a fixed budget.Stronger gate retention allows existing errors to persist through subsequent decoding steps.Source: §1 · pp. 1–2
  • Key-row errors can affect outputs differently, and state magnitudes vary across rows and columns; STEPQuant fits separate row and column scales using those patterns.Row scales account for row magnitude and measured output-error impact, while column scales emphasize rows where errors matter more.Source: §1 · p. 2
  • At 6 bits, STEPQuant closely matches FP32-state accuracy on both models; at 4 bits, it outperforms uniform INT8 on both.With optimized SGLang kernels, 6-bit STEPQuant compresses recurrent-state memory by 5.03× on Qwen and 5.08× on Kimi, reducing total memory by 68.7% and 53.7%, respectively.Source: §1 · p. 2

2 PRELIMINARIES

Gated Delta-rule models maintain fixed-size recurrent states that are updated and retained across decoding steps. The quantization setup updates a reconstructed state in floating point, computes the output, and quantizes the updated state for storage.

2.1 GATED DELTA-RULE LINEAR ATTENTION

  • A gated Delta-rule update retains and modifies the previous state, then uses the head state to compute its readout.GDN uses a scalar per-head retention gate, while KDA uses channel-wise gates; the state has d_k · d_v elements per head, independent of sequence length, and persists for each active request.Source: §2.1 · p. 3

2.2 RECURRENT-STATE QUANTIZATION

  • Symmetric uniform quantization encodes each value as a b-bit integer using a positive scale shared within a quantization group.The setup updates the reconstructed state in floating point and computes the output before quantizing the updated state for storage.Source: §2.2 · p. 3

3 TEMPORAL DIMENSION: LIFETIME-AWARE BIT ALLOCATION

Quantization errors are added at each recurrent-state update and propagated through subsequent transitions, with gate retention influencing how long they persist. Lifetime-aware Bit Allocation weights reconstruction distortion by persistence and assigns precision under a fixed bit budget.

3.1 LIFETIME-DEPENDENT ERROR ACCUMULATION

  • Previously accumulated error passes through the state transition, while each quantization step adds new error; when retention is close to one, errors in directions rarely aligned with subsequent keys can persist for many steps.Under the stated conditions ∥k_t∥2 ≤ 1, 0 ≤ β_t ≤ 1, and 0 ⪯ D_t ⪯ I, the transition has spectral norm at most one.Source: §3.1 · p. 3
  • Longer gate half-lives correlate with larger accumulated state errors under uniform INT6 quantization, with Spearman’s ρS ≈0.80 across Qwen3.8-27B recurrent heads on C4.The study examines all recurrent heads; the trend supports memory lifetime as a predictor of quantization risk.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 LIFETIME-AWARE BIT ALLOCATION

  • Lifetime-aware Bit Allocation estimates each unit’s distortion at candidate bit widths and mean log retention, which indicates its memory lifetime.A unit is an entire head in Qwen and a key row in KDA; gate retention approximates how quantization error persists across updates.Source: §3.2 · p. 4
  • Under an average bit budget, the method selects bit widths to minimize lifetime-weighted reconstruction distortion, accounting for both error magnitude and persistence.Units that remain difficult to quantize with mixed precision can be retained as sparse FP16 pivots; allocation and pivot selection are fixed across requests after offline calibration.Source: §3.2 · p. 4

4 SPATIAL DIMENSION: KEY-ROW-AWARE DUAL-AXIS FITTING

Key-row errors differ in their effect on readout, while state magnitudes vary across both key rows and value columns. STEPQuant addresses these spatial patterns with impact-aware dual-axis scales and fused decoding kernels.

4.1 KEY-ROW IMPACT ON READOUT ERROR

  • Groups of key rows with larger impact scores generally cause greater perplexity degradation when quantized, supporting impact-aware scale selection.The evaluation quantizes each group to INT4 while keeping remaining rows in full precision.Source: §4.1 · p. 5

4.2 TWO-AXIS STATE GEOMETRY

  • Maximum-to-median RMS contrasts reach 10.3× across key rows and 19.4× across value columns, with both contrasts exceeding 3× in 98.6% of sampled states.Pronounced magnitude differences persist across decoding, motivating dual-axis scaling.Source: §4.2 · p. 5

4.3 KEY-ROW-AWARE DUAL-AXIS FITTING

  • STEPQuant combines calibrated row-impact scores with separate row and column scales to represent the updated state.Row factors widen the range for larger-magnitude rows and improve resolution for higher-impact rows; column scales minimize impact-weighted reconstruction error.Source: §4.3 · pp. 5–6

4.4 KERNEL IMPLEMENTATION IN SGLANG

  • With fitted scales, each state entry is quantized to its nearest representable level at its assigned precision and reconstructed from the row and column scales.SGLang kernels fuse tilewise reconstruction, the Delta update, and readout, while overlapping scale updates with model computation on a separate CUDA stream.Source: §4.3 · p. 6, §4.4 · p. 6

5 EXPERIMENTS

Across long- and short-generation benchmarks, STEPQuant retains accuracy at low state precision, including with W4 weights. Component results and serving measurements assess its design and deployment efficiency.

5.1 SETUP

  • Evaluation covers Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct on four A800 GPUs, with 32 WikiText-2 segments used for calibration.Benchmarks include seven long-reasoning tasks and six short language-understanding tasks.Source: §5.1 · p. 6

5.2 ACCURACY ACROSS LONG AND SHORT TASKS

  • 80.59% on Qwen and 61.47% on Kimi for 6-bit STEPQuant versus 45.04% and 45.70% for uniform INT6; 4-bit STEPQuant trails FP32 by 0.15 and 0.25 points on short tasks.The long-task values are mean accuracies; the short-task gaps are for Qwen and Kimi, respectively.Source: §5.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

5.3 COMPATIBILITY WITH W4 WEIGHTS

  • With 4-bit AWQ weights, 6-bit STEPQuant reaches 79.27% on Qwen and 58.62% on Kimi, 0.05 and 0.33 points below their FP32-state baselines.The results show STEPQuant remains effective when combined with weight quantization.Source: §5.3 · p. 7, Table 3 · p. 8

5.4 COMPONENT ABLATION

  • At 4 bits, STEPQuant scores 84.72% on Qwen, versus 73.95% for Spatial only and 12.87% for Temporal only; at 6 bits, it reaches 84.51% versus 84.61% for FP32.The ablation evaluates three long-reasoning tasks; the authors report similar trends on Kimi.Source: §5.4 · pp. 8–9, Table 4 · p. 8

5.5 GENERATION LENGTH ON SEVEN LONG-GENERATION TASKS

  • Under uniform 4-bit quantization, Kimi averages 63.40K tokens on AIME and 64.22K on HMMT while achieving near-zero accuracy; STEPQuant keeps output lengths close to FP32.The authors report that uniform quantization substantially increases output length, whereas STEPQuant preserves long-task accuracy.Source: Figure 3 · p. 8, §5.5 · p. 9

5.6 EFFICIENCY EVALUATION

  • At batch size 512, STEPQuant@6bit reduces Qwen total serving memory from 419.73 to 131.18 GiB, a 68.7% reduction.It also reduces recurrent-state memory by 80.1% (5.03× compression) and state-update time by 65.6% (2.91× faster).Source: Figure 3 · p. 8, §5.6 · p. 9

6 RELATED WORK

Linear-attention and state-space architectures use recurrent states to reduce sequence-dependent storage, but those states still consume memory across active requests. Prior quantization work covers weights, activations, KV caches, and some recurrent states; STEPQuant targets lower-bit quantization of GDN states.

6 RELATED WORK

  • STEPQuant combines recurrent-state quantization with broader low-bit inference approaches to reduce inference memory footprint.The related-work discussion positions it among efficient decoding, pruning, quantization, and architectural approaches.Source: §6 · p. 9, §6.1 · p. 9

6.1 LINEAR ATTENTION TRANSFORMERS

  • Linear attention stores past interactions in a fixed-size recurrent state, while GDN and KDA use Delta-rule updates with forgetting mechanisms.State memory still scales with batch size, layer count, and state dimensions.Source: §6.1 · p. 9

6.2 POST-TRAINING QUANTIZATION

  • Prior work quantizes weights, activations, KV caches, and some recurrent states; DAMP reports preserved accuracy at 9.9 bits per state value, while STEPQuant reports negligible degradation at 6 bits on evaluated benchmarks.Q-Mamba applies dual-axis scaling to state-cache quantization; DAMP protects selected key channels in FP16 and quantizes the rest to INT8.Source: §6.2 · p. 10

7 CONCLUSION

The paper concludes that Delta-rule state quantization errors depend on temporal persistence and spatial structure, and presents STEPQuant as a way to quantize these states accurately while reducing memory for concurrent serving.

7 CONCLUSION

  • STEPQuant combines lifetime-aware bit allocation with Key-Row-Aware Dual-axis Fitting for Delta-rule recurrent states.The paper identifies temporal persistence and spatial structure as shaping quantization error.Source: §7 · p. 10
  • Experiments across long- and short-generation tasks show accurate low-bit state quantization with STEPQuant.The conclusion reports evaluations across both task types.Source: §7 · p. 10
  • STEPQuant substantially reduces recurrent-state memory for concurrent serving.Source: §7 · p. 10

AI USE STATEMENT

AI assistants were used for language polishing, LaTeX checking, and debugging assistance; the authors state that they developed, conducted, and verified the scientific work.

AI USE STATEMENT

REPRODUCIBILITY STATEMENT

The reproducibility material specifies the error analysis, calibration statistics, and experiments, including controlled studies of Delta feedback and prediction drift during compressed-state decoding.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement locates definitions, allocation and fitting procedures, calibration details, benchmark settings, serving implementation, and ablations across the paper and appendices.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PROOF OF PROPOSITION 1

  • Under the proposition’s conditions, the proof bounds the transition operator’s norm by one, while the cumulative-error analysis relates propagated state perturbations to squared output error over future reads.The appendix also relates cumulative squared output error to row-impact scores used in spatial fitting.Source: §A.1 · p. 15, §A.2 · p. 15, A RECURRENT-STATE ERROR PROPAGATION… · p. 15

A.3 CALIBRATION STATISTICS FOR TEMPORAL AND SPATIAL COMPONENTS

  • The calibration statistics apply lifetime weights to candidate-format distortion after averaging over state samples, with Qwen using row-impact-weighted distortion and KDA using unweighted per-element MSE.Source: §A.3 · p. 16

B.1 EFFECT OF DELTA FEEDBACK ON QUANTIZATION ERROR

  • 26.82× for INT6 and 18.65× for INT8: the exact-read oracle increases cumulative squared output error relative to native Delta feedback in the matched single-injection experiment.Both ratios exceed one on each of the four replay trajectories.Source: §B.1 · pp. 16–17

B.2 PREFILL AND DECODE PREDICTION DRIFT

  • Mean excess NLL rises from 0.102 to 2.102 for Qwen INT6 and from 0.0228 to 0.1422 for KDA INT6 between the first and last 256 compressed-read predictions.Across all 6,144 decode predictions, STEPQuant@6bit has mean excess NLL of 0.0011 on Qwen and −0.0120 on KDA.Source: §B.2 · p. 17

B.3 READOUT ERROR AT FIXED PRECISION

These appendix sections report fixed-precision row-impact tests, lifetime and state-geometry analyses, calibration and storage procedures, and task-scoring and generation-length details for STEPQuant.

B.3 READOUT ERROR AT FIXED PRECISION

  • 117.4 for Qwen and 35.1 for KDA: these are the highest-to-lowest held-out read-error bin ratios across rows ranked by independently calibrated impact within fixed-precision groups.The contrasts motivate Key-Row-Aware Dual-axis Fitting even after lifetime-aware precision allocation.Source: §B.3 · p. 17

B.4 KEY-ROW QUANTIZATION AND PERPLEXITY

  • Higher-impact key-row groups generally cause larger perplexity increases when quantized to INT4, in both models.The experiment quantizes one independently ranked group at a time while keeping other rows in full precision.Source: §B.4 · pp. 17–18

B.5 GATE HALF-LIFE AND ACCUMULATED INT6 STATE ERROR

  • 52.5% for Qwen and 78.8% for KDA: the longest-lived quarter of units accounts for these shares of cumulative INT6 error.Qwen units are whole heads; KDA units are key rows.Source: Figure B1 · p. 18, §B.5 · p. 19

B.6 LIFETIME RANKINGS AND DUAL-AXIS STATE GEOMETRY

  • WikiText lifetime rankings closely match C4 and LiveCodeBench rankings, with Spearman correlations of 0.989 and 0.985 for Qwen, and 0.991 and 0.982 for KDA.Source: §B.6 · p. 19, Figure B3 · p. 20
  • KDA’s state geometry shows median maximum-to-median RMS contrasts of 12.7× for key rows and 5.4× for value columns across 3,840 snapshots.Qwen’s C4 medians are 10.3× for key rows and 19.4× for value columns; both contrasts exceed 3× in 98.6% of its snapshots.Source: §B.6 · p. 20

C CALIBRATION, PRECISION ALLOCATION, AND DECODE PROCEDURE

  • KDA calibration combines gate-derived lifetime with per-row candidate-format MSE, uses row-impact scores for spatial preconditioning, and refits shared scales to each updated state.The appendix also describes row-impact calibration in KDA’s native state coordinates and the recurrent update and quantized-writeback workflow.Source: §C.1 · pp. 20–21, §C.2 · p. 21, §C.3 · p. 21

D STORAGE COST OF CODES, SCALES, AND FP16 PIVOTS

  • 7.44× and 5.08×: KDA’s analytical compact recurrent-state representation gives these compression ratios versus FP32 at the four- and six-bit budgets, respectively.The counts are analytical and exclude tensor-parallel page padding and allocator overhead; Qwen’s four-bit representation is 6.9251× versus FP32.Source: D STORAGE COST OF CODES, SCALES, AND… · p. 22, Table D3 · p. 23

E.1 SCORING GENERATED ANSWERS ON SHORT TASKS

  • Uniform INT4 scores 46.57% on Qwen and 17.36% on KDA for WinoGrande, partly because some generations fail to provide an extractable answer.With W4A16 weights at four bits, mean accuracy is 0.34 percentage points below FP32 for Qwen and 2.29 points below for KDA, while mean generation length increases for both.Source: §E.1 · p. 23, §E.3 · p. 23, Table E5 · p. 24

E.4 ADDITIONAL COMPONENT ABLATION

The section reports component-ablation behavior and the SGLang serving implementation, alongside compressed-state storage and reserved-pool capacity measurements.

E.4 ADDITIONAL COMPONENT ABLATION

  • Spatial fitting outperforms DSQ, sparse FP16 pivots improve the temporal component, and combining both components yields further accuracy gains on Kimi.Table E6 reports accuracy with BF16 weights and averages three long-generation benchmarks.Source: §E.4 · p. 24, Table E6 · p. 24

F.1 PACKED-STATE INTEGRATION IN SGLANG

  • Qwen STEPQuant@6bit uses 28.609 MiB per request versus 144 MiB for FP32, a 5.03× storage reduction (80.13%).In SGLang, decode updates integer pages directly using floating-point tiles, without a persistent full-matrix FP32 shadow.Source: §F.1 · p. 24

F.2 DECODE TIMING PROTOCOL AND THROUGHPUT RESULTS

  • Throughput is the median tokens/s across three post-warmup runs, measured on the slowest TP rank with 128-token prompts and 1,024 generated tokens.The measured intervals include inter-step gaps; prefill, startup, warmup, and final output delivery are excluded.Source: §F.2 · pp. 24–25

F.3 MEMORY RESERVED FOR CONCURRENT REQUESTS

  • With the five-slot reservation, the reserved persistent-state pool is 9.85 GiB at B = 64, versus 46.02 GiB in FP32.The state curves exclude weights and attention KV.Source: §F.3 · p. 25

F.4 ADDITIONAL RESULTS ON KDA

The additional evaluations report KDA serving-memory and state-update gains, higher decode throughput at large batches, a low-precision comparison with DAMP, and remaining scope limits.

F.4 ADDITIONAL RESULTS ON KDA

  • At batch size 512, STEPQuant@6bit reduces KDA total serving memory from 149.70 to 69.36 GiB (53.7%) and state-update time from 8.48 to 4.76 ms (43.9%, 1.78× faster).State memory falls from 100.04 to 19.70 GiB (80.3%, 5.08× compression).Source: Figure F4 · p. 25, §F.4 · p. 25

F.5 DECODE THROUGHPUT

G COMPARISON WITH DAMP

  • Across three shared KDA benchmarks, DAMP retains 100.99% at 9.9 bits, versus STEPQuant’s 100.51% at 6.30 bits and 92.91% at 4.30 bits; settings differ.The differing evaluation settings preclude a strictly controlled cross-study comparison.Source: G COMPARISON WITH DAMP · pp. 26–27, Table G8 · p. 27

H LIMITATIONS

  • The lifetime weight approximates persistence through gate decay, while gains beyond two models and fixed workloads remain unverified.With BF16 weights at four bits, KDA loses accuracy on long-generation tasks.Source: H LIMITATIONS · p. 27

Related papers

Loading 2609.38169v1…