Low-Bit Recurrent States in Hybrid Language Models

1 Introduction

• Recurrent states commonly remain FP32 even when projections use NVFP4; Nemotron 3 uses FP16 states with stochastic rounding.The introduction also describes KDA's per-token row decay and models with per-head decay.Source: §1 · p. 1
• Slowly decaying rows accumulate recurrent-state errors across write-backs, unlike rapidly decaying rows that dissipate them.Range-based quantization controls error at a single write-back but does not account for its later readout effects.Source: §1 · p. 1
• Observability-Gramian distortion weights combined with normalized state ranges support variable-width allocation and logarithmic decay-rate quantization without calibration data or training.The paper connects established persistence weighting and rate–distortion allocation to readout distortion.Source: §1 · pp. 1–2
• Figure 1 depicts error decay and write-back sampling, bit-allocation contributions and widths, and relative readout MSE for natural versus equal row ranges.The illustrated synthetic recurrence uses C = 64 and a two-bit mean payload.Source: Figure 1 · p. 2

2 Propagation of state quantization error

• Under stationary inputs independent of quantization noise and zero-mean noise uncorrelated across rows and write-backs, Proposition 1 gives time-averaged steady-state readout MSE.The result assumes finite accumulated error energy and per-element row variance σ_i^2.Source: §2 · p. 2
• For diagonal decay, CΓ_i is a diagonal entry of the expected observability Gramian; constant decay without erasure gives Γ_i = ρ_i/[C expm1(2s_i)].The stated bound for constant per-head decay is Γ_i ≤ E∥q∥_2^2/[C expm1(2s)].Source: §2 · p. 2
• At three to five bits, simulations show lag-one error correlations up to 0.23 for s_i ≤ 10^-3, and Proposition 1 overestimates the uniform-four-bit gain by 1.7–2.6 times.Readout error is 0.17–2.4 times that of independent noise at step size Δ; discrepancies are within 20% from six bits.Source: §2 · p. 3
• The channel-wise erasure approximation is not a bound: errors orthogonal to a repeated key are never erased.For a fixed key with a = k/2, s = 0.01, and C = 1, exact and approximate Γ_i are 24.9 and 2.06.Source: §2 · p. 3
• Normalizing row ranges within output normalization groups motivates a proxy, but the assumed uniform ρ_i and proportional output mean square are not exact RMSNorm properties.The authors evaluate these assumptions through ablations; teacher forcing also does not eliminate cross-layer noise propagation.Source: §2 · p. 3

Decay-gate quantization

• At two bits, logarithmic gate quantization raises WikiText perplexity by at most 0.27 across the tested Qwen and Kimi models and outperforms levels uniform in g = −s on all models.Uniform-in-g levels yield Kimi perplexity 86.9 even at four bits, versus 9.33 in full precision; subsequent experiments keep gates in full precision.Source: §3 · p. 3

State bit allocation

• The state allocator minimizes the high-rate proxy Σ_i c_i 2^-2b_i, using decay weights and row ranges normalized within each output group.The proxy's unconstrained real-valued solution is given for layer-wise mean budget b̄.Source: §3 · p. 3
• The unconstrained proxy predicts an AM(c)/GM(c) distortion reduction over uniform allocation, but this optimality does not extend to low-bit integer quantizers.The method rounds and clips widths to [b_min, b_max], then adjusts them to meet the layer budget.Source: §3 · p. 3
• Chunk-aware EMA estimates of decay and erasure increase simulated readout error by at most 2.5% over exact-Gramian allocation across tested write-back intervals.A one-token estimate gives comparable decoding NLL.Source: §3 · p. 3

Allocation units and storage

• For per-head decay, each head gets one width and block scales, with a two-bit minimum; stored scales and width maps add overhead.The stated overhead is 19/d_v bits per row element for an FP16 scale and three-bit width, with block scales adding 0.5 bits per element.Source: §3 · p. 4

4 Bit-allocation ablations

• At C = 1, 16, and 64, decay weighting reduces excess NLL versus normalized range alone by 0.038, 0.021, and 0.005, respectively.The C = 64 comparison ties on the second held-out set; versus raw range alone, reductions are 0.013, 0.008, and 0.003, with the last a tie.Source: Table 1 · p. 4, §4 · p. 5
• At C = 64 on Kimi, decay-only allocation is 0.059 worse than the combined method, while combining decay with raw range is 0.012 worse at C = 16.Slowly decaying heads tend to have larger ranges, so raw range can already carry an indirect decay signal.Source: §4 · p. 5
• Shuffling decay weights across rows worsens NLL in all 32 windows at both C = 16 and 64, supporting the importance of matching weights to rows.The observed C = 16-to-64 gain ratio is 4.0 versus normalized range and 1.9 versus uniform widths, with large uncertainty.Source: §4 · p. 5
• At C = 16, persistence alternatives tie the method and improve on normalized range alone by 0.020–0.024; at C = 64, alternatives remain within 0.004.The write-back-time factor agrees qualitatively with the observed 1/C distortion scaling; omitting erasure ties at both intervals.Source: §4 · p. 5

5 Experimental comparisons

• At a four-bit mean (C = 1), per-token quantization outperformed all seven baselines in every window across eight 2K-token WikiText windows, reducing excess NLL versus the best baseline by factors of 3.3, 4.8, and 27.9 on Kimi, Qwen3.8, and Qwen3.6, respectively.Source: §5 · p. 5, Table 2 · p. 6
• The evaluations use simulated quantizers without kernel timings, and eight decoding windows limit bootstrap reliability.The four-bit per-token comparisons use eight 2K-token WikiText windows.Source: §5 · p. 5, Table 2 · p. 6
• At C = 64, the method is best in four of 15 model–budget pairs, tied in seven, and worse in four.At four bits it outperforms tested formats on Qwen3.6, Qwen3.5-9B, and Nemotron; on held-out Kimi data at three bits, normalized range outperforms it and FP32.Source: Table 3 · p. 6, §5 · p. 7

6 Conclusion

• Decay weighting with normalized ranges improves low-bit recurrent-state quantization, especially with frequent write-backs, while correlated rounding and implementation costs require further study.Source: §6 · p. 7

8 Compliance with Ethical Standards

• The study used numerical simulations and released models on public text benchmarks, with no human participants or animal experiments.Source: §8 · p. 7

• The authors report no external or institutional funding and no relevant financial or nonfinancial interests to disclose.Source: §9 · p. 7