PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Discrepancy Dither for Quantized Recurrent State Caches

Snigdha Chandan Khilar

arXiv:2609.39185v1cs.LG

TL;DR

Low-precision recurrent-state caches can save memory bandwidth, but rounding errors feed back through generation, raising the question of which rounding rule to use. The paper evaluates deterministic golden-ratio Weyl dither against stochastic rounding and round-to-nearest. Weyl dither improves on stochastic rounding across the reported settings, while round-to-nearest can lose its short-term advantage over long decoding.

  • Problem

    Recurrent-state rounding errors feed into later updates, motivating the practical question of which rule to use when storing the state in low precision.

    Source: §1 · p. 1

  • Method

    The paper compares round-to-nearest, stochastic rounding, and deterministic golden-ratio Weyl dither for quantized recurrent-state caches.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Weyl dither lowers KL than stochastic rounding in all 15 model-format cells, with significant reductions ranging from 11% to 49%, and remains below it through 4096-step hybrid decoding horizons.

    Source: §6.1 · p. 6

  • Takeaways & Limitations

    The conclusion recommends Weyl dither over stochastic rounding in the measured settings and careful implementation, while warning that round-to-nearest can fall behind over long generations.

    Source: §8 · p. 10

    The models are small and run in research code without end-to-end throughput results; long-horizon evaluation uses eight teacher-forced documents and KL rather than generation length or task accuracy.

    Source: §8 · p. 10

Abstract

from arXiv · show

Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed back into the next update and can accumulate over long generations. Production systems round the state stochastically; we ask which rounding rule such caches should use. We find that a deterministic golden-ratio Weyl dither, which needs no random numbers, consistently brings the quantized model closer to the full-precision one than stochastic rounding, across pure and hybrid models, storage formats, and long decoding horizons, at no extra cost. Round-to-nearest behaves differently: because it discards small updates, its error keeps growing, so it can look best in short evaluations yet falls far behind over long generations. A discrepancy analysis explains this ordering, and we document implementation pitfalls that silently remove the benefit.

1 Introduction

Low-precision recurrent-state caches save memory bandwidth, but their rounding errors feed back across generated tokens. The paper compares rounding rules and finds that golden-ratio Weyl dither consistently improves on stochastic rounding across its main settings.

Motivation

  • Mamba-style models compress past tokens into a fixed-size state rewritten at every generated token, making low-precision storage an attractive bandwidth-saving option.Hybrid models interleave state-space layers with attention.Source: §1 · p. 1
  • Recurrent-state rounding errors can accumulate because each stored state becomes the next step’s starting point, unlike errors in a weight matrix.Some state entries remember for thousands of steps.Source: §1 · p. 1

Question and rules

  • The paper asks which rounding rule to use when a recurrent state must be stored in a given low-precision format.The compared rules are round-to-nearest, stochastic rounding, and Weyl dither.Source: §1 · pp. 1–2

What we find

  • Weyl dither lowers prediction damage versus stochastic rounding across pure Mamba and hybrid models, four storage formats, and horizons up to 4096 steps, at no extra cost.It needs no random-number generator and is never significantly worse in the main comparison.Source: §1 · p. 2

Scope and honesty

  • The contribution is studying deterministic low-discrepancy rounding where errors are fed back, rather than proposing deterministic low-discrepancy rounding itself.The authors report models of 130M–1.4B parameters and experiments on a single T4 GPU.Source: §1 · p. 2

2 Setup: States, Formats, and Rounding Rules

The paper studies quantizing recurrent states when writing them back during decoding. Its Weyl rule replaces fresh random thresholds with a deterministic sequence that spreads rounding decisions evenly over time.

The recurrent state and its memory

  • Each Mamba state entry evolves through an input-dependent decay and a new write, and its time constant describes how long it remembers.Across five measured models, 13–23% of entries have time constants of at least 100 steps and up to 5% have time constants of at least 1000.Source: §2 · p. 2

Low-precision storage

  • Decoding kernels compute updates in FP32 and quantize only when writing the new state; INT8 entries share a block scale that sets the spacing between representable values.Blocks contain 16 entries for Mamba and 128 for hybrids; the main INT8 setting stores its scale in FP32.Source: §2 · pp. 2–3

Rounding with a threshold

  • Weyl thresholds use a golden-ratio increment and fixed per-entry offsets, spreading rounding decisions nearly evenly over time with less clumping than random draws.The rule applies to floating-point formats on their local grid of representable values.Source: §2 · p. 3

Two regimes

  • The main decode regime processes the prompt in full precision and quantizes only states written during generation; an every-token regime also quantizes the prompt as a stress test.The every-token regime is used in earlier experiments.Source: §2 · p. 3

3 Why the Rules Behave Differently

The rules differ in how their rounding errors accumulate: round-to-nearest can erase small writes, while the other rules retain them. In an idealized setting, accumulated error is governed by the threshold sequence’s discrepancy, though the analysis predicts ordering rather than real-model gain size.

Round-to-nearest discards small updates

  • Round-to-nearest loses an update entirely when the stored value is on-grid and the exact update is smaller than half a quantization step.Repeated small losses can accumulate; long-memory entries often combine near-unit decay with small writes.Source: §3 · p. 3

How errors accumulate: discrepancy

  • Under perfect memory and constant writes, accumulated error is a threshold-counting error bounded by the sequence’s star discrepancy.The bound is attained for the worst fractional write size.Source: §3 · pp. 3–4

Corollary 1

  • In the idealized setting, round-to-nearest error grows linearly, while stochastic rounding has zero average error but typical error grows like the square root of the number of steps.Weyl dither’s error stays below a bound proportional to 1 + log T.Source: §3 · p. 4
  • A Weyl increment near a simple fraction, such as 1/2 + ε, can behave as badly as round-to-nearest for up to 1/(8ε) steps.This qualification concerns the chosen increment, not the ideal Weyl bound for a well-spread sequence.Source: §3 · p. 4

Round-to-nearest’s short-term advantage

  • When update fractions are effectively random, round-to-nearest has half the mean-squared rounding error per write of stochastic rounding or evenly spread Weyl dither.Which effect dominates depends on the model, its use of state, and decoding duration.Source: §3 · p. 4

What the theory does not cover

  • The analysis predicts the ordering of rounding rules, not the size of real-model gains, because real entries decay, writes vary, and block scales move.In real models the Weyl-to-stochastic error ratio is roughly constant rather than widening with memory.Source: §3 · p. 4

4 How We Measured

The evaluation measures how quantized recurrent states change next-token distributions relative to full precision, using checked model implementations and document-level comparisons. It uses different datasets and prompt/decode setups for pure Mamba and hybrid models.

Models

  • Experiments use Mamba and hybrid models with FP32 weights and activations, quantizing only the recurrent state.The evaluated families include Mamba-1, Mamba-2, Granite 4.0-H, and Falcon-H1 0.5B.Source: §4 · p. 4

Verification

  • Model checks verify decoding against full forward passes and confirm that cached Mamba states affect subsequent predictions; two additional hybrids were excluded after checks failed.The failed checks involved a decoding mismatch for Falcon-H1 1.5B and a loading error for Zamba2 1.2B.Source: §4 · p. 4

What we measure

  • The main metric is per-token KL divergence between full-precision and quantized-state next-token distributions, averaged over positions and documents under teacher forcing.It measures prediction differences whether or not a downstream task notices them.Source: §4 · p. 5
  • Pure Mamba evaluation uses 32 WikiText-103 chunks with 1024 full-precision tokens followed by 1024 quantized positions; hybrids use 32 PG-19 books with a 2048-token prefill and 512 quantized decoding steps.The hybrid long-horizon study uses 8 books and 4096 quantized steps.Source: §4 · p. 5

Statistics and decision rules

  • Relative KL reductions use 95% bootstrap intervals over documents, with both documents and seeds resampled for Mamba comparisons; significance requires a lower bound above zero.Each experiment’s hypotheses and pass criteria were recorded before it ran.Source: §4 · p. 5

5 Experimental Environment and Code

Experiments ran on a single T4 with FP32 model arithmetic and state rounding as the only low-precision operation. Self-contained notebooks verified models, ran comparisons on identical inputs, and recorded per-document results.

Experimental environment

  • All experiments ran on a single NVIDIA Tesla T4, with weights, activations, and matrix multiplications in FP32 and state rounding as the only low-precision arithmetic.The GPU had 16 GB of memory, about 14.6 GB usable; Triton timings used the same T4.Source: §5 · p. 5

Experimental workflow

  • Each experiment used a self-contained notebook to verify the model, run rounding rules on identical inputs, and save per-document results.Every reported number came from a notebook or was computed from its printed values.Source: §5 · p. 5
  • Before GPU runs, notebooks were tested end to end on CPU-only machines with tiny randomly initialized models of the same architectures.Statistical analysis and discrepancy calculations also ran on the CPU.Source: §5 · p. 5

Code

  • The released code includes experiment notebooks, rounding rules, model runners, hooks, a Triton kernel, and scripts for producing figures.The repository maps tables, figures, and paper numbers to their producing notebooks and outputs.Source: §5 · p. 5

Evaluation scope

  • Table 1 reports KL divergence to the full-precision model and relative KL reduction over stochastic rounding for five models across three formats.It covers 32 documents per model; Mamba uses 1024 quantized positions after a 1024-token prefill, while hybrids use 512 after a 2048-token prefill.Source: Table 1 · p. 6

6 Results

Across the main comparison, long generations, and several model and storage settings, Weyl dither generally reduces prediction divergence relative to stochastic rounding. The results connect threshold-sequence evenness to error, while showing that round-to-nearest and some implementation choices can change the comparison.

6.1 The Weyl dither beats stochastic rounding

  • Weyl dither lowered KL than stochastic rounding in all 15 main-comparison cells, with significant reductions of 11%–49% and a 31% median.The smallest reduction was 11% for Granite 1B at FP8, with a 2% lower bound; stochastic rounding was not best in any cell.Source: §6.1 · p. 6
  • Over 4096 decoding steps, Weyl dither lowered KL in all six hybrid-model and format panels, with reductions ranging from 12% to 56%.Its advantage shrank by no more than five points in four panels; it also matched stochastic rounding’s KL at roughly 0.25 additional bits, with a +0.16 to +0.30-bit range.Source: §6.1 · p. 6, Figure 1 · p. 7

6.2 Why the Weyl dither wins

  • Across nine threshold sequences, local discrepancy rankings tracked rounding damage (ρ = 0.93–1.00), and well-spread increments differed by less than 5%.An increment near 1/2 was about 40 times worse than stochastic rounding.Source: §6.2 · p. 7
  • Applying Weyl dither only to the 25% longest-memory entries retained 75%–87% of its full gain, while recurrence replay reproduced the measured advantage at INT8.Replay error ratios were 0.71–0.74 across five models; gains varied with memory length and were not an ever-widening gap.Source: §6.2 · pp. 7–8

6.3 Round-to-nearest: good at first, then worse and worse

  • Round-to-nearest was worst on pure Mamba cells but initially best on Granite hybrids at INT8 and FP8; its KL then climbed 4–45 times over 4096 steps as dithers leveled off.Discarded updates accumulate, and the reason for Granite’s early advantage remains unexplained and model-specific.Source: §6.3 · p. 8

6.4 Three silent pitfalls

  • Offsets generated with the same increment as time alias neighboring entries’ thresholds; Knuth-hash offsets raised Mamba-2 INT4 KL by 34%–40%.Offsets from a different low-discrepancy sequence removed that loss.Source: §6.4 · pp. 8–9
  • At t = 32767, float32 phase precision leaves only 512 distinct threshold sequences, and this can make Weyl dither appear worse at 32K-token context.Reducing the phase modulo 1 in double precision or using an integer phase counter fixes the issue; reported results use the corrected version.Source: §6.4 · p. 9

6.5 Other settings and cost

  • Periodic full-precision checkpointing mostly removed Weyl’s hybrid advantage, while mixed FP16/INT4 storage retained a 23%–24% gain over stochastic rounding.Hadamard rotation reversed the INT4 comparison, but rotation plus Weyl was best at INT8; the Weyl threshold took 1.00–1.13 times round-to-nearest kernel time.Source: §6.5 · p. 9

7 Related Work

Prior work establishes stochastic and dithered rounding and explores quantization within feedback loops. Related state-space and cache-quantization work focuses largely on which precision to store, whereas this paper studies how to round into that precision.

Stochastic and dithered rounding

  • Stochastic rounding is unbiased and prevents stagnation in long sums, while deterministic low-discrepancy dither rounding was proposed by Wu (2021).Noise shaping and sigma–delta methods address feedback-loop quantization but require storing the previous rounding error, doubling cache size.Source: §7 · p. 9

Quantizing state-space models

8 Limitations and Conclusion

The conclusion favors golden-ratio Weyl dither over stochastic rounding for low-precision recurrent-state storage, while noting that round-to-nearest can fare poorly over long generations. The authors also bound the evidence to small research-code experiments and identify implementation and verification limits.

Limitations

  • Experiments used 130M–1.4B models in FP32 research code, reported no end-to-end throughput, and could not verify two larger hybrids.The long-horizon study used eight documents and teacher-forced text, measured KL rather than generation length or task accuracy, and Mamba tests did not exceed its 2048-token training length.Source: §8 · p. 10
  • The analysis explains the ordering of rounding rules, not the size of their gains; an independent reimplementation has not yet been done.The authors also could not explain why Granite favors round-to-nearest early in decoding.Source: §8 · p. 10

Conclusion

  • Weyl dither was better or statistically tied to stochastic rounding in every measured setting, including long generations, at no extra cost.The conclusion recommends using it for low-precision recurrent-state storage.Source: §8 · p. 10
  • In five of six long-generation settings, round-to-nearest fell behind Weyl dither by up to 400 times after looking best in some short hybrid evaluations.The authors attribute its growing error to discarded small updates.Source: §8 · p. 10

A Proofs

The proofs relate accumulated rounding error to discrepancy, derive error behavior for several rounding rules, and discuss recurrent decay and finite-precision implementation effects.

Rounding-rule proofs

  • Round-to-nearest erases updates smaller than half a quantization step; under constant fractional updates, its error can grow linearly with T.The proof also expresses accumulated dither error through the count of sequence values below a threshold.Source: A Proofs · p. 13

Discrepancy and accumulated error

  • For constant writes, accumulated error under a deterministic dither is bounded by the sequence’s star discrepancy, yielding O(log T) growth for the golden-ratio increment.The proof contrasts this with linear growth for round-to-nearest and stochastic fluctuations under stochastic rounding.Source: A Proofs · p. 13

Error with decay

  • When decay varies by step, final error is a nonnegative weighted combination of tail residual sums, so it is bounded by the largest such sum.For constant decay, the analysis motivates measuring discrepancy over windows near the state’s memory length, without claiming a general windowed bound.Source: A Proofs · p. 13

Offset relation

  • If the offset increment aliases with time, the resulting streams can repeat across neighboring entries, undermining their intended separation.The fixed-point proof notes that near-matching increments differ only when a carry crosses the shifted-out bits.Source: A Proofs · p. 14

Residual variance

  • With independent uniform dither, residual variance is s^2/6; equidistributed deterministic dither has the same time-average result under the stated independence condition.For i.i.d. residuals and constant decay a, stationary error variance is σ^2/(1 − a^2).Source: A Proofs · p. 14

Finite-precision offsets

  • Computing offset plus tα directly in float32 can collapse distinct offsets into identical dither streams; reducing tα modulo 1 in float64 preserves offset precision.The proof gives examples where the number of distinct streams shrinks as t increases.Source: A Proofs · p. 14

B Experimental Details

The appendix documents the tested models, data, runners, quantization configurations, and evaluation protocols, alongside replay and sensitivity analyses used to examine state errors.

Models and assets

  • The experiments used Mamba-1, Mamba-2, Granite hybrids, and Falcon-H1, with WikiText-103, PG-19, LAMBADA, and GSM8K data.The appendix lists checkpoints, dataset sources, and software versions.Source: B Experimental Details · p. 14

Runners

  • Mamba and Granite runners apply state-write hooks while computing updates in FP32; replacement scans and chunked attention were checked against reference computations.Reported maximum discrepancies include 6 × 10^-7 for the scan and 1.3 × 10^-6 for attention.Source: B Experimental Details · pp. 14–15

Decode regime

  • The main decode comparison used 32 chunks or books with INT8 and an FP32 block scale; long-horizon runs used eight books, 4096 teacher-forced steps, and KL averaged over 512-step blocks.The long-horizon protocol began from a 2048-token FP32 prefill.Source: B Experimental Details · p. 15

Memory bins and replay

  • Memory-bin analysis measured time-constant shares, while open-loop replay and injected noise estimated state error and sensitivity by bin.The replay iterates the quantized recurrence from the quantized end-of-prefill state; the composed prediction weights each bin’s error by its sensitivity.Source: B Experimental Details · p. 15

Offset near-aliasing

  • Knuth offsets have exact aliasing, whereas the two tested plastic-number generators have separation d = 0.0015.The separation metric compares neighboring offset shifts with time shifts up to 16.Source: B Experimental Details · pp. 15–16

Formats and dithers

  • The study compared block-scaled integer and FP8 formats with FP16/BF16, using deterministic Weyl dithers or per-write stochastic draws.The appendix specifies scales, dither offsets and increments, hardware implementation, and the local-discrepancy measure.Source: B Experimental Details · p. 16

Evaluation

  • KL and NLL compare final hidden states through the same output layer, with relative reductions and intervals estimated using paired bootstrap resamples.The passage specifies datasets, chunk or book selection, seeds, and bootstrap resampling details.Source: B Experimental Details · p. 16

C Additional Results

Additional results test rounding rules in free-running generation, longer-prefill settings, benchmark accuracy, and state-error analyses. They also report a corrected-quantizer rotation study and withdraw an earlier 8K-context comparison.

Error magnitude versus structure

  • 1.3–2.8× higher output KL than sr despite rtn having the smallest RMS state error at every memory length.For small writes, rtn residuals were biased against the write direction; Weyl state error was up to 23% lower than sr’s.Source: C Additional Results · p. 16, Figure 2 · p. 17

Free-running generation

  • 29–37% lower per-step KL versus sr at FP8 and INT4 in every-token greedy decoding; INT8 was unresolved.In sampled decode, Weyl also reduced per-step KL versus sr by 15.2–29.0% across the reported Mamba-1 sizes and formats.Source: Table 2 · p. 17, C Additional Results · p. 17

Long prefills

  • At 8K and 32K prefills, hash sr stayed within 9% of Philox; at 2K, Weyl INT8 lowered KL versus sr by 21.9% on 350M and 21.6% on 1B.The reported 2K comparison used Weyl block size 128 and sr block size 32.Source: C Additional Results · p. 17

Benchmark results

  • On INT4 LAMBADA, Weyl was more accurate than sr on all six Mamba models, with a mean gain of 1.0 point.The supplied table caption identifies the metric as accuracy (%) with an INT4 state cache in the every-token regime.Source: Table 3 · p. 18

Rotation study

  • With the corrected INT4 quantizer, shared-increment Weyl lost to sr, per-entry increments reached parity, and rotation hurt every rule versus no rotation.For Mamba-1 130M, shared-increment Weyl versus sr KL was 0.235 versus 0.166; per-entry increments reached 0.174.Source: C Additional Results · p. 18

Withdrawn comparison

  • The earlier 8K-context comparison was withdrawn because the float32-phase dither implementation produced only 2,048 distinct streams for 24,576–32,768 entries per layer.Those runs also extended Mamba-1 beyond its training length; the reported beyond-training-length observations do not involve the dither.Source: C Additional Results · p. 19

D Earlier Protocols, the Search for a Mechanism, and Long Horizons

Earlier decode protocols favor Weyl over stochastic rounding, and long-horizon tests show round-to-nearest’s early Granite advantage is temporary. Attempts to explain that reversal from state errors and output sensitivities were largely unsuccessful.

Earlier decode protocols

  • Weyl beat stochastic rounding with a positive lower bound in 26 of 28 earlier-protocol settings above resolution floors; the other two estimates were positive.Under a common 10^-3 floor, the count was 25 of 27; checkpointed settings were excluded.Source: D Earlier Protocols, the Search for a… · p. 20

Search for a mechanism

  • For τ < 10, replayed INT8 rtn/sr error ratios were 0.46, 0.50, 0.50, and 0.51 across four models; long-memory rtn error was up to 30× worse at BF16.The 0.85 ratio on Mamba-1 370M was unexplained.Source: D Earlier Protocols, the Search for a… · pp. 20–21
  • Replayed state error favored Weyl on every model, including Granite by 1.16–1.38× at INT8 and FP8, but predicted the winning rule in only 10 of 19 cells.Weighting replayed errors by measured output sensitivities still mispredicted 8 of 10 Granite cells and 11 of 19 overall.Source: D Earlier Protocols, the Search for a… · p. 21
  • Neither a memory-gated mix of round-to-nearest and Weyl nor drift-sensitivity weighting met its pre-specified decision criteria.The mixed rule was within 5% of the better pure rule in 10 of 15 cells; drift-sensitivity composition picked the winner in 12 of 19 cells.Source: D Earlier Protocols, the Search for a… · p. 22

Long horizons

  • Table 15 reports KL per token (×10−3) for the first and last 512-step blocks of 4096 quantized decoding steps, plus Weyl-vs-sr reduction estimates with 95% lower bounds across 8 documents.Source: Table 15 · p. 24

Earlier protocols

  • Weyl at INT4, INT6, and INT7 matched stochastic rounding at INT4.16–4.22, INT6.25–6.28, and INT7.27–7.30, respectively.The equivalence is based on linearly interpolating log KL for stochastic rounding.Source: D Earlier Protocols, the Search for a… · p. 24

E Protocol Log and Negative Results

The protocol log records corrections to invalid or unstable experiments, verification-based model exclusions, and negative results that narrowed the paper’s conclusions.

Protocol corrections

  • All reported results came from later runs after the first run’s randomly initialized Mamba output layer produced implausible perplexities near 10^12.An output-weighted diagonal rescaling was tested only in that invalid run and not pursued because block-absmax quantization favors equalizing ranges.Source: E Protocol Log and Negative Results · pp. 24–25
  • Inflating FP16 scales by 1 + 2^-9 destabilized round-to-nearest: INT8 KL rose from 0.0025 to 10.3, while FP32 scales avoided the instability.In a scalar simulation, a state expected to decay to 3 × 10^-4 over 8000 steps grew to about 10^3.Source: E Protocol Log and Negative Results · p. 25
  • Float32 phase precision created aliased dither streams; after rerunning affected configurations with float64 phase reduction, the 32K Granite INT8 KL ratio was 0.47–0.84.The median corrected/flawed KL ratio was 0.999 on Mamba and 0.974 on Granite.Source: E Protocol Log and Negative Results · p. 25

Verification exclusions

  • Falcon-H1 1.5B and Zamba2 1.2B were excluded after verification failed: one produced 12–20 logit errors at every step, and the other failed to load.No results were recorded for either model.Source: E Protocol Log and Negative Results · p. 25

Conclusion changes

  • Earlier Granite evaluations covered only 512 quantized steps, the window where round-to-nearest looked best; corrected long-horizon results reversed the early Weyl conclusion.The phase correction did not affect round-to-nearest’s Granite advantage.Source: E Protocol Log and Negative Results · p. 25

Negative results

  • Hadamard rotation modestly helped stochastic rounding at INT8 but hurt every rule at INT4; per-entry Weyl increments were 17–20% worse than shared increments at FP8.The 8K results beyond Mamba’s training length were withdrawn.Source: E Protocol Log and Negative Results · p. 25

Related papers

Loading 2609.39185v1…