PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows

Qinghang Huang, Xuchu Huang, Xunzhao Yin

arXiv:2610.39931v1cs.LG

TL;DR

Fixed-size recurrent states still carry storage and access costs, and their quantization errors can affect later recurrences. The paper proposes gradient-aware head-bit allocation and a sliding window for recent writes; across the reported evaluations, these approaches improve scores and shorten outputs, with recent-write maintenance also outperforming higher accumulated-state precision at similar storage budgets.

  • Problem

    Fixed-size recurrent states still require per-request storage and access, while local distortion and error lifetime alone do not determine how perturbations affect final predictions.

    Source: §1 · pp. 1–2

  • Method

    GGA weights candidate state-quantization residuals by future-loss gradients to allocate head precision, while PW propagates recent pure-write branches through GDN transitions before merging them by write age.

    Source: §1 · p. 2

  • Results

    At @4, adding PW-S8 to GGA yields task macro-averages of 69.83%/78.63%, improving over STEPQuant@4 by 38.39/41.21 percentage points on Qwen3.5-4B/9B.

    Source: §7 · p. 14

  • Takeaways & Limitations

    At similar storage budgets, maintaining recent writes achieves higher macro-averages and shorter mean outputs than using capacity solely to increase accumulated-state bit width.

    Source: §6 · p. 14, §7 · p. 14

    The evaluation mainly considers a fixed window configuration, and systematic comparisons of window length, queue precision, and branch-merging strategies remain incomplete.

    Source: §6 · p. 14

Abstract

from arXiv · show

Linear attention stores historical information in a recurrent state of fixed size, providing a compact memory representation for long-context inference. Reducing state precision further lowers per-request storage requirements, but quantized writeback errors enter subsequent recurrences, degrade prediction quality on long-reasoning tasks, and can be accompanied by repetitive generation and excessively long outputs. We propose Global Gradient-aware Allocation (GGA) and the Pure-Write Sliding Window (PW) to address precision allocation and recent-information maintenance. GGA combines candidate quantization residuals with gradient sensitivity to future prediction loss to allocate head bit widths under a model-wide mixed-precision budget. PW independently maintains recent pure writes as FP8 vector pairs, propagates them through the full GDN state transition, and merges them into the low-bit historical state only after they leave the window. Across seven evaluations on Qwen3.5-4B/9B, GGA allocation alone improves over STEPQuant@4 by 19.48/29.84 percentage points. Adding PW-S8 yields seven-task macro-averages of 69.83%/78.63%, improving over STEPQuant@4 by 38.39/41.21 percentage points, respectively. Both precision reallocation and window maintenance improve task scores while mitigating excessively long outputs: at @6, GGA reduces mean generation length, averaged equally across tasks, by 10.45%/14.65% relative to STEPQuant; adding PW-S8 to GGA@4 further reduces mean generation length by 32.32%/36.77%. Comparisons at similar storage budgets further show that using additional capacity to maintain recent writes yields higher task scores and shorter mean outputs than solely increasing the bit width of the accumulated state.

1 Introduction

Fixed-size recurrent states reduce memory growth with sequence length, but still incur per-request storage and access costs. The paper addresses state-quantization sensitivity and writeback errors with gradient-aware precision allocation and a sliding window for recent writes.

1 Introduction

  • Fixed-size recurrent states still require per-request storage and are read and updated at every decoding step, motivating compression for inference deployment.Linear attention compresses historical interactions into a fixed-size recurrent state rather than a sequence-growing KV cache.Source: §1 · p. 1
  • Different heads’ state perturbations can affect later predictions differently, while stepwise writeback errors enter future recurrences.These challenges motivate allocating precision by future prediction loss and maintaining recent information according to write age.Source: §1 · pp. 1–2
  • GGA weights candidate state-residuals by gradients of future language-model loss, then reallocates head bit widths across layers under STEPQuant’s nominal budget.Gradient computation is used only for offline calibration.Source: §1 · p. 2
  • PW separates recent pure writes into floating-point vector branches, propagates them through the full state transition, and merges them into the low-bit historical state according to write age.The old state remains quantized at every step; PW delays only the recent branches’ first incorporation into the low-bit matrix.Source: §1 · p. 2

2 Background and Related Work

The background describes recurrent-state compression, the GDN update and quantized writeback, and prior precision-allocation and delayed-quantization approaches. It positions GGA and PW as addressing future-loss sensitivity and the transformation of recent writes, respectively.

2.1 Linear Recurrent Models and State Compression

  • Linear attention stores historical interactions in a fixed-size recurrent state, but each concurrent request still retains a state that is read and updated during decoding.The section contrasts recurrent-state compression with sequence-growing KV caches and notes related state-compression approaches.Source: §2.1 · p. 2

2.2 GDN Recurrence and Quantized Writeback

  • GDN updates a state/value head using processed query, key, and value activations, retention and key-update coefficients, then reads the current head output.The raw readout is followed by normalization, output gating, and downstream projection.Source: §2.2 · pp. 2–3
  • Naive stepwise quantization reads the candidate state before quantizing it for the next step, so each new writeback error affects subsequent states and predictions.The decoded prior state already contains historical quantization errors.Source: §2.2 · p. 3

2.3 Recurrent-State Quantization

  • GGA weights candidate recurrent-state residuals by gradients of future language-model loss, extending prior allocation methods that capture magnitude, persistence, and local readout effects.Unlike SqueezeLLM’s fixed model weights, recurrent states change with input and their writeback perturbations undergo later transitions and readouts.Source: §2.3 · p. 3
  • PW stores recent pure writes as FP8 vector pairs, transforms them through the full state transition, and merges them individually into the low-bit state according to write age.Unlike LeapQuant’s block buffer and boundary reconstruction, PW leaves the old state updated and quantized at each step.Source: §2.3 · p. 3

3 Motivation

The motivation section identifies a mismatch between head precision and measured prediction sensitivity, and illustrates how a verified reasoning conclusion can be lost during quantized generation. It motivates GGA’s sensitivity-based allocation and PW’s maintenance of recent writes.

3.1 Matching Head Precision to Prediction Sensitivity

  • Figure 1 compares head precision across empirical-impact bins; both allocations average 4.33 bits per value over all heads, with FP16 counted as 16 bits.Source: Figure 1 · p. 4
  • STEPQuant assigns 5.28 bits in the [1.50, 1.80) × 10−4 sensitivity bin but only 3.73 bits in the higher-impact [2.40, 2.70) × 10−4 bin.The authors motivate GGA by weighting candidate residuals with propagated future-loss gradients before recomputing the model-wide allocation.Source: §3.1 · p. 4

3.2 Preserving Recent Conclusions during Reasoning

  • Under GGA@4 without PW, the model correctly lists all ten divisors of 48, then rejects that verified count and resumes searching for missing divisors.The authors connect this failure to the immediate low-bit writeback and repeated quantization of the recent state contribution supporting the conclusion.Source: §3.2 · p. 4
  • The trajectory acknowledges 2 as a divisor on line 1933 but treats it as missing on line 1940; rechecking gives way to 831 consecutive lines of “Wait.”Generation is stopped by the repetition detector after 81,921 tokens, without a final answer.Source: §3.2 · p. 4
  • On the same problem, without PW the model rejects a verified divisor count, rechecks a known factor, and repeats “Wait.”; with PW it retains the count, continues searching, and returns 48.Source: Figure 2 · p. 5

4 Method

The method pairs offline, gradient-aware allocation of head precision with an online window that keeps recent pure writes separate from the low-bit history. GGA uses future-loss sensitivity to allocate a model-wide bit budget, while PW propagates recent writes through the full state transition before merging them into history.

4.2.1 Sensitivity of Future Loss to the Writeback State

  • GGA estimates writeback sensitivity by propagating loss adjoints backward along recorded reference state transitions, excluding the already completed current readout from the new write’s contribution.The writeback-sensitivity adjoint is propagated independently for each calibration segment and is zero at a segment end when no subsequent computation uses that state.Source: §4.2.1 · p. 6

4.2.2 Squared-Gradient-Weighted Candidate Cost

  • GGA weights candidate quantization residuals coordinatewise by squared gradients and averages over calibration snapshots, emphasizing residuals at loss-sensitive coordinates.Unlike the STEP allocation cost, GGA does not additionally multiply Equation (8) by the lifetime term or normalize costs separately by layer or head.Source: §4.2.2 · p. 6

4.2.3 Global Allocation under a Nominal Budget

  • GGA retains STEPQuant’s candidate formats and nominal budget, selects FP16 pivots, and uses global dynamic programming to allocate integer bit widths across the remaining heads.The allocation is determined jointly across layers by candidate costs and the total budget; gradients and costs are used only for offline calibration.Source: §4.2.3 · pp. 6–7
  • At the same all-head average of 4.33 bit/value, GGA assigns 4.56 versus STEPQuant’s 3.73 bits in the [2.40, 2.70) × 10−4 bin and 6.96 versus 5.02 bits in the [2.70, 3.00) × 10−4 bin.Lower-impact bins receive fewer bits on average.Source: §4.2.3 · p. 7

4.3.1 Exact Expansion and Vector Representation of Recent Writes

  • PW expands recent pure writes as rank-one branches, storing each as a vector pair while propagating its left factor and keeping its right factor fixed.The earlier state and recent-write contributions together represent the current state; Figure 3 illustrates the rolling update with m = 3.Source: §4.3.1 · pp. 7–8, Figure 3 · p. 7

4.3.2 Online State Maintenance and Readout Timing

  • At each decoding step, PW uses the old state, propagated recent branches, and current write for readout, then merges the oldest branch if the queue is full and requantizes the old state.The current write enters the queue after readout; the persistent queue holds at most m vector pairs, delaying each new write’s first incorporation into the low-bit old state.Source: §4.3.2 · p. 8
  • With PW-S8, the case-study output terminates naturally with answer 48 at 57,682 tokens, 24,239 fewer tokens than without PW, a reduction of 29.59%.The comparison uses the same prompt, random seed, sampling parameters, and GGA@4 bit-width configuration.Source: §4.3.2 · p. 8

5 Experiments

Across Qwen3.5-4B/9B evaluations, GGA improves task scores under fixed bit-width budgets, and PW further improves scores while reducing output lengths and abnormal stops. At similar storage budgets, maintaining recent writes outperforms directly increasing the accumulated state's bit width on both macro-average scores and mean output length.

5.1 Experimental Setup

  • The evaluation compares Qwen3.5-4B/9B across seven tasks, with @4 and @6 quantized settings, floating-point references, and five or two sampled responses per problem depending on the dataset.Offline calibration uses 32 WikiText-2 training segments of 2048 tokens; generation settings fix prompts and seeds and impose a 100,000-token output limit.Source: §5.1 · p. 9

5.2 Main Results

  • Adding PW-S8 to GGA@4 raises the 4B/9B macro-averages to 69.83%/78.63%, while GGA alone improves over STEPQuant@4 by 19.48/29.84 percentage points.With PW-S8 fixed, GGA also exceeds STEP allocation by 6.16/8.61 percentage points.Source: Table 1 · p. 10, §5.2 · p. 11
  • GGA@6 exceeds STEPQuant@6 by 1.29/2.23 percentage points on 4B/9B, and adding PW-S8 to GGA@6 raises scores to 73.96%/79.88%.At @6, the allocation gap narrows relative to @4, while PW continues to improve task performance.Source: §5.2 · p. 11
  • At @6, GGA reduces task-equal mean output lengths relative to STEPQuant by 10.45%/14.65% on 4B/9B, and adding PW-S8 to GGA@4 reduces them by 32.32%/36.77%.The GGA length reductions occur on all seven tasks; adding PW-S8 to STEPQuant@4 also reduces lengths by 39.63%/41.56%.Source: §5.2 · p. 11
  • With GGA@4+PW-S8, 100K-cap stops fall to 4.61%/2.44%, repetition stops to 0.46%/0.10%, and natural EOS rates reach 94.93%/97.47% on 4B/9B.Naive INT4 also produces very short outputs, but its task scores are near zero; GGA and PW combine shorter outputs with higher scores and fewer abnormal stops.Source: §5.2 · pp. 12–13, Figure 8 · p. 12, Figure 9 · p. 12

5.3 Supplementary Experiments: Higher Precision versus Window Maintenance at Similar Budgets

  • At similar storage budgets, @4+PW-S8 beats @5 on macro-average scores and mean output lengths under both allocations and both models.Under GGA, score gains are 2.61/0.41 percentage points and mean-length reductions are 9.91%/10.80%; the window configurations use 0.10–0.11 more bits/value than @5.Source: §5.3 · p. 13, Figure 10 · p. 13, Figure 11 · p. 13

6 Discussion and Limitations

GGA and PW improve recurrent-state quantization through precision allocation and write maintenance, while also shortening mean outputs on evaluated models. The discussion notes that output mechanisms, PW configurations, efficient kernels, and broader validation remain open areas.

6 Discussion and Limitations

  • At similar storage budgets, window maintenance achieves higher macro-averages and shorter mean outputs than using capacity solely to increase accumulated-state bit width.PW also further mitigates excessively long generation on top of existing precision allocations.Source: §6 · p. 14
  • The mechanisms behind excessively long outputs and repetitive generation remain insufficiently characterized; some responses still hit the token limit or stop through repetition.The paper has not fully characterized how written information is retained and read during recurrences or how quantization perturbations affect later use.Source: §6 · p. 14
  • The evaluation mainly considers a fixed PW window; systematic comparisons of window length, queue precision, and branch-merging strategies remain incomplete.The paper also identifies joint optimization of window maintenance and global precision allocation as future work.Source: §6 · p. 14
  • Dedicated efficient inference kernels for PW have not yet been developed, and end-to-end latency and throughput remain to be evaluated.Planned work includes reducing computation and memory-access overhead in queue maintenance.Source: §6 · p. 14
  • Validation mainly covers Qwen3.5-4B/9B, with limited coverage of other model sizes, families, and recurrent-state quantization methods.The authors plan broader evaluation across architectures and deployment conditions using task accuracy, generation length, and inference-performance metrics.Source: §6 · p. 14

7 Conclusion

The conclusion reports task-score gains from GGA and further gains from PW-S8, alongside shorter outputs. The appendices specify the assumptions behind PW's exact-arithmetic equivalence and detail the allocation, calibration, and online-update procedures.

7 Conclusion

  • At @4, GGA alone improves scores by 19.48/29.84 percentage points, while GGA+PW-S8 reaches 69.83%/78.63% macro-averages, +38.39/41.21 points over STEPQuant@4.At similar storage budgets, GGA@4+PW-S8 reduces mean output length by 9.91%/10.80% versus GGA@5 while increasing macro-average by 2.61/0.41 points.Source: §7 · p. 14

A Equivalence of PW under Exact Arithmetic

  • PW's logical state and current readout match the ordinary recurrence at every step under identical inputs, empty initial queue, and infinite-precision operations.Finite-precision rounding, quantization feedback, and activation changes during free generation fall outside the equivalence assumptions.Source: A Equivalence of PW under Exact… · pp. 15–16
  • Under quantization, Q(O_t)+H_t generally differs from Q(O_t+H_t), so exact-arithmetic equivalence does not guarantee smaller finite-precision errors or matching free-generation trajectories.Here H_t is the sum of recent branches.Source: A Equivalence of PW under Exact… · p. 16

B.1 Frozen Plans and Budget Settings

  • Each model has 768 state/value heads; plans retain 32 FP16 pivots, leaving 736 heads with integer bit-width-sum budgets of 2816, 3584, and 4352 at @4, @5, and @6.Table 2 lists the frozen plan distributions; PW configurations use the corresponding no-window bit-width tables.Source: §B.1 · p. 16, Table 2 · p. 16

B.2 Solving the Global Allocation by Dynamic Programming

  • The global allocation dynamic program minimizes cost over head prefixes at an exact bit-width sum, then backtracks from the total budget to recover the allocation.The preliminary allocation uses the same recurrence with a target budget of average bit width times the number of heads.Source: §B.2 · p. 16

B.3 Calibration and Deployment Settings

  • Standalone GGA calibration uses 32 WikiText-2 sequences of 2048 tokens, samples 64 snapshots from 8192 candidate positions per layer, and freezes allocations for inference.GGA costs are rescaled by a common positive median without changing the optimum; GGA@6+PW-S8 calibration uses identity STE for FP8 and FP16 rounding.Source: §B.3 · p. 17

B.4 FP8 Encoding and Online Updates for PW

  • PW stores recent branches as FP8 vector pairs with per-vector FP32 scales, propagates and re-encodes left factors, and keeps each value vector fixed after enqueueing.The current write contributes to readout using FP32 intermediates before enqueueing; the oldest branch is evicted only after the current readout.Source: §B.4 · p. 17
  • The PW decode algorithm updates the historical state from the evicted branch and then quantizes that state at every step, while retaining a bounded queue for recent writes.The algorithm's reconstructed persistent matrix, FP8 queue, and current q, k, v, γ, and β are its stated inputs.Source: §B.4 · p. 18

B.5 State Storage Accounting and Queue Capacity

The storage accounting measures persistent data arrays for one request's state slot, including state codes, scales, pivots, and reserved PW queues. Under this accounting, PW-S8 adds storage, and the reported values depend on whether queues are reserved for every head.

B.5 State Storage Accounting and Queue Capacity

  • The accounting covers persistent data arrays for one request's state slot but excludes indices, counters, shared plan metadata, unused request slots, temporary workspace, weights, and KV or convolution caches.Queue storage is reserved for all heads when PW is enabled and for none when it is disabled.Source: §B.5 · p. 18
  • For 128 × 128 state matrices and an eight-write window, a queue for each head adds 1.03125 bit/value; reserving queues only for 736 non-pivot heads would add about 0.98828 bit/value averaged over 768 heads.The current layout reserves queues for every head, including pivots.Source: §B.5 · p. 19
  • At similar nominal budgets, @5 requires approximately 5.68 bit/value, while @4+PW-S8 requires approximately 5.78–5.79 bit/value; @6 and @6+PW-S8 require 6.573 and 7.604 bit/value.The accounting includes integer codes, quantization scales, FP16 pivots, and reserved PW queues.Source: §B.5 · p. 19, Table 3 · p. 19

C Protocol and Binned Data for Precision Allocation Diagnostics

This appendix supplements the precision-configuration analysis with its experimental protocol, metric definitions, and binned data. Its analysis uses all state heads of Qwen3.5-4B.

C Protocol and Binned Data for Precision Allocation Diagnostics

C.1 Reference Trajectories and Single Interventions

The diagnostic analysis intervenes on one head’s accumulated state and replays a fixed continuation to measure downstream effects. Its sampling and replay choices distinguish the intervention from both method derivations and method comparisons.

C.1 Reference Trajectories and Single Interventions

  • The H256 diagnostic set uses questions selected independently of method differences and answer correctness, with results averaged equally over questions.Questions come from evaluation tasks, exclude calibration and previously used development questions, and must meet the replay-length requirement.Source: §C.1 · p. 19
  • At each after-call-d checkpoint, the entire accumulated state of only the target head is quantized, reconstructed in FP32, and replayed over the fixed next 256 tokens.The logits predicting g[d] are already determined before intervention.Source: §C.1 · p. 19
  • Teacher forcing fixes token inputs but not post-intervention activations or model states, unlike the fixed exogenous-activation condition used in the algebraic derivations.The already-produced current logits are excluded from the 256 replay steps.Source: §C.1 · p. 19
  • Each head is tested at INT2, INT4, INT6, and INT8 with the frozen candidate quantizer and its specified fitting, impact, grouping, and scale settings.The settings include dual-axis fitting, row-impact, INT2 group32, and FP16 scales.Source: §C.1 · p. 19

C.2 Empirical Quantization Impact and Binned Statistics

The section measures single-head quantization impact over diagnostic questions and checkpoints, then describes that impact by precision and bins heads under a common metric. At assigned widths, the two configurations have nearly equal mean single-head local KL values.

C.2 Empirical Quantization Impact and Binned Statistics

  • The composite impact averages checkpoint results within each question, averages equally over diagnostic questions, and combines the four integer precisions.The underlying predictive-divergence measure covers the subsequent 256 steps; the composite metric is used as a common observational coordinate for the two precision configurations.Source: §C.2 · pp. 19–20
  • Heads are partitioned by composite impact into 11 equal-width bins, with empty bins omitted and all 768 heads included.The bin width is 0.30 × 10−4; the binning is descriptive and follows aggregation across questions and checkpoints.Source: §C.2 · p. 20
  • Both configurations use 32 FP16 pivots and 2,816 total bits across 736 integer heads, giving a weighted mean of 4.33 bit/value across all heads.This accounts for state-value encoding; scale and metadata storage overhead is accounted for separately.Source: §C.2 · p. 20, Table 4 · p. 20
  • 2.16013 × 10−4 for GGA versus 2.15559 × 10−4 for STEPQuant in mean single-head local KL, with GGA approximately 0.21% higher.The statistic queries each head at its assigned bit width and measures a single intervention; persistent joint effects are assessed separately through task evaluation.Source: §C.2 · p. 20

D Evaluation Protocol and Supplementary Statistics

The evaluation freezes task subsets and dependencies, applies shared stopping and answer-accounting rules, and reports supplementary length and termination statistics. Tables and figures cover task-level lengths, similar-budget comparisons, and stopping categories across the stated settings.

D Evaluation Protocol and Supplementary Statistics

  • Task subsets are fixed before inference and shared across models and methods, with frozen LiveCodeBench ordering and upstream evaluation dependencies.For IFBench, the paper uses prompt strict rather than the prompt loose metric usually reported in its paper.Source: D Evaluation Protocol and… · p. 21, Table 5 · p. 21
  • Repetition stops and length-cap terminations are reported separately, while missing final answers remain in the evaluation denominator and may overlap with stopping categories.The final answer is the content after the last </think>; an empty answer after removing termination markers is missing.Source: D Evaluation Protocol and… · p. 21
  • Table 6 reports task-level full output lengths for 4B/9B at @4 and @6 alongside floating-point references, while Table 7 pairs similar-storage-budget scores and mean lengths.Table 6 lengths include thinking and final output, exclude prompts, and retain all stopping types; Table 7 reports seven-task macro-average scores and equally weighted mean lengths.Source: Table 6 · p. 21, D Evaluation Protocol and… · p. 21, Table 7 · p. 22
  • Figures 12 and 13 supplement the @6 stopping-category plots, and Tables 8 and 9 report termination statistics for floating-point references and the @4, @5, and @6 settings.Stopping percentages are aggregated over responses, while mean lengths retain equal task weighting.Source: D Evaluation Protocol and… · p. 22, Table 8 · p. 22, Table 9 · p. 23
  • Table 10 gives per-task mean full output lengths for the supplementary similar-budget comparison between @5 and @4+PW-S8.The settings are paired within each allocation method, using the same accounting convention as Table 6.Source: D Evaluation Protocol and… · p. 22, Table 10 · p. 24
Loading 2610.39931v1…