PaperlayerView exact revision on arXiv

Source-linked AI summary

PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

Patrick Haller, Filipe Laitenberger, Alan Akbik

arXiv:2610.29545v1cs.CL

TL;DR

Linear-attention models store a recurrent state for each active sequence, and existing per-token quantizers report accuracy loss at four bits. PACE rounds periodically with a decode buffer and channel-aligned scales. It approaches DAMP INT8 accuracy on three larger models at about two thirds of its resident memory, while PACE-4 scores above Q-Mamba DSQ at equal resident bits.

  • Problem

    Existing recurrent-state quantizers round after every generated token, and methods evaluated at four bits report accuracy loss unless precision is raised or weights are fine-tuned.

    Source: §1 · p. 2

  • Method

    PACE is a post-hoc four-bit state quantizer that rounds every few generated tokens, buffers intervening tokens, and uses channel-aligned scale groups.

    Source: §1 · p. 2, §5.2 · p. 7

  • Results

    PACE-4/8 comes close to DAMP INT8 on the three larger backbones at about two thirds of its resident memory, while PACE-4 scores higher than Q-Mamba DSQ at equal resident bits.

    Source: §8 · p. 11

  • Takeaways & Limitations

    Periodic rounding also keeps the decoding speedup of the decode buffer.

    Source: §8 · p. 11

    The error-bound decay, column structure, and head ranking are measured only on Qwen3.5 and Qwen3.6 hybrid models and are not shown to carry over to other model families or token mixers.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator. Existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is fine-tuned.\par We introduce PACE, a post-hoc four-bit state quantizer. PACE rounds the state only once every few generated tokens, as chunked kernels already do once per chunk during prefill, and keeps the tokens generated in between in a small buffer. Rounding less often reduces the error that accumulates in the state. Additionally, we find that the large entries of the gated-delta state gather in columns, because some channels of the update written at each token remain large across many tokens. PACE therefore shares each quantization scale within one column. It also keeps the heads that retain rounding error longest at eight bits. On four hybrid models from 4B to 35B parameters, PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two thirds of its memory, PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory, and rounding every few tokens preserves the decoding speedup of the buffer.

1 Introduction

Linear-attention models store a fixed-size recurrent state for every active sequence, but existing per-token state quantizers report accuracy loss at four bits. PACE combines periodic rounding, channel-aligned scales, and selective eight-bit heads to improve the accuracy–memory trade-off.

Motivation

  • At 64 concurrent sequences, fp32 recurrent states use 10–41% of combined weights-and-states memory, while existing per-token four-bit quantizers report accuracy loss.Prior remedies raise precision to eight bits or fine-tune model weights.Source: §1 · p. 2

Periodic rounding

  • PACE is a post-hoc four-bit quantizer that rounds every dP generated tokens and buffers the intervening tokens for a single state advance.The buffer’s memory is included in reported totals.Source: §1 · p. 2

Channel alignment and head precision

  • PACE groups state entries within columns aligned to delta-value channels, while PACE-4/8 keeps the quarter of heads retaining rounding error longest at eight bits.PACE-4 instead keeps all heads at four bits.Source: §1 · p. 2

Contributions

  • PACE’s contributions include a rounding-error bound and buffer-aware memory accounting, plus Triton kernels that preserve the decode buffer’s speedup.The paper also reports comparisons with DAMP and Q-Mamba on four hybrid models from 4B to 35B parameters.Source: §1 · p. 2

2 Background and Related Work

The background describes recurrent gated-delta states, chunk-wise computation, decode buffers, and prior state-quantization approaches. It also situates recurrent-state rounding error within the fixed-point filter problem.

2.1 Gated Delta Rule

  • The gated-delta rule applies a data-dependent decay and removes the state component along the current key before writing; its delta value is the residual value scaled by write strength βt.The state is read through the unchanged read-out.Source: §2.1 · p. 3

2.2 Chunk-wise Kernels

  • Chunk-wise kernels compute a chunk’s outputs from its starting state and tokens, using dense arithmetic; only the states at chunk boundaries are formed.The states inside the chunk are never written.Source: §2.2 · p. 4

2.3 Decode Buffers

  • Decode buffers hold recent keys and values and advance the state once per flush, removing per-token state writes from memory traffic.Prior buffer methods do not quantize the state.Source: §2.3 · p. 4

2.4 State Quantization

  • Prior state quantizers use different scaling and rounding schemes, but round after every generated token; published four-bit methods either lose accuracy or require fine-tuning.DAMP retains selected values in fp16, while other approaches use eight-bit state storage.Source: §2.4 · pp. 4–5

Error propagation.

  • For a recurrence with decay a, rounding noise injected at every step accumulates to σ2/(1 − a2), a fixed-point filter error setting.Prior work reduces error per rounding, whereas this paper reduces the number of roundings.Source: Error propagation · p. 5

3 Rounding Period and Error Propagation

The rounding period is the interval between state writes: buffering lets decoding advance the state once per flush rather than once per token. Longer periods reduce the rounding error retained by a head, with retention also depending on the head’s decay.

3.1 The Rounding Period

  • In unbuffered decoding, the state is formed at every token; under a decode buffer, it advances once per flush, making the period the number of buffered tokens.Prefill rounds once per chunk without an additional kernel call; the prefill period equals chunk length C and is fixed at 256 tokens in benchmark evaluations.Source: §3.1 · p. 5

3.2 Error Propagation

  • Between roundings, the recurrence carries injected error forward while decay reduces the retained energy.For a unit-norm key, the recurrence multiplier’s norm is at most the head’s decay; independent rounding errors accumulate across successive roundings.Source: §3.2 · p. 5
  • A head retains less rounding error as the rounding period grows, while greater head decay increases retained error.In the gated-delta rule, decay is a per-head scalar, so the retained-error quantity is per head.Source: §3.2 · p. 5

4 Channel Structure of the Recurrent State

The state’s large entries concentrate in columns, and experiments trace that structure to delta-value channels that remain large across tokens. This motivates grouping quantization scales along the key axis within one column.

4 Channel Structure of the Recurrent State

  • A group-wise quantizer shares one scale across consecutive state entries, and its error depends on which entries share a group.Groups can run along the key axis within a column or along the value axis within a row.Source: §4 · pp. 5–6

4.1 Column Structure of the State

  • Groups along the value axis have a larger max/RMS ratio than key-axis groups at every tested group width, with the difference growing as width increases.The larger ratio means fewer quantization levels for a typical entry in the group.Source: §4.1 · p. 6
  • The large entries of the recurrent state concentrate in columns: within a head, they occur at the same value index for every key index.Source: §4.1 · p. 6

4.2 Channel Identity of the Delta Value

  • Permuting delta-value channels at every token yields higher max/RMS for key-axis groups on all four backbones, supporting the finding that persistent large channels produce the columns.Permuting key entries instead widens the difference between the two axes.Source: §4.2 · p. 6

5 Method

PACE combines periodic state rounding with a decode buffer, channel-aligned scale groups, and optional per-head precision allocation. PACE-4 uses the first two components, while PACE-4/8 adds head allocation.

5 Method

  • PACE-4 uses periodic rounding, a decode buffer, and channel-aligned groups; PACE-4/8 adds head-level precision allocation.Figure 4 plots mean score against resident bits per value for uniform value-axis states on three Qwen3.5 backbones, with lines for widths and decode periods.Source: §5 · p. 6, Figure 4 · p. 7

5.1 Periodic Rounding with a Decode Buffer ( A⃝B⃝)

  • PACE rounds every dP tokens and buffers intervening tokens; resident bits per value count both the stored state and buffer.The buffer stores the decay, key, and value for each token; uniform configurations use eight-bit keys and values, while PACE-4/8 uses four-bit keys and bf16 values.Source: §5.1 · pp. 6–7

5.2 Channel-Aligned Scale Groups ( C⃝)

  • Channel-aligned groups take consecutive entries from one state column and share an fp16 scale, unlike prior value-axis groups that span columns.Uniform configurations use symmetric rounding at b + 16/g bits per value; the mixed configuration uses an affine grid and stores a zero point.Source: §5.2 · p. 7

5.3 Head-Level Precision Allocation ( D⃝)

  • PACE-4/8 assigns eight bits to heads retaining the largest rounding error and four bits to the rest, allocating precision per head.The ranking is computed once per checkpoint on eight held-out 2,048-token windows; a quarter of heads per layer at eight bits gives a 5.66-bit payload per value, the same as DAMP INT4.Source: §5.3 · p. 7

6 Experimental Setup

The evaluation compares quantized states on four hybrid Qwen backbones across language, instruction-following, reasoning, and long-context benchmarks. It includes full-precision, DAMP, and Q-Mamba DSQ baselines under a specified zero-shot greedy-decoding protocol.

Models

  • The study evaluates Qwen3.5-4B, 9B, and 27B, plus Qwen3.6-35B-A3B hybrid models with gated-delta and full-attention layers.The gated-delta layers run on fla kernels.Source: §6 · p. 8

Benchmarks

  • Benchmarks cover GSM8K, MMLU-Pro, IFEval, RULER variable tracking at 32,768 tokens, MATH-500, and WikiText-103 perplexity.Generative tasks are zero-shot with the chat template and thinking disabled.Source: §6 · p. 8

Baselines

  • Baselines are full precision, DAMP, and Q-Mamba DSQ, reimplemented without its published fine-tuning step.DAMP INT8 is the resident-memory reference at its published 9.9 bits per value; DAMP rounds after every generated token.Source: §6 · p. 8

Protocol

  • Benchmarks use greedy decoding; quantized configurations default to a four-bit base width, int8 decode buffer, and dP=8.GSM8K and MATH-500 use exact match, IFEval uses prompt-level strict accuracy, and RULER uses variable-tracking accuracy.Source: §6 · p. 8

7 Results

The results show that periodic rounding improves accuracy over per-token rounding, while PACE’s key-axis groups and head allocation contribute to its benchmark comparisons. The evaluation also measures state drift and decoding throughput.

7.1 Effect of the Decode Period

  • Rounding every eight tokens scores above per-token rounding at every width below eight bits on all four backbones.A period of eight closes most of the gap closed by a period of 32, with a quarter of the buffer memory; a full-precision gap remains on the three larger backbones, concentrated in IFEval and RULER-32k.Source: §7.1 · p. 8

7.2 Main Results

  • PACE-4 scores higher than Q-Mamba DSQ on every backbone at the same resident bits, while PACE-4/8 holds 0.69 of DAMP INT8’s resident bits and is close to its average on the three larger backbones.PACE-4/8 scores below DAMP INT8 on IFEval on those larger backbones; on Qwen3.5-4B, its average is lower.Source: §7.2 · pp. 8–9, Table 1 · p. 9

7.3 Ablations

  • Key-axis groups yield lower perplexity than value-axis groups with and without head allocation, while head allocation yields lower perplexity than DAMP INT4 at equal resident bits.Without head allocation, key-axis groups also yield a higher IFEval score than value-axis groups.Source: §7.3 · pp. 9–10

7.4 State Drift under Periodic Rounding

  • Drift decreases as the decode period grows at every token position for all three four-bit quantizers, with the same ordering on Qwen3.6-35B-A3B.The comparison uses median relative state error across recurrent layers on Qwen3.5-4B; Figure 5 plots relative state error against token position, with one curve per period.Source: §7.4 · p. 10, Figure 5 · p. 10

7.5 Throughput

  • Periodic INT4 decoding has lower time per output token than per-token bf16 decoding from batch 16 upward, and the difference grows with batch size.The bf16-state decode buffer accounts for most of the reduction, and the INT4 state reduces time per output token slightly further.Source: §7.5 · p. 11, Figure 6 · p. 11

8 Conclusion

PACE is a post-hoc four-bit state quantizer that rounds periodically, aligns scale groups with state columns, and allocates eight bits to selected heads. Its mixed-precision variant approaches DAMP INT8 accuracy on the three larger backbones at about two thirds of its resident memory, while PACE-4 exceeds Q-Mamba DSQ at equal resident bits.

8 Conclusion

  • PACE-4/8 comes close to DAMP INT8 accuracy on the three larger backbones at about two thirds of its resident memory, while PACE-4 scores higher than Q-Mamba DSQ at equal resident bits.A period of eight reduces accuracy loss at every width below eight bits on four backbones, and periodic rounding keeps the decode-buffer speedup.Source: §8 · p. 11

Limitations

The evaluation and design claims are bounded by the tested models, quantizers, and configuration choices. Several broader settings remain unmeasured.

Limitations

  • Four evaluated hybrids come from the Qwen3.5 and Qwen3.6 families; transfer to other model families, all-recurrent models, or token mixers with different updates or decay is unshown.The head decays, column structure, and head ranking are measured only on these models.Source: Limitations · p. 12
  • Periodic rounding is tested with PACE, DAMP INT4, and Q-Mamba DSQ, but not with static scales, stochastic rounding, or state rotations.Periodic rounding changes the rounding schedule while leaving quantizer choice open.Source: Limitations · p. 12
  • Benchmark evaluations fix prefill rounding at the kernel’s 256-token chunk, so they do not measure the effect of changing prefill frequency.The explored configuration space also covers only part of the possible periods, widths, scale groups, and eight-bit head fractions.Source: Limitations · p. 12
  • Throughput is timed only for a uniform four-bit state, not for PACE’s key-axis groups or head-level precision allocation.The configuration space is large, and only part of it is explored.Source: Limitations · p. 12

A Per-Benchmark Scores across Widths and Periods

The section supplies per-benchmark score and state-drift visuals, but the provided passages describe their scope rather than their outcomes.

A Per-Benchmark Scores across Widths and Periods

B State Drift

  • Figure 7 presents state drift for Qwen3.6-35B-A3B, measured as in Figure 5.The supplied passage does not state the plotted values or comparison outcome.Source: Figure 7 · p. 16

C State Statistics along the Two Axes

The section contrasts state-axis statistics with accuracy effects from buffered-key and buffered-value rounding, then describes zero-point storage and throughput and memory measurements.

C State Statistics along the Two Axes

  • The bulk log-|S| variance decomposition favors the key-index row effect, but max/RMS tail statistics and channel permutations support key-axis groups for large entries.At g=128, Table 4 reports that permuting delta-value channels gives key-axis groups the larger max/RMS, while permuting key entries widens the difference; the table also tracks large entries in the heaviest 5% of columns.Source: C State Statistics along the Two Axes · p. 16, Table 4 · p. 16

D Separate Key and Value Buffer Widths

  • Rounding buffered values to four bits reduces accuracy more than rounding buffered keys, despite the rank-one update predicting the opposite ordering.The kernel L2-normalizes keys after rounding, so only key-error direction reaches the state, while value error enters unchanged; either tensor has similar relative input error.Source: D Separate Key and Value Buffer Widths · p. 16, Table 5 · p. 17

E Integer Zero Point

  • Snapping the affine zero point to the payload grid makes decayed entries round to zero by construction and stores the zero point as a b-bit code.The resulting side information is (16 + b)/32 bits per value; affine groups use 4.625 bits per value at four bits and 8.75 at eight bits.Source: E Integer Zero Point · p. 17

F Throughput Measurement and Memory

  • State-update latency is TPOT minus TPOT with the recurrent operator replaced by a constant; Table 6 reports per-sequence recurrent-state memory and implied bits per value.Timings use CUDA graphs and the median over 15 timing blocks; Table 6 also lists resident bits and an “other” category for buffered decays and kernel scratch space.Source: F Throughput Measurement and Memory · p. 17, Table 6 · p. 17

Related papers

Loading 2610.29545v1…