PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Sergii Kozyrev, Davyd Maiboroda

arXiv:2609.04098v1cs.AI

TL;DR

Earlier 4-bit Qwen3.8-27B builds protected GDN because recurrent quantization errors were expected to accumulate over long contexts. This paper quantizes all 496 linear layers, including GDN, with NVFP4 W4A4 and tests the resulting model. Minima matches BF16 within seed noise across the reported accuracy tasks while being the smallest and fastest-prefill recipe compared.

  • Problem

    Earlier community 4-bit Qwen3.8-27B builds kept GDN projections, especially its decay and write-strength gates, in 8- or 16-bit because recurrence was expected to compound quantization error over long contexts.

    Source: §1 · p. 2

  • Method

    Minima applies post-training NVFP4 W4A4 quantization to all 496 linear layers of Qwen3.8-27B, including GDN.

    Source: §3 · p. 3, §10 · p. 10

  • Results

    All quantized recipes match BF16 task accuracy within seed noise, while Minima’s perplexity gap to BF16 is +0.72 at 4K and +0.49 at 32K.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The paper’s practical recipe is to quantize the recurrent half as well and serve FP8 KV cache with calibrated scales.

    Source: §7 · p. 9, §10 · p. 11

    Evidence covers one model family and size, one quantization format, perplexity to 32K, and retrieval to 64K; behavior at 128K+ is extrapolation.

    Source: §9 · p. 10

Abstract

from arXiv · show

Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4

1 Introduction

The paper tests the intuition that quantization errors accumulate in Gated DeltaNet’s recurrent state by quantizing the recurrent blocks as well as the rest of a hybrid model. It reports that the full W4A4 recipe matches BF16 within seed noise and explains the result through quantization and serving-stack analyses.

1 Introduction

  • The study challenges the assumption that recurrence makes GDN especially vulnerable to quantization because its state is carried across tens of thousands of tokens.Earlier community 4-bit builds protected GDN projections, especially decay and write-strength gates, in 8- or 16-bit.Source: §1 · p. 2
  • Minima quantizes all 496 linear layers, including GDN, and matches BF16 within seed noise across six accuracy suites while being the smallest and fastest-prefill recipe compared.The evaluation uses a fixed serving regime with FP8 KV cache, an identical harness, and per-sample validity checks.Source: §1 · p. 2
  • The mechanism study links NVFP4 block scaling, gate nonlinearities, and delta-rule overwriting to stable errors and a perplexity gap that shrinks with context.It also finds that protected projections were the safest ones to quantize.Source: §1 · p. 2
  • The serving analysis identifies fused-GEMM scale mismatch, multimodal serving paths, and chat-template handling as sources of corrupted or invalid measurements.The global-scale mismatch can silently corrupt GDN gates and make long-context perplexity appear better.Source: §1 · p. 2
  • Calibrated per-layer FP8 KV-cache scales recover 83% of the 32K perplexity penalty without changing throughput, supporting the recipe to quantize everything and ship calibrated scales.The paper reports throughput unchanged within 0.4%.Source: §1 · p. 2

2 Background

Gated DeltaNet combines projections and gates with a per-head recurrent state updated by a gated delta rule. NVFP4 represents 4-bit values with block-level scales, so an outlier’s scale effect is localized to its block.

2 Background

  • A GDN layer projects the residual stream through qkv, output-gate, and two scalar-per-head gate maps; qkv passes through a causal convolution and SiLU before splitting.The per-head state has dimensions 128 × 128 and evolves over L2-normalized keys and queries.Source: §2 · pp. 2–3
  • The decay gate is formed through negative exponential and softplus parameterization, while the write-strength gate uses a sigmoid, placing both gates in (0, 1).The equations define g_t, alpha_t, and beta_t from the gate projections.Source: §2 · p. 2
  • The decay gate forgets state per token, while the write-strength gate scales a correction that replaces the state’s prediction for the current key with its value.The output is gated using RMSNorm modulated by z, then mixed back through out_proj.Source: §2 · p. 3
  • NVFP4 uses E2M1 4-bit values with an E4M3 scale per 16-element block; W4A4 quantizes both weights and activations at this granularity.A block’s largest value fixes its scale, so an outlier degrades only its own 15 neighbors; block max/RMS is at most 4.Source: §2 · p. 3

3 Experimental setup

The experiments compare BF16, two public NVFP4 checkpoints, Minima, and Minima with calibrated KV scales under a shared serving regime. Minima applies NVFP4 W4A4 to all linear layers except the listed non-linear components and output head.

3 Experimental setup

  • Minima quantizes 496 linear projections across GDN, attention, and MLP layers, excluding the lm_head, embeddings, convolutions, and norms.The model is calibrated on a frozen 128-sample × 32K-token set and served after global-scale harmonization.Source: §3 · p. 3
  • Every model uses FP8 KV cache, GPU utilization 0.85, and a 32K generation cap, with task-specific harnesses and per-sample validity checks.WikiText-2 perplexity is measured at 4K and within a single 32K request; RULER retrieval is evaluated at 32K and 64K.Source: §3 · p. 3

4 Main results

The quantized recipes match BF16 task accuracy within seed noise, while Minima quantizes GDN and leads the compared recipes in model size and prefill speed. Its perplexity gap to BF16 is smaller at 32K than at 4K.

4 Main results

  • All quantized recipes match BF16 task accuracy within seed noise; Minima is 2.9× smaller than BF16 and delivers +14–19% prompt throughput at 8K over Unsloth and RadixArk.The table covers four models in one regime; its caption reports that no pair of models is CI-separated on any task.Source: §4 · p. 4, Table 1 · p. 4
  • Minima’s perplexity gap to BF16 is +0.72 at 4K and +0.49 at 32K, and the paper describes it as a per-token effect that context washes out.The recipe ordering is Unsloth < RadixArk < Minima at both context lengths.Source: §4 · p. 4

5 Why GDN survives 4 bits

Using captured 32K inputs, standalone GDN replays, and fake quantization, the study traces why 4-bit errors remain bounded rather than accumulate. Block scaling localizes outliers, gate nonlinearities compress error, recurrence erases state perturbations, and the end-to-end perplexity gap shrinks with position.

5.1 The inputs are not the reason

  • Despite median-layer max/RMS outliers of 63.5 and 10–32% one-hot blocks, A4 error stays 7.5–9.2% across layer roles under block scaling.Weight error (10.5–11.9%) exceeds activation error everywhere, and both remain flat across the 32K window.Source: §5.1 · p. 5, Table 2 · p. 5

5.2 The protected projections are the safest ones

  • Gate projections a and b produce only 2.1% and 2.6% output error despite 11.0% and 8.5% GEMM error, respectively.The gate nonlinearities compress pre-activation error; the three plain GEMMs contribute the larger output errors.Source: §5.2 · p. 5, Table 3 · p. 6

5.3 The recurrence bounds and erases the noise

  • 12.6% state-error plateau (maximum 14.9%) persists across 32K tokens, while a 1% state impulse decays to 1/e within 80–1,382 steps.Each write overwrites state along the current key direction; decay-implied horizons reach 44K–62K tokens.Source: §5.3 · p. 6, Figure 1 · p. 7

5.4 End to end, context washes the error out

  • +0.081 nats in the first half of the 32K window versus +0.011 in the second; Minima’s final-2K-token gap is −0.053.The FP8-KV cost behaves oppositely, rising with position and reaching about 3× the cost for Minima.Source: §5.4 · pp. 6–7

6 Measuring it correctly: serving-stack findings

The serving pipeline required repairs to NVFP4 fused-GEMM scales and evaluation paths before its measurements were reliable. The paper also distinguishes weight and KV-cache costs across the context window and shows why 32K perplexity comparisons require a fixed serving path and protocol.

Per-module calibration vs. fused-GEMM scaling

  • Fused serving GEMMs mis-scaled GDN gates when per-module NVFP4 scales differed, producing deceptively plausible but corrupted results until a checkpoint-side scale repair was applied.The paired scales differed by 1.82× for qkv/z and 2.75× for b/a; after repair, kernel-vs-reference error fell from 0.35/0.57 to 0.002.Source: §6 · pp. 7–8

Position-dependent quantization costs

  • Across the 32K window, weight-quantization cost falls from its largest value at the start to zero or below by the end, while the smaller KV-cache cost rises with position.The KV-cache component is larger for Minima and is the component calibrated scales remove.Source: Figure 2 · p. 8

Composite vs. text-only serving path

  • Text-only extractions were necessary because multimodal-composite serving changes long-context scoring even for pure text.For BF16, PPL@32K was 10.04 on the composite path versus 10.22 text-only.Source: §6 · p. 8

Raw-completion harnesses are invalid for thinking models

  • Raw-completion evaluation omitted the chat template, so thinking-disabled prompts could still trigger <think> and yield invalid MMLU-Pro scores.The valid scores were 80.4/79.7 for BF16/Minima; all short-tier numbers use a chat-template harness with thinking disabled and per-sample validity gates.Source: §6 · p. 8

Context inversion in the base model

  • BF16 Qwen3.8-27B scores the same tokens worse in a 32K request than in isolated 4K windows, so PPL@32K comparisons require a fixed serving path and window protocol.The deterministic perplexity shift was 6.95 →10.35, reproduced in vLLM and the reference implementation.Source: §6 · pp. 8–9

7 KV-cache precision

FP8 KV caching preserves task scores within seed spread and increases capacity, but incurs a larger 32K perplexity cost for Minima than for BF16. Calibrated per-layer scales recover most of that penalty without measurable throughput loss.

FP8 KV-cache cost

  • Across four seeds and six suites, FP8 KV caching at scale 1.0 moved no task score outside seed spread and increased capacity 1.8–1.9×.The 48 GDN layers carry no KV cache; only the 16 attention layers do.Source: §7 · p. 9
  • At 32K, FP8 KV caching added +0.13 perplexity for BF16 and +0.41 for Minima, a 3× larger cost for Minima.The passage says this may reflect Minima’s W4A4 K/V projections leaving less headroom before cache rounding.Source: §7 · p. 9

Calibrated scales

  • Calibrated scales reduced Minima PPL@32K from 10.84 →10.50, recovering 83% of the penalty, while throughput stayed within 0.4%.PPL@4K was unchanged and RULER remained 100 at 32K/64K.Source: §7 · p. 9

8 Related work

The paper places its study at the intersection of low-bit quantization and hybrid recurrent-state mixers, where prior work had not established whether large hybrid models’ recurrent projections could be quantized. It contrasts its calibration-only results with concurrent QAT and related engineering work, and highlights the interaction between FP8 KV caching and W4A4 weights.

Linear attention and hybrids

  • Gated DeltaNet combines the parallelizable delta rule with Mamba2-style gating, and the paper frames its mechanism findings as relevant to the operator class rather than one checkpoint’s training choices.Qwen3.8-27B uses GDN as the dominant mixer in a hybrid stack.Source: §8 · p. 9

Low-bit LLM quantization

  • Prior low-bit work addressed transformer attention and MLPs, while recurrent-state mixer quantization in large hybrids had not, to the authors’ knowledge, been studied.The public Qwen3.8-27B recipes examined exempted GDN.Source: §8 · p. 9
  • Concurrent QUASAR quantized all 496 projections, including GDN, using weights learned by distillation from a BF16 teacher; the paper reports calibration-only PTQ reaches BF16-level task accuracy without that training.QUASAR appeared after the measurement campaign, and its model card reported a brief two-task spot check without a controlled study or account of why the configuration survives.Source: §8 · p. 9
  • A concurrent sub-4-bit engineering variant builds on this result, keeping GDN and attention projections sealed at 4-bit weights while using 3-bit MLP codebooks with a distillation-healed low-rank residual.It loads 12.3 GiB of weights versus Minima’s 17.5 GiB, but its accuracy evaluation uses different protocols and is outside this paper’s scope.Source: §8 · pp. 9–10

KV-cache compression

  • In this hybrid, only 16 of 64 layers cache KV, and FP8 KV costs a W4A4 model 3× more perplexity than BF16 unless calibrated scales are shipped.With calibrated scales, the cost falls below the BF16 model’s own.Source: §8 · p. 10

9 Limitations

The evidence is bounded to one model family and size, one quantization format, and evaluations through 32K perplexity and 64K retrieval; longer-context behavior is extrapolated. The authors also identify recipe, performance, and architecture boundaries that limit broader conclusions.

Scope of evidence

  • The evaluation covers Qwen3.8-27B with NVFP4, reaching 32K-token perplexity and 64K retrieval; 128K+ behavior is extrapolation.The concurrent QAT checkpoint was not benchmarked because it arrived after the measurement campaign and uses a different recipe class.Source: §9 · p. 10
  • Minima+scales task scores are inherited from Minima rather than re-measured, although PPL, RULER, and throughput were re-measured.The authors justify the inherited task scores with the KV ablation showing no task movement for either model.Source: §9 · p. 10

Decode overhead

  • Minima trails RadixArk by 2–4% on decode, which profiling attributes to small-batch NVFP4 activation-quantization overhead rather than the recipe itself.Source: §9 · p. 10

Gates on other architectures

  • The gate-shielding argument depends on log-space softplus/exponential parameterization, so recurrent mixers with linearly parameterized decay may not share it.The passage notes that 0.1% direct noise on α demonstrably harms the state.Source: §9 · p. 10

10 Conclusion

The conclusion reports that full NVFP4 W4A4 quantization, including GDN, matches BF16 across the evaluated tasks while being the smallest and fastest-prefill recipe in its cohort. It connects this result to position-dependent perplexity gaps and describes the kernel numerics used to assess recurrence behavior.

10 Conclusion

  • Minima quantizes all 496 linear layers, including GDN, and matches BF16 across reasoning, knowledge, code, retrieval, and long-context evaluation while being the smallest and fastest-prefill recipe in its cohort.The conclusion attributes the result to block scaling, gate parameterization, and delta-rule overwriting that erases state error faster than decay gates alone.Source: §10 · pp. 10–11

B.2 Throughput

  • The throughput sweep compares decode at concurrency 1/8/32 and single-request prefill; Minima is reported as the fastest-prefill recipe in its cohort.Minima+scales differs from Minima only by the 32 KV scale tensors.Source: §10 · p. 10, Table 5 · p. 13

A Kernel-vs-reference numerics

  • 7.7 × 10^-4 median relative output difference separates the FP32 kernel from the reference, with final states agreeing to 4.4 × 10^-4; the served BF16 kernel sits at a 3.8 × 10^-3 rounding floor.Because that floor exceeds the study’s smallest perturbation, its error-propagation comparisons use clean and perturbed trajectories within the FP32 reference path.Source: A Kernel-vs-reference numerics · p. 12

B.1 Perplexity gap by position

  • The weight gap falls from +0.081 in the first half of the 32K window to +0.011 in the second half, then turns negative in the last bins; the KV gap rises with position and is larger for Minima at every bin.Table 4 reports mean NLL by position and separates the weight-quantization gap from the KV-compression gap.Source: §B.1 · p. 12, Table 4 · p. 13

B.3 Error propagation, full variant table

The error-propagation table reports layer-specific impulse-decay times and gate statistics over 32K tokens. The accompanying precision and statistics notes specify Minima’s quantization map and key evaluation details.

B.3 Error propagation, full variant table

  • 1/e impulse decay occurs after 550/164/281/80/1,382 steps across layers 0/14/30/46/62, while 1/10 decay takes 2,937/2,233/2,270/2,239/2,693 steps.Table 6 reports state error relative to the clean FP32 recurrence and resulting output errors; its impulse case perturbs state once by 1% at t0=1,024.Source: §B.3 · p. 13, Table 6 · p. 14
  • Mean α is 0.862 and mean β is 0.447; per-layer mean horizons 1/(1 − α) are 43,970/1,895/4,156/8,342/61,659 tokens.The table also includes synthetic multiplicative noise applied directly to the α and β gates.Source: §B.3 · p. 13, Table 6 · p. 14

B.4 Precision maps and statistics notes

  • Minima quantizes 496 linear tensors to NVFP4 W4A4 while keeping embeddings, lm_head, GDN conv1d, norms, and A_log/dt_bias in BF16.The quantized tensors comprise five projections per GDN layer, four per attention layer, and three MLP projections per layer.Source: §B.4 · p. 14
  • Minima scored 26/30 on AIME’25 in each of four seeds, giving a degenerate confidence interval; the passage identifies per-seed scores as the honest statement.There were zero request errors in every run.Source: §B.4 · p. 14

Related papers

Loading 2609.04098v1…