PaperlayerView exact revision on arXiv

Source-linked AI summary

SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Qingwen Lin, Boyan Xu, Xiao Liu, Zhifeng Hao, Ruichu Cai

arXiv:2608.22354v2cs.LGcs.AI

TL;DR

Delta-Rule recurrent models can retain O(1) inference memory yet become unstable during extreme-context extrapolation, and existing approaches do not directly address fixed-size recurrent-state instability. The paper diagnoses localized norm explosion in RWKV-7 and proposes SANE, adaptive tanh compression at chunk boundaries that preserves intra-chunk parallel structure. SANE retains short-context performance and functional reasoning after a 100M-token prefix within a safe threshold range, while high thresholds expose a capacity–stability trade-off.

  • Problem

    Extreme-context extrapolation can destabilize fixed-size recurrent states, while existing methods do not directly address this instability.

    Source: §1 · p. 2, §6 · p. 15

  • Method

    SANE applies adaptive tanh compression at chunk boundaries to suppress localized state anomalies while preserving intra-chunk recurrence and parallel structure.

    Source: §1 · p. 2, §7 · p. 15

  • Results

    SANE matches baseline short-context reasoning without significant degradation and retains functional reasoning after a 100M-token prefix, whereas the baseline overflows.

    Source: §1 · p. 2, §7 · p. 16

  • Takeaways & Limitations

    Threshold choice reveals a capacity–stability trade-off: numerical stability alone does not guarantee functional reasoning.

    Source: §1 · p. 2, §7 · p. 16

    Evaluation is confined to RWKV-7 (0.4B), and applicability to other Delta-Rule models remains to be empirically verified.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: \textbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose \textbf{State Anomaly Neutralization (SANE)}, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3 \le α\le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$, SANE retains functional reasoning ($33.46$--$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($α\ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

1 Introduction

Delta-Rule models offer fixed-size recurrent memory but can fail under extreme-context extrapolation through localized state growth. The paper diagnoses this failure and introduces SANE, which stabilizes state transfer while preserving reasoning within a safe threshold range.

Motivation

  • RWKV-7’s fixed-size state provides O(1) inference-state memory, but extreme-context extrapolation can still produce instability and numerical overflow.The state norm eventually exceeds 10^19 near 20M tokens.Source: §1 · pp. 1–2

Diagnosis

  • Localized norm explosion, rather than global state saturation, explains the observed extreme-context failure.Most state entries remain small while a few directions accumulate extreme values through uneven injections and transition-induced amplification.Source: §1 · p. 2

Method

  • SANE applies adaptive element-wise tanh compression at chunk boundaries to suppress anomalies while preserving intra-chunk recurrence and parallel scanning.The intervention adds fewer than 0.1% parameters and approximately preserves the low-magnitude background.Source: §1 · p. 2

Evaluation

  • 43.30–44.23 vs. 43.88 on 11 short-context reasoning benchmarks shows no statistically significant degradation for α ≥3.The reported tests have p ≥0.154, whereas α ≤2 impairs short-context performance.Source: §1 · p. 2
  • 33.46 at α=5 after a 100M-token prefix demonstrates retained functional reasoning, while α ≥8 remains numerically stable but collapses reasoning capability.The threshold sweep exposes a capacity–stability trade-off: numerical stabilization alone is insufficient.Source: §1 · p. 2

2 Preliminaries

Delta-Rule models use affine fixed-size state updates that compose associatively, enabling chunkwise parallel computation. RWKV-7 instantiates this framework with a diagonal-plus-low-rank transition and rank-one state injection.

Chunkwise Computation

  • Transformations can be inserted at chunk boundaries without altering intra-chunk parallelism.This compatibility motivates chunk-boundary interventions such as SANE.Source: §2 · p. 3, §2.1 · p. 4

Affine Delta-Rule Recurrence

  • Delta-Rule models maintain a fixed-size state matrix with affine updates St = St−1Mt + Kt.Mt governs state transition, while Kt injects information from the current token.Source: §2.1 · p. 3
  • Two consecutive affine updates compose associatively, allowing historical injections and transformed initial states to be combined chronologically.The resulting state represents the transformed initial state plus propagated token injections.Source: §2.1 · p. 3

Chunkwise Computation

  • For a chunk of size C, prefix states can be computed with an associative scan of O(log C) parallel depth.The scan computes all within-chunk prefix pairs from inputs independent of the initial chunk state.Source: §2.1 · pp. 3–4

RWKV-7 Instantiation

  • RWKV-7 uses a data-dependent vector decay and a generalized Delta Rule with diagonal-plus-low-rank transitions and rank-one state injection.The low-rank modification changes the transition, while the rank-one term injects the current token’s state information.Source: §2.2 · p. 4

3 Method: State Anomaly Neutralization

SANE diagnoses RWKV-7’s extreme-context failure as localized norm explosion atop a relatively sparse substrate and applies adaptive compression during inter-chunk state transfer. The method preserves low-magnitude state entries, suppresses anomalously large values, and retains chunkwise execution with bounded overhead.

Diagnosis

  • RWKV-7’s extreme-context instability is characterized by localized hotspots and norm growth atop a persistent low-magnitude background, rather than global state saturation.Most entries remain small while a few locations intensify by several orders of magnitude; the observed pattern is explained through decay and uneven recurrent updates, without constituting a general sparsity guarantee.Source: §3.1 · pp. 4–5

State Anomaly Neutralization

  • SANE applies adaptive element-wise compression to each chunk’s terminal state before it is transferred to the next chunk.The intervention targets inter-chunk state transfer, preventing extreme values from being repeatedly carried forward while leaving intra-chunk recurrence unchanged.Source: §3.2 · pp. 5–6

Compression behavior

  • Low-magnitude entries are approximately preserved, whereas entries large relative to their head-specific threshold are smoothly compressed toward ±τ.Independent thresholds vary across heads, accommodating spatially nonuniform state growth.Source: §3.2 · pp. 5–6

Adaptive thresholding

  • The threshold is generated from the current chunk’s last-token representation, with α controlling compression strength.Smaller α produces stronger compression, while larger α preserves a wider range of state magnitudes.Source: §3.2 · p. 5

Cost and compatibility

  • SANE preserves the existing intra-chunk parallel structure and sequence-length-independent inference memory.In the RWKV-7-0.4B implementation, it adds fewer than 0.1% parameters; detailed end-to-end throughput and latency evaluation is left for future work.Source: §3.2 · p. 6, §3.3 · p. 6

4 Experiments

Experiments evaluate SANE on RWKV-7 for short-context mathematical reasoning and extreme-context extrapolation. SANE preserves short-context performance in a safe threshold regime and prevents the baseline’s instability over 100M-token streams, but excessive thresholds trade functional reasoning for numerical stability.

Experimental Setup

  • RWKV-7 serves as the experimental testbed, with mathematical reasoning used to assess stable capabilities after an ultra-long prefix rather than verbatim prefix memorization.The experiments use a pretrained 0.4B checkpoint under a constrained computational setup.Source: §4.1 · p. 7

Short-Context Mathematical Reasoning

  • 43.30–44.23 average accuracy matches the 43.88 baseline without statistically significant differences for α ≥3 across 11 short-context benchmarks.All paired tests have p ≥0.154; α ≤2 instead significantly impairs short-context reasoning.Source: §4.2 · pp. 9–10
  • SANE’s post-hoc SFT setting introduces an adaptation cost, and the hypothesis that pretraining integration would absorb it remains experimentally unverified.After convergence, short-context performance is almost fully recovered relative to the baseline.Source: §4.2 · p. 10

Extrapolation Reasoning: Functional Reasoning at 100M Tokens

  • 33.46 average accuracy at α=5 remains functional after 100M tokens, whereas accuracy declines monotonically as α increases across three threshold regimes.The safe regime is α ≤5, α=6–7 shows a steep decline, and α ≥8 scores 0.15 or below despite bounded perplexity.Source: §4.4 · p. 11
  • Numerical stability and functional health diverge: α ≥8 remains numerically bounded but loses symbolic reasoning capability, exposing a capacity–stability trade-off.The results attribute SANE’s effectiveness to selective compression that suppresses anomalous growth while preserving functional reasoning in the safe regime.Source: §4.4 · pp. 11–12

5 Analysis

SANE’s effectiveness is explained by the interaction between relative sparsity in the recurrent state and scale-selective compression. The threshold α creates a capacity–stability trade-off: moderate values preserve reasoning while suppressing anomalies, whereas extreme values damage either stability or capability.

5.1 Sources of Relative Sparsity

  • Relative sparsity reflects global L2 shrinkage pressing most infrequently updated directions toward small magnitudes while low-rank updates sustain larger values in limited subspaces.This is presented as a mechanistic interpretation rather than a rigorous sparsity guarantee.Source: §5.1 · pp. 12–13

5.2 Why SANE Works

  • SANE’s scale selectivity preserves small background values while compressing anomalously large state entries.Healthy background values remain below 0.1, whereas only a few directions accumulate substantially larger magnitudes.Source: §5.2 · p. 13
  • Chunk-boundary tanh compression interrupts cumulative amplification of anomalous values without changing the intra-chunk parallel computation structure.The intervention compresses values before they can be repeatedly amplified across chunks.Source: §5.2 · p. 13

5.3 The Capacity–Stability Dilemma

  • Within α ≤5, selective compression suppresses anomalous growth while preserving functional reasoning, but larger α values become insufficient to cover anomaly magnitudes.The threshold spectrum experimentally identifies scale selectivity as the core mechanism of effectiveness.Source: §5.2 · p. 13, §5.3 · pp. 13–14
  • The safe range 3 ≤α ≤5 balances capacity and stability because legitimate large values overlap numerically with early anomalies.α=3 is reported as Pareto-optimal within the tested range.Source: §5.3 · pp. 14–15

6 Related Work

Related long-context methods largely target Transformer-specific positional encodings or attention mechanisms, while Delta-Rule models face a state-based bottleneck. Existing RWKV and linear-RNN approaches also alter the native architecture or depend on model-specific components, limiting direct transfer to Delta-Rule extrapolation.

Transformer Extrapolation

  • Transformer extrapolation methods primarily modify rotary positional embeddings or attention to address Transformer-specific long-context failures.These approaches include interpolation, extrapolation, and attention-window expansion strategies.Source: §6 · p. 15

Delta-Rule Bottleneck

  • Delta-Rule models instead transfer cross-context information through a fixed-size state matrix, so their bottleneck originates in the state rather than positional encoding.Consequently, Transformer-focused methods cannot be directly transferred.Source: §6 · p. 15

Related Linear-RNN Approaches

  • Existing improvements for RWKV and similar linear RNNs either require hybrid architectures pretrained from scratch or rely on selective-scan mechanisms specific to Mamba-like models.These approaches alter Delta-Rule’s native architecture or depend on components it does not use.Source: §6 · p. 15

7 Conclusion

The paper identifies localized state norm explosion as the extreme-context failure mode of RWKV-7 and introduces SANE to compress anomalies at chunk boundaries. SANE preserves short-context reasoning within a safe threshold range and retains functional reasoning after a 100M-token prefix, while threshold choices expose a capacity–stability trade-off.

7 Conclusion

  • RWKV-7 exhibits localized norm explosion atop a relatively sparse substrate rather than global state saturation.The proposed interpretation attributes this pattern to uneven injections and transition-induced amplification overcoming decay in a small number of directions.Source: §7 · p. 15
  • SANE applies adaptive tanh compression at chunk boundaries to preserve low-magnitude state values while suppressing extreme values.It is introduced into a pretrained RWKV-7 model through supervised fine-tuning.Source: §7 · p. 15
  • SANE causes no statistically significant short-context degradation within a safe threshold range and retains functional symbolic reasoning after a 100M-token prefix, unlike the overflowing baseline.The full threshold spectrum also shows that numerical stabilization alone does not guarantee functional health.Source: §7 · p. 16

Limitations

The evaluation is limited to a 0.4B-parameter RWKV-7 model and a plug-and-play supervised-fine-tuning setup. Generalization to other Delta-Rule architectures, larger models, and SANE integrated during pretraining remains unverified.

Scope

  • Empirical validation is confined to RWKV-7 (0.4B), so effectiveness on other Delta-Rule architectures and larger scales remains to be verified.The paper specifically identifies 7B and 14B models as future evaluation targets.Source: Limitations · p. 16

Training Setup

  • SANE is added to an already pretrained checkpoint through supervised fine-tuning, incurring an adaptation cost and making current short-context results a conservative lower bound.Integrating SANE during pretraining is left for future work.Source: Limitations · p. 16

A Training and Evaluation Details

The study releases a RWKV-7 SANE kernel and evaluates models trained from a shared checkpoint using JAX-based training and multiple reasoning and long-context evaluation procedures.

Evaluation and release

Evaluation

  • The 100M-token reasoning evaluation places the entire mixed_chat sequence before the mathematical test problems.The mixed_chat sequence is constructed by concatenating training splits from DeepMath-103K and Chinese-DeepSeek-R1-Distill-data-110k.Source: A Training and Evaluation Details · p. 19

Training setup

  • Both baseline and SANE models start from the same pretrained RWKV-7-0.4B-G1D checkpoint and use KerasHub with the JAX backend.Source: A Training and Evaluation Details · p. 19
  • Training uses mixed bfloat16 precision, Muon optimization, cosine learning-rate scheduling, and a peak learning rate of 5e-5.The learning rate warms up over the first 3% of steps before cosine decay.Source: A Training and Evaluation Details · p. 19

SANE implementation

Evaluation

Related papers

Loading 2608.22354v2…