PaperlayerView exact revision on arXiv

Source-linked AI summary

SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Qingwen Lin, Boyan Xu, Xiao Liu, Zhifeng Hao, Ruichu Cai

arXiv:2608.22354v2cs.LGcs.AI

TL;DR

Delta-Rule recurrent models can retain O(1) inference memory yet become unstable during extreme-context extrapolation, and existing approaches do not directly address fixed-size recurrent-state instability. The paper diagnoses localized norm explosion in RWKV-7 and proposes SANE, adaptive tanh compression at chunk boundaries that preserves intra-chunk parallel structure. SANE retains short-context performance and functional reasoning after a 100M-token prefix within a safe threshold range, while high thresholds expose a capacity–stability trade-off.

  • Problem

    Extreme-context extrapolation can destabilize fixed-size recurrent states, while existing methods do not directly address this instability.

  • Method

    SANE applies adaptive tanh compression at chunk boundaries to suppress localized state anomalies while preserving intra-chunk recurrence and parallel structure.

  • Results

    SANE matches baseline short-context reasoning without significant degradation and retains functional reasoning after a 100M-token prefix, whereas the baseline overflows.

  • Takeaways & Limitations

    Threshold choice reveals a capacity–stability trade-off: numerical stability alone does not guarantee functional reasoning.

  • Takeaways & Limitations

    Evaluation is confined to RWKV-7 (0.4B), and applicability to other Delta-Rule models remains to be empirically verified.

Abstract

from arXiv · show

Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: \textbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose \textbf{State Anomaly Neutralization (SANE)}, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3 \le α\le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$, SANE retains functional reasoning ($33.46$--$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($α\ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

1 Introduction

Delta-Rule models offer fixed-size recurrent memory but can fail under extreme-context extrapolation through localized state growth. The paper diagnoses this failure and introduces SANE, which stabilizes state transfer while preserving reasoning within a safe threshold range.

  • Motivation: RWKV-7’s fixed-size state provides O(1) inference-state memory, but extreme-context extrapolation can still produce instability and numerical overflow.The state norm eventually exceeds 10^19 near 20M tokens.
  • Diagnosis: Localized norm explosion, rather than global state saturation, explains the observed extreme-context failure.Most state entries remain small while a few directions accumulate extreme values through uneven injections and transition-induced amplification.
  • Method: SANE applies adaptive element-wise tanh compression at chunk boundaries to suppress anomalies while preserving intra-chunk recurrence and parallel scanning.The intervention adds fewer than 0.1% parameters and approximately preserves the low-magnitude background.
  • Evaluation: 43.30–44.23 vs. 43.88 on 11 short-context reasoning benchmarks shows no statistically significant degradation for α ≥3.The reported tests have p ≥0.154, whereas α ≤2 impairs short-context performance.
  • Evaluation: 33.46 at α=5 after a 100M-token prefix demonstrates retained functional reasoning, while α ≥8 remains numerically stable but collapses reasoning capability.The threshold sweep exposes a capacity–stability trade-off: numerical stabilization alone is insufficient.

2 Preliminaries

Delta-Rule models use affine fixed-size state updates that compose associatively, enabling chunkwise parallel computation. RWKV-7 instantiates this framework with a diagonal-plus-low-rank transition and rank-one state injection.

  • Chunkwise Computation: Transformations can be inserted at chunk boundaries without altering intra-chunk parallelism.This compatibility motivates chunk-boundary interventions such as SANE.
  • Affine Delta-Rule Recurrence: Delta-Rule models maintain a fixed-size state matrix with affine updates St = St−1Mt + Kt.Mt governs state transition, while Kt injects information from the current token.
  • Affine Delta-Rule Recurrence: Two consecutive affine updates compose associatively, allowing historical injections and transformed initial states to be combined chronologically.The resulting state represents the transformed initial state plus propagated token injections.
  • Chunkwise Computation: For a chunk of size C, prefix states can be computed with an associative scan of O(log C) parallel depth.The scan computes all within-chunk prefix pairs from inputs independent of the initial chunk state.
  • RWKV-7 Instantiation: RWKV-7 uses a data-dependent vector decay and a generalized Delta Rule with diagonal-plus-low-rank transitions and rank-one state injection.The low-rank modification changes the transition, while the rank-one term injects the current token’s state information.

3 Method: State Anomaly Neutralization

SANE diagnoses RWKV-7’s extreme-context failure as localized norm explosion atop a relatively sparse substrate and applies adaptive compression during inter-chunk state transfer. The method preserves low-magnitude state entries, suppresses anomalously large values, and retains chunkwise execution with bounded overhead.

  • Diagnosis: RWKV-7’s extreme-context instability is characterized by localized hotspots and norm growth atop a persistent low-magnitude background, rather than global state saturation.Most entries remain small while a few locations intensify by several orders of magnitude; the observed pattern is explained through decay and uneven recurrent updates, without constituting a general sparsity guarantee.
  • State Anomaly Neutralization: SANE applies adaptive element-wise compression to each chunk’s terminal state before it is transferred to the next chunk.The intervention targets inter-chunk state transfer, preventing extreme values from being repeatedly carried forward while leaving intra-chunk recurrence unchanged.
  • Compression behavior: Low-magnitude entries are approximately preserved, whereas entries large relative to their head-specific threshold are smoothly compressed toward ±τ.Independent thresholds vary across heads, accommodating spatially nonuniform state growth.
  • Adaptive thresholding: The threshold is generated from the current chunk’s last-token representation, with α controlling compression strength.Smaller α produces stronger compression, while larger α preserves a wider range of state magnitudes.
  • Cost and compatibility: SANE preserves the existing intra-chunk parallel structure and sequence-length-independent inference memory.In the RWKV-7-0.4B implementation, it adds fewer than 0.1% parameters; detailed end-to-end throughput and latency evaluation is left for future work.

4 Experiments

Experiments evaluate SANE on RWKV-7 for short-context mathematical reasoning and extreme-context extrapolation. SANE preserves short-context performance in a safe threshold regime and prevents the baseline’s instability over 100M-token streams, but excessive thresholds trade functional reasoning for numerical stability.

  • Experimental Setup: RWKV-7 serves as the experimental testbed, with mathematical reasoning used to assess stable capabilities after an ultra-long prefix rather than verbatim prefix memorization.The experiments use a pretrained 0.4B checkpoint under a constrained computational setup.
  • Short-Context Mathematical Reasoning: 43.30–44.23 average accuracy matches the 43.88 baseline without statistically significant differences for α ≥3 across 11 short-context benchmarks.All paired tests have p ≥0.154; α ≤2 instead significantly impairs short-context reasoning.
  • Short-Context Mathematical Reasoning: SANE’s post-hoc SFT setting introduces an adaptation cost, and the hypothesis that pretraining integration would absorb it remains experimentally unverified.After convergence, short-context performance is almost fully recovered relative to the baseline.
  • Extrapolation Reasoning: Functional Reasoning at 100M Tokens: 33.46 average accuracy at α=5 remains functional after 100M tokens, whereas accuracy declines monotonically as α increases across three threshold regimes.The safe regime is α ≤5, α=6–7 shows a steep decline, and α ≥8 scores 0.15 or below despite bounded perplexity.
  • Extrapolation Reasoning: Functional Reasoning at 100M Tokens: Numerical stability and functional health diverge: α ≥8 remains numerically bounded but loses symbolic reasoning capability, exposing a capacity–stability trade-off.The results attribute SANE’s effectiveness to selective compression that suppresses anomalous growth while preserving functional reasoning in the safe regime.

5 Analysis

SANE’s effectiveness is explained by the interaction between relative sparsity in the recurrent state and scale-selective compression. The threshold α creates a capacity–stability trade-off: moderate values preserve reasoning while suppressing anomalies, whereas extreme values damage either stability or capability.

  • 5.1 Sources of Relative Sparsity: Relative sparsity reflects global L2 shrinkage pressing most infrequently updated directions toward small magnitudes while low-rank updates sustain larger values in limited subspaces.This is presented as a mechanistic interpretation rather than a rigorous sparsity guarantee.
  • 5.2 Why SANE Works: SANE’s scale selectivity preserves small background values while compressing anomalously large state entries.Healthy background values remain below 0.1, whereas only a few directions accumulate substantially larger magnitudes.
  • 5.2 Why SANE Works: Chunk-boundary tanh compression interrupts cumulative amplification of anomalous values without changing the intra-chunk parallel computation structure.The intervention compresses values before they can be repeatedly amplified across chunks.
  • 5.3 The Capacity–Stability Dilemma: Within α ≤5, selective compression suppresses anomalous growth while preserving functional reasoning, but larger α values become insufficient to cover anomaly magnitudes.The threshold spectrum experimentally identifies scale selectivity as the core mechanism of effectiveness.
  • 5.3 The Capacity–Stability Dilemma: The safe range 3 ≤α ≤5 balances capacity and stability because legitimate large values overlap numerically with early anomalies.α=3 is reported as Pareto-optimal within the tested range.

6 Related Work

Related long-context methods largely target Transformer-specific positional encodings or attention mechanisms, while Delta-Rule models face a state-based bottleneck. Existing RWKV and linear-RNN approaches also alter the native architecture or depend on model-specific components, limiting direct transfer to Delta-Rule extrapolation.

  • Transformer Extrapolation: Transformer extrapolation methods primarily modify rotary positional embeddings or attention to address Transformer-specific long-context failures.These approaches include interpolation, extrapolation, and attention-window expansion strategies.
  • Delta-Rule Bottleneck: Delta-Rule models instead transfer cross-context information through a fixed-size state matrix, so their bottleneck originates in the state rather than positional encoding.Consequently, Transformer-focused methods cannot be directly transferred.
  • Related Linear-RNN Approaches: Existing improvements for RWKV and similar linear RNNs either require hybrid architectures pretrained from scratch or rely on selective-scan mechanisms specific to Mamba-like models.These approaches alter Delta-Rule’s native architecture or depend on components it does not use.

7 Conclusion

The paper identifies localized state norm explosion as the extreme-context failure mode of RWKV-7 and introduces SANE to compress anomalies at chunk boundaries. SANE preserves short-context reasoning within a safe threshold range and retains functional reasoning after a 100M-token prefix, while threshold choices expose a capacity–stability trade-off.

  • 7 Conclusion: RWKV-7 exhibits localized norm explosion atop a relatively sparse substrate rather than global state saturation.The proposed interpretation attributes this pattern to uneven injections and transition-induced amplification overcoming decay in a small number of directions.
  • 7 Conclusion: SANE applies adaptive tanh compression at chunk boundaries to preserve low-magnitude state values while suppressing extreme values.It is introduced into a pretrained RWKV-7 model through supervised fine-tuning.
  • 7 Conclusion: SANE causes no statistically significant short-context degradation within a safe threshold range and retains functional symbolic reasoning after a 100M-token prefix, unlike the overflowing baseline.The full threshold spectrum also shows that numerical stabilization alone does not guarantee functional health.

Limitations

The evaluation is limited to a 0.4B-parameter RWKV-7 model and a plug-and-play supervised-fine-tuning setup. Generalization to other Delta-Rule architectures, larger models, and SANE integrated during pretraining remains unverified.

  • Scope: Empirical validation is confined to RWKV-7 (0.4B), so effectiveness on other Delta-Rule architectures and larger scales remains to be verified.The paper specifically identifies 7B and 14B models as future evaluation targets.
  • Training Setup: SANE is added to an already pretrained checkpoint through supervised fine-tuning, incurring an adaptation cost and making current short-context results a conservative lower bound.Integrating SANE during pretraining is left for future work.

A Training and Evaluation Details

The study releases a RWKV-7 SANE kernel and evaluates models trained from a shared checkpoint using JAX-based training and multiple reasoning and long-context evaluation procedures.

  • Evaluation and release: Mathematical reasoning results are scored with the math_verify library, and the SANE kernel implementation is released online.
  • Evaluation: The 100M-token reasoning evaluation places the entire mixed_chat sequence before the mathematical test problems.The mixed_chat sequence is constructed by concatenating training splits from DeepMath-103K and Chinese-DeepSeek-R1-Distill-data-110k.
  • Training setup: Both baseline and SANE models start from the same pretrained RWKV-7-0.4B-G1D checkpoint and use KerasHub with the JAX backend.
  • Training setup: Training uses mixed bfloat16 precision, Muon optimization, cosine learning-rate scheduling, and a peak learning rate of 5e-5.The learning rate warms up over the first 3% of steps before cosine decay.
  • SANE implementation: SANE operates at chunk boundaries with chunk size 16 during both forward and backward passes.The chunk size matches the native RWKV-7 chunk size.
  • Evaluation: Short-context mathematical reasoning uses top-p=0.8, top-k=20, temperature=1.0, eight samples per problem, and majority voting as maj@8.
Loading 2608.22354v2…