PaperlayerView exact revision on arXiv

Source-linked AI summary

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, Christian Zhou-Zheng

arXiv:2503.14456v2cs.CLcs.AIcs.LG

TL;DR

Transformers face sequence-length-dependent inference costs, motivating recurrent models with constant memory and efficient per-token computation. RWKV-7 generalizes the delta rule with expressive, channelwise state updates and demonstrates broad theoretical and language-modeling results. The paper also releases a large multilingual corpus and models, while noting deployment-precision, compute, and checkpoint limitations.

  • Problem

    Transformers incur sequence-length-dependent inference costs, while recurrent alternatives seek constant memory and efficient per-token computation without sacrificing parallelizable training.

  • Method

    RWKV-7 generalizes the delta rule with vector-valued gating and learning rates, relaxed state replacement, and an RWKV-6-based architecture.

  • Results

    RWKV-7 performs state tracking, recognizes all regular languages with constant-depth recurrence, and achieves state-of-the-art performance for its size across reported benchmarks.

  • Takeaways & Limitations

    RWKV-7 offers an open-source recurrent alternative combining constant memory, linear time complexity, editable state evolution, and parallelizable training.

  • Takeaways & Limitations

    Training was limited to at most 96 Nvidia H800 GPUs and continued from earlier checkpoints, while some kernels are sensitive to numerical precision.

Abstract

from arXiv · show

We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves a new 3B SoTA on multilingual tasks and matches the current 3B SoTA on English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to $\mathsf{TC}^0$. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0 License.

M Initial Token Sensitivity

Table 1 compares recent RNN language-modeling architectures using four properties: large state, flexible decay, dynamic dependence, and generalized eigenvalues.

  • LS denotes matrix-valued states or state sizes at least four times larger than the model dimension.
  • FD denotes decay terms whose dimension is not smaller than the model dimension.
  • DD denotes decay terms that are functions of the input x_t.
  • GE denotes evolution matrices admitting eigenvalues outside the interval [0,1].

1 Introduction

The introduction frames Transformer inference costs as a motivation for recurrent alternatives and presents RWKV-7, its corpus, releases, and claimed theoretical and empirical contributions.

  • Softmax attention gives Transformers parallelizable training but incurs quadratic complexity and memory growth with sequence length.
  • Recurrent alternatives aim to provide linear computation, constant memory, and highly parallel training.
  • RWKV-7 generalizes the delta rule with vector-valued state gating, channelwise learning rates, decoupled keys, and a modified RWKV-6 design.
  • The work introduces the 3.1 trillion-token RWKV World v3 corpus and trains open-source models from 0.19 to 2.9 billion parameters.
  • The paper reports state-tracking and regular-language results beyond TC^0, alongside releases of models, dataset information, and code.

2 Background

The background contrasts efficient recurrent sequence models with their state-retention limitations and motivates extending the delta rule while preserving parallelizable training.

  • Linear attention offers constant per-token time and constant memory, unlike softmax attention’s sequence-length-dependent costs.
  • Fixed-size linear-attention states accumulate values over time, eventually mixing memories and degrading key-specific retrieval.
  • Per-step decay removes older values data-dependently, but cannot selectively remove values stored at specific keys.
  • DeltaNet partially replaces the value at the current key, enabling removal of old memories and addition of new ones on a per-key basis.
  • Recent work parallelizes DeltaNet across time, and RWKV-7 extends parallelization to its generalized delta rule.
  • RWKV-7’s stated theoretical result is recognition of all regular languages with a small constant number of layers, beyond prior negative-eigenvalue results.

3 Architecture

RWKV-7 replaces RWKV-6’s diagonal transition with a generalized delta-rule update that increases state expressivity while retaining efficient parallelization and modifies surrounding modules.

  • RWKV-7 uses a diagonal-plus-rank-one state update with vector-valued, input-dependent decay and small data-dependent vectors for efficient parallelization.
  • Its extended delta rule replaces data-dependent vector amounts of state, allowing key channels to vary independently rather than using a fixed scalar fraction.
  • RWKV-7’s input-dependent non-diagonal transition matrix supports functions beyond TC^0 and recognition of all regular languages under the paper’s stated conjecture.
  • The architecture replaces RWKV-6’s diagonal transition matrix and changes channel mixing and token shift to improve training and inference speed.
  • The pretrained large language models use only part of the possible negative-eigenvalue range because of observed training instabilities.

4 Method

RWKV-7 replaces RWKV-6’s diagonal transition with an extended delta rule featuring channelwise, data-dependent state evolution. The design combines dynamic updates, stable decay, and efficient parallelizable computation with revised mixing and MLP components.

  • Architecture changes: RWKV-7 removes data dependency from token-shift interpolation and channel-mixing receptance gating, and expands low-rank projections to balance parameters, training, and inference.The MLP becomes a two-layer network with hidden dimension four times the model dimension after removing its gating matrix.
  • State evolution: RWKV-7 generalizes the delta rule with vector-valued gating, vector-valued in-context learning rates, decoupled removal and replacement keys, and relaxed state replacement.These changes allow channelwise control over how state information is removed and added.
  • Efficient computation: The model retains efficient parallelization because its transition remains diagonal plus rank one, while small data-dependent vectors reduce non-SRAM memory bandwidth.The transition design also decouples decay, normalized keys, and the amount added to the state on a per-channel basis.
  • Time Mixing: The WKV state is a multi-headed matrix-valued fast-weight state that learns at test time to map keys to values.Receptance applies a query-like operation to this state, followed by normalization and an optional current-token bonus.
  • Time Mixing: RWKV-7 uses a data-dependent vector-valued decay whose transition matrix has eigenvalues in [−1,1], supporting dynamic evolution and forgetting across subspaces.The paper calls the decay in-context weight decay and the removal rate in-context learning rate.

5 RWKV World v3 Dataset

RWKV World v3 is a 3.119 trillion-token multilingual dataset assembled from diverse publicly available sources for training RWKV-7 models.

  • Dataset composition: 3.119 trillion tokens comprise RWKV World v3, a multilingual corpus drawn from a wide variety of publicly available data sources.Its composition follows earlier World datasets across English, multilingual data, and code, with slightly enhanced Chinese novels.

6 Pre-Trained Models

The authors publicly release RWKV-7 models trained on Pile and RWKV World v3, spanning 0.1B to 2.9B parameters under Apache 2.0 licensing. World models use converted earlier checkpoints and additional World v3 training.

  • Released models: Seven Apache 2.0-licensed RWKV-7 models are released, including four RWKV7-World-3 models sized 0.1B, 0.4B, 1.5B, and 2.9B.The remaining three RWKV7-Pile models are sized 0.1B, 0.4B, and 1.4B.
  • Pile models: RWKV7-Pile models are trained from scratch on the 332-billion-token Pile dataset using the GPT-NeoX-20B tokenizer.All three Pile models use this tokenizer and dataset.
  • World models: RWKV7-World-3 models use the RWKV World Tokenizer and are initialized from pre-existing RWKV-5 or RWKV-6 checkpoints before World v3 training.The 0.1B and 0.4B models start from RWKV-5 checkpoints, while the 1.5B and 2.9B models start from RWKV-6 checkpoints.
  • Training data: The World v1, v2, v2.1, and v3 corpora contain 0.6, 1.1, 1.4, and 3.1 trillion tokens, respectively.The paper reports total training amounts for RWKV-7 World 3 models in Table 2.

7 Language Modeling Experiments

RWKV-7 is evaluated across English, multilingual, synthetic, retrieval, and long-context tasks, with strong benchmark, recall, and state-tracking results. These experiments also assess performance on temporally novel data designed to avoid benchmark leakage.

  • LM Evaluation Harness Benchmarks: RWKV-7-World models outperform SmolLM2, Llama-3.2, and Qwen-2.5 by a significant margin on multilingual benchmarks.RWKV-7 generally matches Qwen2.5’s English performance while using less than one third as many training tokens.
  • LM Evaluation Harness Benchmarks: RWKV-7 shows similar English evaluation scores to highly trained open Transformer models with dramatically lower total training FLOPs.The multilingual evaluations show a dramatic Pareto improvement versus Transformer models.
  • Recent Internet Data Evaluation: RWKV-7 Goose maintains competitive performance on temporally novel internet data created after the models’ training periods.The evaluation uses post-training arXiv, GitHub, Wikipedia, fiction, and news data to remove data-leakage concerns.
  • Associative Recall: 72.93% recall is achieved with 256 key-value pairs using an 8192-dimensional WKV state.The reported estimate corresponds to 4480.8 stored information bits and an information density of 0.547 bits per dimension.
  • Associative Recall: RWKV-7 achieves the highest average score across all six MAD tasks, with perfect accuracy on In-Context and Noisy Recall.It matches DeltaNet on those perfect-accuracy tasks while setting a new state-of-the-art for Fuzzy Recall.
  • Long Context Experiments: RWKV7-World3-2.9B retrieves pass keys perfectly up to 35000 tokens, while fine-tuning extends reliable retrieval to 30k tokens with degradation around 50k tokens.The 1.5B model reaches perfect retrieval to 19600 tokens before degradation beyond 20600, and fine-tuning extends reliable retrieval to 29k tokens.
  • Evaluating State Tracking Using Group Multiplication: RWKV-7 exhibits stronger state-tracking capabilities than Transformers, Mamba, and S4 on group multiplication tasks.The results align with the theory that RWKV-7 can perform state tracking and recognize any regular language with a constant number of layers.

8 Speed and Memory Usage

RWKV-7’s recurrent kernels provide linear scaling with sequence length and constant-memory single-token inference, while supporting chunked pre-fill. On long sequences, the optimized kernel is faster than both RWKV-6 and Flash Attention v3.

  • Kernel Scope: The CUDA kernels are tuned for head dimension 64, with efficiency declining at larger head dimensions.A Triton-based Flash Linear Attention implementation is cited for configurations with head dimensions greater than 128.
  • Speed: RWKV models scale linearly with sequence length, whereas Flash Attention v3 scales quadratically and becomes slower for large sequences.The optimized RWKV-7 kernel is about three times faster than the official RWKV-6 kernel.
  • Speed: At sequence length 16k, RWKV-7 forward inference without state storage takes 7.9 ms versus 33.9 ms for Flash Attention v3.Forward inference with state storage takes 11.2 ms, while backward computation takes 22.5 ms.
  • Memory: RWKV-7 uses 18 variable equivalents during training at head size 64, compared with 10 for RWKV-6 and Flash Attention v3.The fp32 RWKV-7 kernel uses 24 variable equivalents because it stores states in float32.
  • Memory: Single-token inference uses constant memory, while chunked pre-fill allows a user-selectable trade-off between parallelization and maximum memory usage.Pre-fill memory grows linearly with chunk size and does not require processing the entire context at once.

9 Multimodal Experiments

RWKV-7 is extended to visual and audio modeling through multimodal encoders and recurrent adaptations. VisualRWKV-7 improves substantially over VisualRWKV-6 on several benchmarks, while AudioRWKV-7 remains competitive with larger CNN, Transformer, and Mamba models.

  • Image Understanding: VisualRWKV-7 combines SigLIP, DINO, and high-resolution SAM image encoders before feeding projected image features with text embeddings into RWKV-7.The architecture concatenates encoder features, applies context-gated MLP projection, and integrates them with text inputs.
  • Image Understanding: VisualRWKV-7 0.1B and 0.4B outperform VisualRWKV-6 1.6B on the in-domain VQAv2 and GQA benchmarks.The 0.4B model uses one quarter the parameters of the 1.6B comparison model.
  • Image Understanding: VisualRWKV-7 2.9B outperforms VisualRWKV-6 3.1B on the out-of-domain SQA benchmark.On TextQA, VisualRWKV-7 2.9B improves by 5.3 points over VisualRWKV-6 3.1B.
  • Audio Modeling: AudioRWKV-7 uses a bidirectional RWKV-7 modification to process high-dimensional spectrogram features and acoustic-temporal information.The approach divides mel-spectrograms into patch tokens for audio embedding analysis.
  • Audio Modeling: AudioRWKV-7 achieves comparable performance with a much smaller parameter count than CNN, Transformer, and Mamba architectures, and exceeds AudioRWKV-6.The comparison is conducted on AudioSet using mean Average Precision.

10 Conclusions

RWKV-7 is presented as an efficient recurrent architecture with strong benchmark performance and expanded expressivity, while the authors identify deployment, training, and usability limitations. The paper also argues that RWKV-7 can solve state-tracking problems beyond Transformers and recognize all regular languages.

  • Conclusions: RWKV-7 offers state-of-the-art performance for its size while maintaining linear time complexity and constant memory usage.The authors position it as an alternative to Transformer-based architectures despite training on many fewer tokens.
  • Limitations: The models remain limited by numerical-precision sensitivity, absent instruction tuning and alignment, prompt sensitivity, and constrained compute resources.The authors specifically report sensitivity in the WKV7 kernel, degraded performance without <|endoftext|>, and reuse of data from earlier checkpoints.
  • Expressivity: RWKV-7 can express NC1-complete state-tracking problems that Transformers, S4, and Mamba cannot under standard complexity conjectures.The paper also proves that RWKV-7 can recognize any regular language.
  • Expressivity: A four-layer RWKV-7 model can recognize any regular language.The construction uses elementary transition matrices and lookup-table mechanisms implemented across layers.
  • Limitations: The regular-language construction may require MLP layers exponentially wide in the number of states of the original DFA.This requirement arises because the construction uses lookup tables with sizes on the order of |Σ|2n.

D.3 Lemmas for Theorem 3

The lemmas construct RWKV-7 mechanisms for representing elementary transitions, tracking position information, storing recent tokens, and implementing lookup tables. Together, these components support the regular-language construction while exposing its width and implementation requirements.

  • Transition decomposition: A DFA transition matrix can be factored into n elementary transition matrices, each representing identity, swap, or copy operations.The construction greedily builds the target matrix from the identity using at most n non-identity transitions, then pads with identity transitions.
  • Construction assumptions: The theoretical construction uses c = 2 in some steps, while the actual model uses c = 1.The text explains that halving c and w_t halves the state magnitude, which group normalization makes irrelevant to subsequent calculations.
  • Transition decomposition: For any elementary transition matrix, RWKV-7 provides vectors and gates that implement the corresponding state update.The lemma specifies a unit-norm key vector and a binary gating vector, with explicit constructions for identity, swap, and copy matrices.
  • Position tracking: A one-layer RWKV-7 can output whether the current position is first and whether it is even or odd.Token shift detects whether a previous token exists, while the WKV state encodes position parity.
  • Position tracking: A two-layer RWKV-7 can output the current position modulo 2n.Multiple WKV heads read sign patterns after group normalization, and a subsequent MLP combines them with parity information.
  • Lookup construction: A state update can replace one indexed column with a token one-hot vector, thereby storing the last 2n tokens.The state and position modulo 2n are then supplied to an MLP that performs a lookup into Ξ.

F Additional Architecture Discussion

RWKV-7 extends delta-rule state evolution with vector-valued, data-dependent mechanisms while retaining efficient training and constant-memory recurrent processing. Its editable state supports computations beyond immutable Transformer caches, alongside numerical-stability and test-time scaling findings.

  • State evolution: RWKV-7 applies delta-rule time mixing by decaying the state, removing old values at the current key, adding new values, and then querying with receptance.The update is described as an SGD-like mechanism that trains the state at test time to associate keys with matching values.
  • State evolution: Vector-valued decay improves loss per step, although it is harder to compute efficiently than scalar decay.The decay also substitutes for positional encoding in RWKV models.
  • Numerical stability: The decay multiplier is constrained above 0.545, so decay removes at most 45.5% of pre-existing state values per timestep before delta-rule removal.The stated bound is intended to support training stability and numerically stable kernels.
  • State evolution: RWKV-7 uses data-dependent vector-valued in-context learning rates, giving each key channel an independent learning rate.The rate is less restrictive than traditional delta-rule or SGD learning rates.
  • Design rationale: RWKV-7 relaxes normalization constraints by letting the replacement key choose its learning rate through a replacement rate booster.The authors report that this latitude performed best and was most efficient experimentally.
  • Expressivity: RWKV-7 edits its state at later tokens, enabling operations such as swapping state entries that immutable Transformer key-value caches cannot perform on fixed inputs.The paper characterizes the editable state as an internal scratchpad and connects this capability to computations beyond TC^0.
  • Numerical stability: Per-head normalization is applied after receptance to prevent changes in state magnitude from affecting state use, while avoiding the cost of denominator-based normalization.The original RWKV-4 formulation used a denominator instead.
  • Evaluation: Increasing Alpha-beta pruning width and depth increases the testing budget, and RWKV-7 shows positive test-time scaling on the evaluated task.The comparison uses a baseline model with depth=1 and width=1.

K Ablation Experiments

The ablations compare RWKV-7 with earlier RWKV models and isolate architectural choices involving decay, learning rates, key separation, and output computation. Additional evaluation shows substantial sensitivity to the initial end-of-text token, especially in smaller models.

  • Pile ablations: The ablation study trains 168M, 421M, and 1.47B parameter models on the full Pile dataset.The setup is designed to evaluate architectural differences across three model sizes.
  • Pile ablations: RWKV-7 consistently improves over earlier RWKV models trained on the same dataset and configurations, with the gap sustained as model size increases.The results suggest that RWKV-7 may scale more effectively than its predecessors.
  • Architecture choice ablations: The architecture ablations test vector-valued decay, vector-valued in-context learning rates, separate removal and replacement keys, and an added output bonus term.These changes are compared with more restrictive choices in DeltaNet and related post-DeltaNet work.
  • Architecture choice ablations: The appendix trains a 6-layer, dmodel = 768 Goose model on the 1.6B-token minipile dataset at context length 512 for loss comparisons.The corresponding results are reported in Table 19.
  • Parameter analysis: The appendix notes that Section 4 does not specify the actual ranges and statistical metrics of parameters within the trained model.It provides empirical statistics for selected parameters from a released RWKV-7 model.
  • Initial token sensitivity: LAMBADA performance changes substantially when the initial <|endoftext|> token is omitted, producing statistically significant PPL and ACC differences for some RWKV-7 models.The analysis identifies 142 affected examples among 5153 questions and links the pattern to answers beginning the paragraph.
  • Initial token sensitivity: Including <|endoftext|> significantly improves RWKV-7 performance, especially for small models, and two consecutive tokens can improve it further.The paper interprets this finding as highlighting the importance of state initialization and context setting for RNN-based architectures.
Loading 2503.14456v2…