RWKV-7 "Goose" with Expressive Dynamic State Evolution

M Initial Token Sensitivity

• Table 1 defines four comparison criteria for recent language-modeling RNN architectures: Large State, Flexible Decay, Dynamic Dependence, and Generalized Eigenvalue.LS denotes matrix-valued states or state size at least four times the model dimension; FD requires decay dimension at least the model dimension; DD makes decay input-dependent; GE permits eigenvalues outside [0,1].Source: Table 1 · p. 3
• The table notes that one result is shown with mini-batch size 1 for simplicity.Source: Table 1 · p. 3

Architecture

• RWKV-7 “Goose” generalizes the delta rule through vector-valued state gating, vector-valued in-context learning rates, decoupled removal and addition keys, and modified RWKV-6 components.These changes enhance expressivity, provide implicit positional encoding, and enable channelwise state replacement while retaining token-shift, bonus, and ReLU2 feedforward features.Source: §1 · p. 3

Data

• The RWKV World v3 corpus contains 3.1 trillion publicly available multilingual tokens designed to improve English, code, and multilingual performance.The corpus supports training the new RWKV-7 World v3 models.Source: §1 · pp. 3–4

Model upgrading

• RWKV-7 also supports upgrading existing RWKV architectures without pre-training from scratch, reducing computational expense while producing increasingly competitive models.This provides a practical path for improving preexisting RWKV-5/RWKV-6 checkpoints.Source: §1 · pp. 3–4

Empirical results

• RWKV-7 improves downstream performance over RWKV-6, reaches state-of-the-art multilingual performance at 3B scale, and approaches English-language SoTA despite substantially fewer training tokens.The introduction frames this result as the architecture’s primary empirical contribution.Source: §1 · p. 4

Open models

• The authors release four RWKV-7 World v3 models spanning 0.19–2.9 billion parameters and three RWKV-7 Pile models spanning 0.17–1.47 billion parameters.The World v3 models use 1.6–5.6 trillion training tokens, while the Pile models use the GPT-NeoX tokenizer for comparative study.Source: §1 · p. 4

Theory

• The generalized delta rule can solve problems outside TC0 under the conjecture TC0 ≠ NC1, including single-layer S5 state tracking and recognition of all regular languages with constant depth.These results establish theoretical capabilities beyond the stated complexity limit for Transformers.Source: §1 · p. 4

• Linear attention provides constant time and memory per token, unlike softmax attention’s O(N) time and memory with respect to sequence length.Source: §2 · p. 4
• Fixed-size linear-attention states continually accumulate values, eventually mixing memories and muddying outputs; decay removes older information globally but cannot target specific keys (Schlag et al., 2021).Source: §2 · p. 4

Delta Rule

• DeltaNet applies the error-correcting delta rule to key-value states, partially replacing the value at the current key so the model can remove old memories and add new ones per key (Schlag et al., 2021).The update is formulated as online learning that trains the recurrent state at test time to retrieve desired values for input keys.Source: §2 · p. 4

• DeltaNet’s diagonal-plus-low-rank state evolution can be parallelized across time, and this work extends parallel training to its generalized delta rule and RWKV-7.Source: §2 · p. 4

Concurrent Work

• Concurrent methods combine delta-style updates with closed-form optimization, scalar gating or decay, batched test-time training, and momentum.Source: §2 · p. 5
• RWKV-7 and its generalized delta rule recognize all regular languages with a small constant number of layers, exceeding the expressiveness result obtained using negative eigenvalues.Source: §2 · p. 5

Extended delta rule

• The extended delta rule uses a data-dependent vector-valued decay and rank-one update, retaining efficient parallelization and reduced non-SRAM memory bandwidth.Its vector-valued keys permit distinct removal and replacement keys while avoiding large matrices.Source: §3 · p. 5
• Vector-valued state replacement lets each key channel vary independently, with z_t = −κ̂_t and b_t = κ̂_t ⊙ a_t, where a_t has elements in (0,1) for stable updates.The design choices’ performance improvements are evaluated through ablations in Appendix K.Source: §3 · p. 5

Expressivity

• RWKV-7’s input-dependent non-diagonal transition can surpass TC^0 under standard complexity conjectures and recognize all regular languages with a constant number of layers.The copy state transition is central to this result.Source: §3 · p. 5

Architecture changes

• RWKV-7 replaces RWKV-6’s diagonal transition and updates channel mixing and token shift, removing selected data dependencies and increasing low-rank projections for speed, parameter, and performance tradeoffs.The token-shift and channel-mixing changes contribute to faster training and inference.Source: §3 · p. 5

Limitations

• Pretrained large language models currently allow only part of the possible negative-eigenvalue range because experimentally observed training instabilities constrain the design.This is a stated limitation of the current implementation.Source: §3 · p. 5

4.1 Time Mixing

• Weight preparation uses low-rank MLPs for data dependency, then splits recurrent quantities into equally sized heads whose WKV state evolves dynamically and can be computed in parallel.Receptance is applied to the WKV state, while a weighted bonus can attend directly to the current shifted input without storing it in the state.Source: §4.1 · pp. 7–8
• RWKV-7 decouples in-context weight decay, learning rate, and replacement boosting per channel, stabilizing state magnitudes while increasing expressivity.The replacement-rate booster interpolates additions between the normal in-context learning rate and 1.0; RWKV-7 state entries generally remain numerically stable.Source: §4.1 · p. 7
• Its transition matrix combines dynamic state evolution with forget-gate-like decay, while vector-valued removal keys allow different removal amounts across state columns.The transition has eigenvalues in [-1,1] and can decay information in all subspaces when necessary.Source: §4.1 · p. 8

4.2 MLP

• The MLP removes the previous gating matrix and uses a two-layer MLP with hidden dimension four times the model dimension to preserve parameter count.This differs from the Channel Mixing modules used in RWKV-4, RWKV-5, and RWKV-6.Source: §4.2 · p. 8

Dataset scale and sources

• 3.119 trillion tokens: RWKV World v3 is a multilingual corpus drawn from a wide variety of publicly available data sources.The dataset is intended to help close the gap with modern LLM training, which may use 15–18 trillion tokens.Source: §5 · p. 9

Dataset composition

• The dataset approximates the distribution of previous World datasets across English, multilingual, and code data.Its composition also slightly enhances Chinese novels.Source: §5 · p. 9

Motivation

• The corpus is designed to narrow the training-data gap with modern large language models.The paper notes that modern LLMs may consume as many as 15–18 trillion tokens.Source: §5 · p. 9

• Seven Apache 2.0-licensed models comprise three RWKV7-Pile models at 0.1B, 0.4B, and 1.4B parameters and four RWKV7-World-3 models at 0.1B, 0.4B, 1.5B, and 2.9B.Source: §6 · p. 9
• Pile models use the GPT-NeoX-20B tokenizer and were trained from scratch on the 332-billion-token Pile dataset.Source: §6 · p. 9
• World models use the RWKV World Tokenizer and begin from converted RWKV-5 or RWKV-6 checkpoints before training on the full 3.1-trillion-token World V3 corpus or an equally weighted subsample.Under this methodology, some documents were seen two or three times.Source: §6 · p. 9
• Checkpoint conversion removes token-shift low-rank MLPs, rescales selected weights and biases, clamps and square-roots normalization weights, widens FFN and time-decay MLPs, and replaces gate weights with an SVD-derived LoRA.The newly initialized regions use uniform scales of 1×10^-3 for the FFN and 1×10^-4 for the time-decay MLP.Source: §6 · p. 9

7.1 LM Evaluation Harness Benchmarks

• RWKV-7 matches Qwen2.5’s English performance with fewer than one third as many training tokens, while RWKV-7-World significantly outperforms SmolLM2, Llama-3.2, and Qwen-2.5 on multilingual benchmarks.The multilingual evaluations show a dramatic Pareto improvement over transformer models, with similar English scores at substantially lower training FLOPs.Source: §7.1 · p. 10, Figure 3 · p. 10, Figure 4 · p. 11

7.2 Recent Internet Data Evaluation

• Despite training on significantly less data than other top models, RWKV-7 Goose achieves competitive compression performance on temporally novel internet data created after January 2025.The evaluation uses post-training arXiv papers, GitHub repositories, Wikipedia entries, fiction, and news, with compression rate as the metric to avoid benchmark leakage.Source: §7.2 · p. 10, Table 5 · p. 13

7.3 Associative Recall

• With a WKV size of 8192, RWKV-7 recalls 72.93% of 256 key-value pairs, storing an estimated 4,480.8 bits at 0.547 bits per dimension.The model was trained on MQAR sequences up to length 2048 using RWKV-7-specific initialization and stabilization settings.Source: §7.3 · pp. 11–12, Table 6 · p. 13

7.4 Mechanistic Architecture Design

• RWKV-7 achieves the highest average score across all six MAD tasks, with perfect accuracy on In-Context and Noisy Recall and a new state-of-the-art on Fuzzy Recall.Its strong memorization and selective-copying performance suggests an effective combination of attention-based and recurrent strengths.Source: §7.4 · p. 12, Table 7 · p. 14

7.5 Long Context Experiments

• The 2.9B model retrieves pass keys perfectly up to 35,000 tokens, while the 1.5B model reaches 19,600 tokens; long-context fine-tuning extends reliable retrieval to 30,000 tokens and degradation to around 50,000 tokens for the 2.9B model.The World-dataset model’s loss increases beyond 10k-token contexts, whereas fine-tuning on long contexts restores long-context capabilities.Source: §7.5 · pp. 12–13

7.6 Evaluating State Tracking Using Group Multiplication

• RWKV-7 exhibits stronger state-tracking capabilities than Transformers, Mamba, and S4, though it is slightly weaker than classical RNNs.The results align with the prediction that RWKV-7 can perform state tracking and recognize any regular language with a constant number of layers, while retaining efficient parallelization unlike classical RNNs.Source: §7.6 · p. 15

Kernel scope

• The kernels are tuned for head dimension 64, remain effective at 128, and lose efficiency at larger dimensions, where Flash Linear Attention provides an alternative implementation.Flash Linear Attention is a Triton-based implementation designed for configurations with head dimensions greater than 128.Source: §8 · p. 15

Speed

• At large sequence lengths, Flash Attention v3 scales quadratically while RWKV models scale linearly, making RWKV faster; optimized RWKV-7 is about three times faster than official RWKV-6.Source: §8 · p. 16
• At sequence length 16k, RWKV-7 takes 7.9 ms for inference without state storage, versus 33.9 ms for Flash Attention v3.Storing state raises the RWKV-7 forward time to 11.2 ms, while the backward pass takes 22.5 ms.Source: §8 · p. 16

Memory

• Single-token inference uses constant memory, while chunked pre-fill grows linearly with chunk size and allows users to select a maximum memory budget.Inference memory excludes gradients and state storage; chunking provides a trade-off between faster parallelized pre-fill and memory usage.Source: §8 · p. 18

RWKV for Image Understanding

• VisualRWKV-7 combines SigLIP, DINO, and high-resolution SAM vision encoders, projecting concatenated image features into the RWKV-7 block before combining them with text embeddings.The SAM encoder increases supported resolution to 1024 x 1024, and the architecture uses context-gated MLP projection.Source: §9 · p. 18, Figure 10 · p. 18
• With one-quarter the parameters, VisualRWKV-7 0.4B surpasses VisualRWKV-6 1.6B on VQAv2 and GQA, while VisualRWKV-7 0.1B and 0.4B rapidly approach it on two other benchmarks.The comparison covers GQA, SQA, TQA, and VQA-related evaluation in Table 9.Source: §9 · pp. 18–19, Table 9 · p. 18
• VisualRWKV-7 2.9B outperforms VisualRWKV-6 3.1B on SQA and improves by 5.3 points on TQA, indicating stronger out-of-domain generalization and associative recall.SQA is described as an out-of-domain benchmark, while TQA assesses associative recall.Source: Table 9 · p. 18, §9 · p. 19

RWKV for Audio Modeling

• Across AudioSet evaluations, AudioRWKV-7 achieves comparable performance with far fewer parameters than CNN-, Transformer-, and Mamba-based architectures and exceeds AudioRWKV-6.AudioRWKV-7 uses bidirectional RWKV-7 processing with patch-tokenized mel-spectrograms ordered time → frequency → window; AudioRWKV-6 was retrained without ensembling for fair comparison.Source: §9 · p. 19, Table 10 · p. 19

10 Conclusions

• RWKV-7 achieves state-of-the-art performance for its size across broad benchmarks while retaining linear time complexity, constant memory usage, and high parameter efficiency.The conclusion says it can rival highly optimized models such as Qwen2.5 despite training on many fewer tokens.Source: §10 · p. 19

10.1 Limitations

• The models remain limited by numerical-precision sensitivity, missing instruction tuning and alignment, prompt sensitivity without <|endoftext|>, and constrained compute and reused data.The reported training budget was at most 12×8 = 96 Nvidia H800 GPUs, and scaling to larger models and datasets requires additional resources.Source: §10.1 · p. 20

10.2 Future Work

• Future work targets larger models and datasets, kernel and distributed-training speedups, and Chain-of-Thought reasoning for multi-step problem solving.Proposed speed techniques include Dual Pipelining Mechanism, Mixture-of-Experts, Multi-Token Prediction, and FP8 Training, while reasoning work awaits suitable reinforcement-learning pipelines.Source: §10.2 · p. 20

B Training Dataset Details

• The RWKV World v3 corpus totals approximately 3.1 trillion tokens, extending World v2.1’s approximately 1.4 trillion tokens with additional entries and generally equal token weighting.Most component sources are used intact, with limited subsampling for over-represented languages in parts of the earlier corpus.Source: B Training Dataset Details · pp. 31–32

C Transition Matrix Eigenvalues and Stability

• RWKV-7’s transition matrix has eigenvalues in (−1,1) and yields a stable update when the parameter a_t is time-independent.The proof establishes similarity to a symmetric matrix, at most one negative eigenvalue, and contraction; the stability proof does not cover time-varying a_t.Source: C Transition Matrix Eigenvalues and… · pp. 32–33

D.1 Warmup: Expressivity Beyond TC0

• RWKV-7 can express NC1-complete state-tracking problems, including tracking swaps on five elements, beyond the TC0 class under standard complexity conjectures.The swap-tracking construction uses a one-layer model whose heads update and jointly check the permutation state.Source: D Expressivity of RWKV-7 · p. 33, §D.1 · p. 34

D.2 Main Result: RWKV-7 Can Recognize Any Regular Language

• A 4-layer RWKV-7 model can recognize any regular language by simulating an arbitrary deterministic finite automaton.The construction factors DFA transitions into elementary matrices and uses three layers for conversion plus a final layer for multiplication.Source: §D.2 · p. 35
• Weight preparation can independently select six WKV variables as arbitrary linear transforms of prior-layer outputs, making the WKV heads the construction’s main component.Each time-mixing block first applies LayerNorm; constant-norm one-hot encodings preserve their encoded information under this operation.Source: §D.2 · p. 35
• For each block, the construction factors the product of DFA transitions into elementary matrices and applies the appropriate factor at each position using a fourth-layer WKV state.The required elementary matrix is computed in the third layer from the last 2n tokens and the position modulo 2n.Source: §D.2 · p. 35
• Duplicating the construction across n WKV heads, each with a distinct receptance, reads out the full encoded DFA state vector.The first block is initialized separately and later transitions implement identity matrices.Source: §D.2 · p. 36
• At the final token, a third-layer lookup supplies the required position- and suffix-dependent multiplier, allowing the final MLP to output the DFA evaluation scalar product.The multiplier depends only on the current position modulo 2n and the last 2n tokens.Source: §D.2 · p. 36
• The construction’s MLP lookup tables scale as |Σ|2n and may require layers exponentially wide in the original DFA’s number of states.This width requirement is the stated limitation of the construction.Source: §D.2 · p. 36

D.3 Lemmas for Theorem 3

• DFA transition matrices factor into at most n elementary identity, swap, and copy matrices, with each greedy multiplication reducing differing columns.This decomposition enables direct simulation of arbitrary DFA transitions from elementary RWKV-7-compatible operations.Source: §D.3 · pp. 36–37
• Lemma 4 provides normalized vectors and binary gates that realize each elementary transition matrix, while rescaling from c = 2 to the actual c = 1 preserves computations after normalization.The construction uses distinct settings for identity, swap, and copy matrices; halving c and w only rescales the state, and the constructions avoid mismatched nonzero updates.Source: §D.3 · pp. 37–38
• A 1-layer RWKV-7 detects the first position and parity, while a 2-layer model outputs position modulo 2n using parity and multiple sign readouts.The modulo construction uses 2n wkv heads because group normalization discards magnitude information.Source: §D.3 · pp. 37–38
• A 3-layer RWKV-7 simulates any lookup table indexed by position modulo 2n and the 2n most recent tokens by storing those tokens in the wkv state.The update replaces the position-indexed state column with the current token, and subsequent heads read the stored state for the lookup MLP.Source: §D.3 · p. 38

E Additional Architectural and Training Details

• RWKV-7 uses four low-rank MLPs for decay w, value residual v, in-context learning rate a, and gate g, with head size Dh = 64.The model uses h = D/64 heads, each with 64×64 time-mixing dimensions; suggested intermediate sizes appear in Table 16.Source: E Additional Architectural and… · pp. 38, 40, Table 16 · p. 40
• The 3,119,194,079,123-token pretraining dataset is memory-mapped and sampled through a bijective cubic map over a finite field, ensuring each index is accessed once per epoch.For sequence length 4096, the largest suitable prime is p = 761521949, with a set near 0.618p to promote mixing.Source: E Additional Architectural and… · p. 40
• Training used bfloat16 on nodes of 8× Nvidia H800, with AdamW, cosine learning-rate decay, and phased dynamic batch-size scaling that progressively increased batch size during training.AdamW used β1 = 0.9, β2 = 0.99, ϵ = 1×10−18, and weight decay 0.1 exclusively on linear and embedding weights; the final learning rate was 1×10−5.Source: E Additional Architectural and… · pp. 40–41
• All four runs showed extremely stable training without loss spikes, but occasional single-step NaNs required rewinding to the prior checkpoint and clearing optimizer states.The authors theorize that the NaNs may result from the extremely low AdamW ϵ.Source: E Additional Architectural and… · p. 41

F Additional Architecture Discussion

• RWKV-7 uses simple learned per-channel token shift, vector-valued decay, data-dependent vector-valued in-context learning rates, and a replacement-rate booster rather than enforced state normalization.Its decay is constrained to remain greater than 0.545, so decay alone removes at most 45.5% of pre-existing state values per timestep; the delta rule can remove more.Source: F Additional Architecture Discussion · pp. 41–43
• RWKV-7 updates its WKV state by decaying it, removing part of the old value at the current key, adding a new value, and querying it with receptance.The state is then normalized per head, gated, and recombined into the output.Source: F Additional Architecture Discussion · pp. 42, 44
• RWKV-7’s editable state can perform computations such as replacing or swapping entries, unlike an immutable transformer KV cache, enabling fixed-input tasks that transformers cannot solve without extra inference-time computation.The state is described as an internal scratchpad, while both architectures become Turing complete with costly additional outputs such as chain of thought.Source: F Additional Architecture Discussion · p. 43

G Pseudocode For RWKV-7

• The RWKV-7 pseudocode implements token shifting, decay, replacement-key construction, recurrent WKV state transitions, readout, and propagation of time-mixing and channel-mixing states through layers.The supplied pseudocode includes the recurrent timestep loop and returns the output together with persistent model states.Source: G Pseudocode For RWKV-7 · pp. 44–45

H PyTorch code For Naive WKV7 Kernel (Forward and Backward)

• The naive PyTorch WKV7 kernel supplies both forward and backward passes, iterating through sequence states and returning gradients for receptance, decay, keys, values, learning-rate parameters, and state.The forward pass combines decay, removal, and replacement updates through tensor einsums; the backward pass traverses timesteps in reverse.Source: H PyTorch code For Naive WKV7 Kernel… · pp. 46–47

I Board Game Modeling

• For Othello, the authors developed RWKV-7a and training samples combining board state and search parameters, legal-move evaluation, Alpha-Beta reasoning, and the final move with the resulting board.The expanded RWKV-7a formulation permits the full range of (-1,1) eigenvalues when c = 2.Source: I Board Game Modeling · p. 47
• During Reversi training, RWKV-7 first learned output formatting, then board-state tracking, while evaluation accuracy continued improving.This pattern was observed by tracking losses across token types in the 9M- and 26M-parameter models trained on 6 million samples.Source: I Board Game Modeling · p. 49, Figure 13 · p. 49
• Increasing Alpha-beta search width and depth improved RWKV-7’s Reversi win rate against the depth=1, width=1 baseline, demonstrating positive test-time scaling.The model’s thinking budget was controlled through search width and depth.Source: I Board Game Modeling · p. 49, Figure 14 · p. 49

J State Inspections

• RWKV-7’s WKV entries stayed O(1) without context-length growth, whereas RWKV-5 and RWKV-6 produced outliers of thousands, indicating better numerical stability.The comparison used 1.5B-parameter RWKV-5, RWKV-6, and RWKV-7 models on PG19 samples of at least 8192 tokens.Source: J State Inspections · pp. 50–51
• For contexts longer than 32, RWKV-7 had lower WKV stable rank than RWKV-5 and RWKV-6, which the authors hypothesize reflects stronger compression and utilization of important information.The authors propose that enhanced state evolution reconciles lower stable rank with better long-term-memory performance.Source: J State Inspections · p. 51

K Ablation Experiments

• RWKV-7 consistently outperforms earlier RWKV models trained on the same dataset and configuration, with the gap persisting as model size increases.The ablations used 168M, 421M, and 1.47B-parameter models trained on the full Pile dataset.Source: K Ablation Experiments · pp. 51–52
• The architecture ablations test vector-valued decay and learning rates, separate removal and replacement keys, and an added output bonus term.These changes target restrictive scalar-gating and shared-key choices in DeltaNet and post-DeltaNet designs.Source: K Ablation Experiments · p. 52
• A controlled Goose ablation trains a six-layer, 768-dimensional model on 1.6B minipile tokens at context length 512 and reports losses in Table 19.Table 19 is labeled as ablation results for this model configuration.Source: K Ablation Experiments · p. 52, Table 19 · p. 52

L Parameters Statistics

• The parameter-statistics appendix supplies empirical statistics for selected parameters in released RWKV-7 models, addressing unspecified parameter ranges and metrics from Section 4.The accompanying plots cover ξ and α distributions and layerwise ξ extrema or means across models.Source: L Parameters Statistics · p. 52, Figure 17 · p. 52, Figure 18 · p. 53

M Initial Token Sensitivity

• Omitting <|endoftext|> at the beginning of LAMBADA inputs causes substantial, statistically significant changes in perplexity and accuracy for some RWKV-7 models.The effect remained after precision issues were ruled out and was linked to input consistency.Source: M Initial Token Sensitivity · p. 53
• Answers that appear as the paragraph’s first word and do not recur elsewhere in the text may indicate difficulty retaining the first token in memory.The LAMBADA test set contained 142 such examples out of 5,153 questions.Source: M Initial Token Sensitivity · p. 54
• Including <|endoftext|> substantially improves RWKV-7 performance, especially for small models, while two initial consecutive tokens can improve it further despite never occurring consecutively in training.The comparison is summarized for a partial set of 142 LAMBADA samples in Table 20.Source: M Initial Token Sensitivity · p. 55, Table 20 · p. 56
• Qwen2.5-0.5B shows a less pronounced response to <|endoftext|>, and the authors recommend always prepending the token when prompting RWKV-7.They give a structured chat format beginning with <|endoftext|> followed by the user and assistant turns.Source: M Initial Token Sensitivity · pp. 55–56