Source-linked AI summary
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland, Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu
TL;DR
Transformers’ quadratic attention motivates efficient recurrent alternatives. This paper introduces Eagle and Finch, which add expressive matrix-valued states and dynamic recurrence to RWKV, and reports competitive performance across diverse benchmarks while releasing multilingual resources and open models. The models’ embedding performance remains weak without an appropriate aggregation method.
Problem
Transformers provide strong sequence modeling but their attention has quadratic sequence-length complexity, motivating efficient alternatives.
Method
The paper develops Eagle and Finch, extending RWKV with multi-headed matrix-valued states, dynamic data-dependent recurrence, a multilingual tokenizer, and a 1.12 trillion-token dataset.
Results
Eagle and Finch perform competitively or improve upon existing models across English, multilingual, associative-recall, music, and vision-language benchmarks.
Takeaways & Limitations
The released architectures and multilingual resources provide publicly accessible alternatives to Transformer-based language models while retaining recurrent inference efficiency.
Takeaways & Limitations
Eagle did not achieve strong MTEB embedding performance, and the authors leave suitable information aggregation for future work.
Abstract
from arXiv · showhide
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks. We release all our models on HuggingFace under the Apache 2.0 license. Models at: https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code at: https://github.com/RWKV/RWKV-infctx-trainer
1 Introduction
Transformers offer strong sequence modeling but incur quadratic sequence-length complexity, motivating recurrent alternatives that retain efficient inference. Eagle and Finch extend RWKV with more expressive recurrence, multilingual resources, and broad evaluations showing competitive performance.
- Transformers’ attention has quadratic time complexity with respect to input sequence length, motivating sub-quadratic alternatives.
- Eagle and Finch improve RWKV-4 using matrix-valued states, reformulated receptance, additional gating, and data-dependent recurrence mechanisms.
- 1.12 trillion tokens comprise RWKV World v2, a multilingual and code-focused dataset designed to address English-heavy training data.
- The models are evaluated across English and multilingual benchmarks, associative recall, music modeling, and vision-language tasks.
- Four Eagle models span 0.46 to 7.5 billion parameters, while two Finch models contain 1.6 and 3.1 billion parameters.
2 Background
RNNs provide constant-time-per-token inference but are difficult to parallelize for training, whereas Transformers parallelize well but have quadratic attention costs. Linear attention and RWKV seek to combine recurrent inference efficiency with parallelizable training, while addressing performance limitations of naive linear attention.
- RNNs update a hidden state recurrently, enabling O(1) time per token but making time-dimension training parallelization difficult.
- Transformers achieve strong sequence-modeling performance, but multi-headed self-attention has quadratic complexity in sequence length.
- Linear attention reorders feature-mapped operations so it can run recurrently in O(1) time per step or train in parallel.
- Naive linear attention meets the efficiency goals but suffers significantly reduced performance compared with multi-headed-attention Transformers.
- RWKV-4 combines token shifting, gating, learned decay, and current-token weighting, enabling fast parallelizable training and O(1) time complexity per token.
3 Eagle/Finch Architecture
Eagle and Finch retain RWKV-4’s residual-block structure while increasing expressivity through matrix-valued states and progressively more data-dependent recurrence. Figure 1 contrasts their token-shift and connectivity choices.
- Eagle adds matrix-valued attention states, head-wise LayerNorm, SiLU attention gating, improved initialization, and removes receptance’s Sigmoid activation.
- Finch extends Eagle by making the decay schedule and token shift data-dependent.
- Each architecture stacks residual blocks containing Pre-LayerNorm Time-Mixing followed by Pre-LayerNorm Channel-Mixing sub-layers.
- Figure 1 shows token-shift modules in the FeedForward block and Eagle time-mixing, plus a Finch-specific token-shift connection.
4 Method
Eagle extends RWKV-4 with matrix-valued states, revised receptance, and gating, while Finch adds input-dependent interpolation to Token Shift and time-varying decay. These changes increase architectural expressivity while retaining recurrent state updates.
- Token Shift linearly interpolates current and previous inputs, letting each channel independently allocate new versus old information in receptance, key, value, and gate vectors.The mechanism can support induction heads within a single layer by accumulating past and current token data in separate subspaces.
- Eagle Time Mixing constrains its learned decay weights to (0,1), making the diagonal decay operator a contraction matrix.The recurrent formulation interprets the state as accumulated k^T v contributions whose channels decay according to w, with a learned boost for the current token.
- Channel Mixing remains essentially RWKV-4’s module, but Eagle reduces its hidden dimension from 4D to 3.5D to offset added gating weights.Finch retains the 3.5D hidden dimension despite adding a small number of LoRA parameters.
- Finch makes Token Shift data-dependent, so the allocation between current and prior inputs depends on both time steps.
- Finch’s decay vector varies independently across channels and over time according to the input, rather than remaining a fixed learned vector.The new LoRA mechanisms augment Eagle’s learned vectors with input-determined offsets and apply a further input-dependent transformation to decay.
5 RWKV World Tokenizer
The RWKV World Tokenizer combines vocabularies with manually selected non-European-language tokens and uses Trie-based longest-match encoding. Its 65,536-token design covers bytes, multilingual text, and programming-language content.
- BPE tokenization can produce unequal token counts for semantic chunks across languages, with lower performance for underrepresented languages and added economic burden.
- The tokenizer merges existing vocabularies and manually selects tokens for non-European languages.
- Its vocabulary contains 65,536 tokens arranged by byte length, including document-boundary tokens, byte encodings, and multilingual UTF-8 tokens.The multilingual range includes words, affixes, accented letters, Chinese characters, Hangul, Hiragana, Katakana, and emojis.
- The token design is intended to improve efficiency on multilingual text and programming-language source code.
- Trie-based encoding matches the longest vocabulary element from left to right to improve tokenization speed while retaining simplicity.The vocabulary construction is intended to reduce the burden that naive BPE and related methods place on minor languages.
6 RWKV World v2 Dataset
RWKV World v2 is a 1.12 trillion-token multilingual corpus built from hand-selected public sources to broaden training beyond English-heavy datasets. It emphasizes factual knowledge, code, and cultural works.
- RWKV World v2 contains 1.12 trillion tokens from diverse hand-selected publicly available data sources.
- The corpus is approximately 70% English, 15% multilingual data, and 15% code.
- Its contents emphasize factual knowledge, code, and cultural works including stories, books, subtitles, and conversations.
- The dataset is designed to support non-native English speakers, improve representation in model responses, and enable transfer across cultures and locales.
7 Pre-Trained Models
The paper publicly releases six Apache 2.0-licensed Eagle and Finch models trained on the 1.12 trillion-token RWKV World v2 multilingual corpus, spanning multiple model sizes.
- Six Apache 2.0-licensed models are released: Eagle 0.4B, 1.5B, 3B, and 7B, plus Finch 1.6B and 3B.
- All released models were trained on the 1.12 trillion-token RWKV World v2 multilingual corpus.
8 Language Modeling Experiments
Eagle and Finch perform strongly across multilingual, English, associative-recall, and long-context language-modeling evaluations. The models improve the multilingual accuracy–training-compute frontier, achieve high MQAR accuracy, and extend performance deeper into long sequences.
- 8.1 LM Evaluation Harness Benchmarks: Eagle and Finch achieve exceptionally high multilingual benchmark performance, with nearly all results significantly outperforming similarly sized tested models.
- 8.1 LM Evaluation Harness Benchmarks: Eagle and Finch substantially improve the multilingual accuracy–FLOPs Pareto frontier while retaining competitive performance on English benchmarks.
- 8.2 Associative Recall: Finch achieves extremely high MQAR accuracy and outperforms previously used non-transformer architectures for large language models.
- 8.3 Long Context Experiments: Eagle improves dramatically over RWKV-4 on PG19 long-sequence loss, while Finch lowers loss further into the sequence despite 4096-token training.
- 8.4 Bamboo Benchmark: At 1.5B and 3B scales, Eagle and Finch outperform vanilla Mamba by at least 7% average score on Bamboo while remaining comparable to Mamba trained on Hermes data.
9 Speed and Memory Benchmarks
Finch’s attention-like kernels provide favorable scaling for long sequences in the reported A100 experiments. Beyond 4k tokens, Finch is faster than Flash Attention and uses less memory than both Flash Attention and Mamba.
- Speed and Memory Benchmarks: Finch training speed scales linearly with sequence length in the reported attention-like kernel benchmarks.
- Speed and Memory Benchmarks: The comparisons use batch size 8, model dimension 4096, and head size 64 for Finch and Flash Attention, with Mamba configured at state dimension 16 and model dimension 8192.
- Speed and Memory Benchmarks: At sequence length 16k, Finch is around 4.2x faster than Flash Attention.
- Speed and Memory Benchmarks: Finch uses 40% less memory than Flash Attention and 17% less than Mamba in the reported comparisons.
10 Multimodal Experiments
Eagle is extended to music and vision-language modeling. RWKV-5 improves music-modeling loss over RWKV-4, while VisualRWKV combines CLIP with Eagle and achieves competitive visual reasoning results using smaller models.
- 10.1 Music Modeling: RWKV-5-Music has approximately 2% lower loss than RWKV-4-Music, mainly on the musical-score portion of the ABC data.
- 10.2 VisualRWKV: VisualRWKV combines a CLIP vision encoder with Eagle 1.5B or 3B language models for visual tasks.
- 10.2 VisualRWKV: VisualRWKV uses two-stage instruction tuning: projection-only feature alignment followed by end-to-end tuning while keeping the vision encoder frozen.
- 10.2 VisualRWKV: With CLIP-L 0.4B and Eagle 1.5B or 3B, VisualRWKV achieves results comparable to systems using larger encoders and 7B or 13B language models, sometimes outperforming them.
11 RWKV on Audio
AudioRWKV adapts RWKV for audio spectrograms with a quad-directional shift that captures neighboring relationships. On AudioSet, its smallest model achieves performance comparable to AST-AT with fewer parameters.
- AudioRWKV processes audio spectrograms using a quad-directional shift across neighboring tokens.The shift linearly interpolates neighboring tokens in two-dimensional spectrograms.
- AudioRWKV models range from 8.7M to 105M parameters and are evaluated on AudioSet.
- AudioRWKV-Tiny achieves comparable performance with a smaller model size than AST-AT.
12 Conclusions
The paper presents Eagle and Finch as progress in RNN-based language modeling through matrix-valued states and dynamic recurrence, with broad evaluations and public releases. It also identifies limitations involving embedding performance, training-data scale, and synthetic-data-induced behavior.
- Eagle and Finch integrate multiheaded matrix-valued states and dynamic data-driven recurrence mechanisms into RNN-based language models.
- The models demonstrate performance on MQAR and diverse linguistic benchmarks while retaining key RNN advantages.
- Eagle’s embedding performance on MTEB was not strong, and the authors leave appropriate state-information aggregation for future work.
- Synthetic GPT-3.5 and ChatGPT data causes released models to exhibit ChatGPT-like conversational behavior, including occasional claims of OpenAI training.
- The 1.12 trillion-token multilingual corpus is smaller than contemporary training datasets, motivating expansion toward more diverse and expansive data.
B Additional Architecture Details
The additional details describe RWKV’s recurrent efficiency, hardware-aware training, and Eagle and Finch mechanisms for richer state updates and data-dependent recurrence. They also cover related architectures, tokenizer results, and dataset-weighting choices.
- RWKV efficiency: Training uses non-time-dimension parallelization with a custom CUDA implementation to avoid inefficient repeated SRAM–HBM transfers.
- RWKV efficiency: RWKV supports unbounded sequence lengths without increased computing power or memory because its recurrence examines only the immediately preceding time step.
- Eagle mechanisms: Eagle uses matrix-valued states with geometrically decaying key-value terms that function as a memory bank for values.
- Finch mechanisms: Finch makes token shifts and decay rates data-dependent by augmenting learned parameters with LoRA-inspired weight matrices.
- Tokenizer and data: The dataset mostly preserves source data without up- or down-sampling, giving tokens equal weighting except for specified overrepresented-language subsampling.
- Tokenizer and data: The tokenizer is as efficient as GPT4’s cl100k_base overall and surpasses it on three non-European languages despite a smaller vocabulary.
G.1 Alignment Benchmark
Eagle and Finch are evaluated for Chinese alignment and related assistant capabilities across benchmark settings. Larger models and newer generations show stronger alignment performance, while additional evaluations compare reasoning, self-learning, and instruction-following behavior.
- Alignment Benchmark: AlignBench evaluates Chinese language models across professional knowledge, reasoning, mathematics, writing, and other categories.The benchmark contains 683 queries across eight categories, while the table lists the evaluated category headers.
- Alignment Benchmark: Eagle and Finch improve consistently on AlignBench as model size and generation progress.The trend spans multiple categories in the Chinese alignment benchmark.
- Alignment Benchmark: Eagle 7B and Finch 3B surpass smaller and previous-generation counterparts on overall alignment performance.The reported progression links higher scores to both increased model size and architectural generation.
- Additional Evaluations: Eagle 3B has a small MTBench advantage over similarly sized Mamba, while Eagle 7B performs similarly to the larger Raven-14B.MTBench contains 80 multi-turn questions across writing, roleplay, extraction, reasoning, math, coding, STEM, and humanities or social science.
- Additional Evaluations: RWKV-5 outperforms instruction-tuned Mistral-7B on intrinsic self-learning but trails a similarly sized DPO-aligned Mistral model slightly.Both models still achieve high SLC scores when using an external self-learning method.
- Additional Evaluations: Eagle 7B consistently outperforms Raven 7B across the evaluated zero-shot tasks, although both remain sensitive to prompt templates.The comparison uses F1-macro except for MathQA, which uses accuracy.