PaperlayerView exact revision on arXiv

Source-linked AI summary

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland, Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu

arXiv:2404.05892v4cs.CLcs.AI

TL;DR

Transformers’ expressive attention has quadratic sequence-length cost, motivating efficient alternatives. Eagle and Finch add matrix-valued states and dynamic recurrence to RWKV, achieving competitive performance across diverse benchmarks while retaining RNN-like inference efficiency.

  • Problem

    Transformer attention has quadratic time complexity in input sequence length, motivating sequence models with more efficient inference.

    Source: §1 · p. 3

  • Method

    Eagle and Finch extend RWKV with multi-headed matrix-valued states, dynamic data-dependent recurrence, and a multilingual training corpus and tokenizer.

    Source: §1 · pp. 3–4, §6 · p. 9

  • Results

    Eagle and Finch perform competitively or improve upon existing models across diverse sequence-modeling benchmarks, outperforming vanilla Mamba by at least 7% average score on Bamboo.

    Source: §1 · p. 3, §8.4 · p. 12

  • Takeaways & Limitations

    The models advance RNN-based language modeling while retaining key RNN advantages and supporting publicly accessible multilingual model development.

    Source: §1 · p. 4, §12 · p. 17

    Eagle showed weak embedding performance on MTEB, and an appropriate method for aggregating contextual information remains unresolved.

    Source: §12 · p. 17

Abstract

from arXiv · show

We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks. We release all our models on HuggingFace under the Apache 2.0 license. Models at: https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code at: https://github.com/RWKV/RWKV-infctx-trainer

1 Introduction

The paper introduces Eagle (RWKV-5) and Finch (RWKV-6) to improve RWKV-4’s expressivity and efficiency while avoiding transformers’ quadratic sequence-length complexity. It also contributes multilingual data and tokenization resources, open model releases, and evidence of competitive performance across diverse benchmarks.

Motivation and related work

  • Transformers’ expressive attention has quadratic time complexity in sequence length, motivating sub-quadratic alternatives such as recurrences, convolutions, and linear attention.The paper builds on RWKV-4, which offers efficient inference and training with parallelizable sequence processing.Source: §1 · p. 3

Architectural contributions

  • Eagle improves RWKV-4 with multi-headed matrix-valued states, reformulated receptance, and additional gating, while Finch adds data-dependent time-mixing and token-shift functions plus context-dependent low-rank decay adaptation.These changes target greater expressivity and flexibility while retaining the RWKV design’s efficiency characteristics.Source: §1 · p. 3

Data and tokenization

  • The RWKV World Tokenizer uses Trie-based greedy matching, and RWKV World v2 provides 1.12 trillion tokens of multilingual and code data, including vocabulary for underrepresented languages.Both resources are designed to improve multilingual performance and tokenization speed.Source: §1 · pp. 3–4

Empirical scope

  • Eagle and Finch perform competitively with, or improve upon, existing models across English and multilingual text, associative recall, music modeling, and vision-language benchmarks.The experiments support progress toward more efficient AI models across varied sequence-modeling domains and tasks.Source: §1 · pp. 3–4

Model releases and openness

  • The authors release four Eagle models ranging from 0.46 to 7.5 billion parameters and two Finch models with 1.6 and 3.1 billion parameters, alongside an open training pipeline.The releases are intended to support meaningful scaling comparisons, interpretability, and reproducibility of alternative-architecture LLMs.Source: §1 · p. 4

2 Background

Eagle and Finch build on RWKV-4 and linear attention to combine recurrent O(1)-per-token inference with highly parallelizable training. Their background traces this design space from classic RNNs and Transformers through linear attention, AFT, RWKV-4, and concurrent recurrent and state-space models.

RNNs

  • RNNs provide O(1) inference per step but classic designs are difficult to parallelize across time during training.They generate a hidden state h_t that is fed into the next step.Source: §2 · p. 4

Transformers

  • Transformers achieve SOTA performance but multi-headed dot-product self-attention has quadratic sequence-length complexity, motivating hybrid architectures.Such hybrids seek both O(1) per-token inference and fast parallelizable training.Source: §2 · p. 4

Linear Attention

  • Linear attention reorders feature-map operations so it can run recurrently in O(1) time per step or train in parallel, but naive versions underperform MHA transformers.The recurrent state accumulates φ(K_i^T)V_i at each time step.Source: §2 · pp. 4–5

RWKV Evolution

  • AFT enabled RWKV by using one attention head per feature dimension and learned pairwise positional biases w, which RWKV-4 replaced with channelwise decay rates.RWKV-4 also introduced a bonus term u that specially offsets the current input’s weight.Source: §2 · p. 5
  • RWKV-4 further combined token-shift, gating, small embedding initialization, and normalization, becoming an RNN that rivaled Transformers while retaining parallel training and O(1) per-token complexity.These changes targeted both attention and feed-forward sub-blocks.Source: §2 · p. 5

Related Work

  • Concurrent alternatives include data-dependent recurrent models such as HGRN, TransNormer, GLA, and Griffin, alongside SSMs with associative-scan parallelization and data-dependent A and B terms.The latter terms function similarly to Finch’s data-dependent dynamic recurrence.Source: §2 · p. 5

3 Eagle/Finch Architecture

Eagle and Finch refine RWKV-4 in two stages while retaining its stacked residual-block structure: Eagle expands the attention mechanism, and Finch adds data-dependent recurrence and token shifting.

Eagle

  • Eagle adds matrix-valued attention states, head-wise LayerNorm, SiLU attention gating, improved initialization, and removes receptance’s Sigmoid activation relative to RWKV-4.These changes each produce significant modeling improvements, according to the supplied passage.Source: §3 · p. 5

Finch

  • Finch further makes the decay schedule and token-shift data-dependent.Figure 1 depicts the Finch-specific token-shift connection in the time-mixing block.Source: §3 · p. 5, Figure 1 · p. 6

Core architecture

  • Both models retain RWKV-4’s core architecture of stacked residual blocks with Pre-LayerNorm Time-Mixing followed by Pre-LayerNorm Channel-Mixing sub-layers.These sub-layers correspond to Transformer attention and feed-forward network components.Source: §3 · p. 5

4 Method

The method develops Eagle and Finch through token shifting, time mixing, and channel mixing, with Finch adding input-dependent interpolation and time-varying decay. These mechanisms preserve recurrent computation while increasing how the models incorporate current and prior token information.

4.1.1 Eagle Token Shift

  • Eagle Token Shift independently learns how much new and old information each channel receives in receptance, key, value, and gate vectors, enabling induction heads within a single layer.The operation is based on linear interpolation between the current and previous token and is applied independently for each head.Source: §4.1.1 · pp. 6–7

4.1.2 Eagle Time Mixing

  • Eagle Time Mixing uses a recurrent decaying state: each channel decays independently, while the current token receives a learned boost before receptance acts like a linear-attention query.The decay vector is parameterized so its entries lie in (0,1), making the diagonal decay operator contractive.Source: §4.1.2 · p. 7

4.1.3 Channel Mixing

  • Channel Mixing remains the RWKV-4 design in Eagle and Finch, but Eagle reduces its hidden dimension from 4D to 3.5D to offset new gating parameters.Finch retains the 3.5D hidden dimension despite adding a small number of LoRA parameters.Source: §4.1.3 · p. 7

4.2.1 Finch Token Shift

  • Finch makes Token Shift data-dependent, so each channel’s allocation of new versus old information depends on both current and prior inputs.The mechanism uses trainable vectors and low-rank matrices to augment Eagle-style shifts with input-determined offsets.Source: §4.2.1 · pp. 7–8

4.2.2 Finch Time Mixing

  • Finch replaces Eagle’s fixed decay vector with channel-wise decay that varies over time according to the input.Its LoRA mechanism operates on Eagle Token-Shifted values and applies a further input-dependent transformation, forming a second-order Token-Shift variant.Source: §4.2.2 · p. 8

5 RWKV World Tokenizer

RWKV World Tokenizer manually balances multilingual vocabulary coverage by merging existing vocabularies and selecting tokens for non-European languages. Its 65,536-token byte-oriented design and Trie-based longest-match encoding target efficient processing of multilingual text and source code.

Vocabulary construction

  • The tokenizer merges GPT-NeoX-20B, GPT2, cl100k_base, Llama2, and Bloom vocabularies before manually selecting tokens for non-European languages.This construction addresses underrepresentation and the inefficiencies that naive BPE can impose on minor languages.Source: §5 · pp. 8–9

Vocabulary design

  • The vocabulary contains V = 65536 tokens numbered 0–65535, organized by byte length from document-boundary and byte tokens to multilingual subword tokens and six reserved tokens.Tokens 257–65529 cover UTF-8 words, affixes, accented letters, Chinese, Hangul, Hiragana, Katakana, and emojis; token 0 marks document boundaries, while 65530–65535 are reserved.Source: §5 · p. 9

Encoding

  • A Trie implements left-to-right longest-vocabulary-element matching to boost encoding speed while maintaining simplicity.The design is intended to improve efficiency on the multilingual corpus and programming-language source code.Source: §5 · p. 9

6 RWKV World v2 Dataset

The RWKV World v2 Dataset is a 1.12-trillion-token multilingual corpus designed to broaden beyond English-heavy training data while supporting factual knowledge, code, cultural works, and cross-cultural transfer.

Dataset composition

  • RWKV World v2 is a 1.12-trillion-token multilingual dataset assembled from diverse, hand-selected public sources, with approximately 70% English, 15% multilingual data, and 15% code.Its contents emphasize factual knowledge, code, and cultural works such as stories, books, subtitles, and conversations.Source: §6 · p. 9

Dataset goals

  • The dataset is designed to support non-native English speakers, improve representation in model responses, and enable knowledge transfer across cultures and locales.It specifically addresses the English-heavy focus of many widely used language-model training datasets.Source: §6 · p. 9

7 Pre-Trained Models

The paper releases six Apache 2.0-licensed Eagle and Finch models, all pretrained on the 1.12-trillion-token RWKV World v2 multilingual corpus.

7 Pre-Trained Models

  • Six publicly released models—Eagle 0.4B, 1.5B, 3B, and 7B, plus Finch 1.6B and 3B—were trained on the 1.12-trillion-token RWKV World v2 multilingual corpus.Appendix E provides detailed parameter counts and FLOPs calculations.Source: §7 · p. 9

8 Language Modeling Experiments

Eagle and Finch achieve strong multilingual, associative-recall, and long-context performance across standard, synthetic, and specialized benchmarks. Finch generally improves over Eagle, while both models compare favorably with competing architectures and models at similar scales.

8.1 LM Evaluation Harness Benchmarks

  • Eagle and Finch substantially improve the multilingual accuracy-versus-training-FLOPs Pareto frontier, while maintaining competitive performance on English benchmarks.Across the evaluated multilingual benchmarks, nearly all results significantly outperform similarly sized tested models.Source: §8.1 · p. 9, Figure 2 · p. 10, Figure 3 · p. 10

8.2 Associative Recall

  • Finch achieves extremely high MQAR accuracy and outperforms previously used large-language-model non-transformer architectures, with Eagle and Finch both improving significantly over RWKV-4.Differences between Mamba and Finch indicate that their differing combinations of matrix-valued state and data-dependent memory modification affect performance.Source: §8.2 · pp. 11–12

8.3 Long Context Experiments

  • On PG19 beyond token 2048, Eagle dramatically lowers loss relative to RWKV-4, while Finch improves further with loss continuing to decline deeper into the sequence.All compared models were pretrained with context length 4096, and Eagle achieves this improvement despite training solely at sequence length 4096.Source: §8.3 · p. 12, Figure 5 · p. 13

8.4 Bamboo Benchmark

  • At 1.5b and 3b scales, Finch and Eagle outperform vanilla Mamba by at least 7% average Bamboo score, while remaining within 0.7% of Mamba trained on Hermes data; Eagle-7b exceeds Pythia by 13.5% on average.Eagle-7b also surpasses LLaMA2-Chat-7b on several Bamboo tasks; code completion is omitted because all tested models failed.Source: §8.4 · p. 12, Table 5 · p. 13

9 Speed and Memory Benchmarks

Finch scales linearly in training speed with sequence length, matching Mamba’s scaling, while outperforming Flash Attention beyond 4k tokens and using less memory than both baselines.

Benchmark setup

  • The comparison evaluates Finch, Mamba2, and Flash Attention3 using attention-like kernels, with Mamba 2x adding a second kernel pass to mimic twice as many layers.Flash Attention uses the PyTorch implementation of Flash Attention v2.Source: §9 · p. 14

Speed

  • 4.2x faster than Flash Attention at sequence length 16k, with Finch becoming significantly faster beyond 4k.Finch’s training speed scales linearly with sequence length, similarly to Mamba.Source: §9 · pp. 14–15

Memory

  • 40% less memory than Flash Attention and 17% less than Mamba, Finch consistently leads both baselines in memory usage.Benchmarks use batch size 8, model dimension 4096, and head size 64 for Finch and Flash Attention; Mamba uses state dimension 16 and model dimension 8192.Source: §9 · pp. 14–15

Limitations

  • Further CUDA and algorithmic optimization of Finch could increase speed and parallelization, but these improvements remain future work.The reported implementation is not presented as fully optimized.Source: §9 · p. 15

10 Multimodal Experiments

Eagle is extended to multimodal tasks through music modeling and VisualRWKV for visual understanding. RWKV-5 improves music loss over RWKV-4-Music, while VisualRWKV achieves competitive results with smaller components and sometimes surpasses larger models.

10.1 RWKV Music Modelling

  • RWKV-5-Music reduces loss by approximately 2% versus RWKV-4-Music, primarily on musical-score content rather than file headers and control codes.The model uses 24 layers, dimension 512, a 128-token byte-level tokenizer, and a 1024-byte training context; Figure 8 plots loss over sequence position.Source: §10.1 · p. 15, Figure 8 · p. 15

10.2 VisualRWKV

  • VisualRWKV combines a CLIP vision encoder with Eagle 1.5B or 3B and uses two-stage instruction tuning for feature alignment followed by end-to-end fine-tuning.The projection layer is trained during alignment while the remaining components stay frozen; subsequent fine-tuning updates the projection layer and RWKV language model while keeping the vision encoder frozen.Source: §10.2 · p. 15
  • VisualRWKV achieves results comparable to larger vision-language systems and outperforms them on some benchmarks despite using smaller CLIP-L and 1.5B/3B Eagle language models.It uses a CLIP vision encoder and Eagle language model within a two-stage instruction-tuning setup; Table 6 covers GQA, ScienceQA-IMG, Text-VQA, and POPE.Source: §10.2 · pp. 15–16, Table 6 · p. 16

11 RWKV on Audio

AudioRWKV adapts RWKV to audio spectrograms by introducing a quad-directional shift that captures two-dimensional neighboring relationships. On AudioSet, AudioRWKV-Tiny achieves comparable performance to AST-AT with a smaller model size.

11 RWKV on Audio

  • AudioRWKV introduces quad-directional shift (Q-Shift) in each spatial-mix and channel-mix module to capture neighboring relationships in two-dimensional audio spectrograms.Q-Shift shifts and linearly interpolates all tokens with their neighboring tokens.Source: §11 · p. 16
  • AudioRWKV-Tiny achieves comparable performance with AST-AT on AudioSet while using a smaller model size.Experiments cover AudioRWKV models ranging from 8.7M to 105M parameters.Source: §11 · p. 16, Table 7 · p. 17

12 Conclusions · A Author Contributions

Eagle and Finch advance RNN-based language modeling through matrix-valued states and dynamic recurrence while retaining RNN advantages, but their limitations motivate broader data, larger models, and future efficiency work. The author contributions span architecture, training, evaluation, multimodal and embedding applications, manuscript development, and supporting experiments.

12 Conclusions

  • Eagle and Finch integrate multiheaded matrix-valued states and dynamic data-driven recurrence, achieving strong MQAR and diverse linguistic benchmark performance while retaining key RNN advantages.The models are publicly available under Apache 2.0 and were trained on an extensive multilingual corpus.Source: §12 · p. 17
  • Eagle’s embedding performance on MTEB was weak, despite its state being viewed as a high-quality context embedding, because an appropriate aggregation method remains unresolved.The authors leave this aggregation problem to future work.Source: §12 · p. 17
  • The models can mimic ChatGPT’s conversational style and occasionally claim OpenAI training because the corpus includes synthetic GPT-3.5 and ChatGPT data, rather than because of RWKV’s general architecture.The passage identifies this behavior as a specific data-and-training outcome.Source: §12 · p. 17
  • Future work prioritizes expanding the 1.12 trillion-token multilingual corpus, releasing 7B and 14B Finch models, and reducing inference and training costs through Mixture of Experts.The current corpus is described as smaller than contemporary training datasets such as LLaMA2.Source: §12 · p. 17

A Author Contributions

  • Bo Peng led the original RWKV-5 and RWKV-6 ideas, code, optimizations, tokenizer and dataset design, experiments, and training models from 0.4B to 7B.Daniel Goldstein and Fares Obeid contributed time-parallelization research, while Daniel Goldstein also supported manuscript organization, experiments, and revisions.Source: A Author Contributions · p. 28
  • Contributors organized, drafted, revised, proofread, formatted, and supported the manuscript, tables, figures, datasets, tokenizer, scaling analyses, and core model evaluations.Named contributions include manuscript leadership, evaluation work, dataset research, tokenizer optimization, and figure and table development.Source: A Author Contributions · p. 28
  • Contributors developed VisualRWKV using RWKV-5 with original code, Table 6 experiments, and 1.5-billion-to-3-billion-parameter models; others trained and evaluated AudioRWKV, investigated embeddings, and evaluated SLC.Source: A Author Contributions · p. 28

B Additional Architecture Details … G Additional Evaluations

The supplementary sections detail RWKV’s architecture, related work, dataset choices, computational costs, and multilingual tokenizer design. The tokenizer is competitive with GPT4’s cl100k_base on efficiency and substantially faster in the reported speed test.

B Additional Architecture Details

  • RWKV supports unbounded-length training and inference because its recurrence uses only the immediately previous timestep and requires no explicit positional encoding.Training parallelizes over non-time dimensions while retaining state operations in fast SRAM; a pure PyTorch time-parallel implementation is also provided.Source: B Additional Architecture Details · p. 29
  • Finch dynamically generates token-shift amounts and decay rates by augmenting learned parameters with LoRA-inspired weight matrices, at modest computational and model-size cost.The update is y′ = y + tanh(xA)B, and Finch’s data-dependent token shift lets information selectively enter or avoid the stream.Source: B Additional Architecture Details · pp. 29–30

C Additional Related Work

  • RWKV is situated among efficient-transformer, recurrent, and selective-state-space approaches that target lower inference cost while retaining long-range modeling.The related work discusses sparse or approximate attention, RetNet, token shift, Mamba, GateLoop, GLA, and independent Hawk and Griffin models.Source: C Additional Related Work · pp. 32–33

D Training Dataset Details

  • The RWKV World v2 dataset largely preserves source distributions with equal token weighting, while selectively subsampling some overrepresented languages.Automated data mixing is identified as future work rather than applied systematically.Source: D Training Dataset Details · p. 33

E Computing Costs

  • RWKV’s larger internal states provide more memory storage than RWKV-4, costing slightly higher FLOPs and memory usage.The state includes previous Time-mixing and Channel-mixing inputs plus WKV head memory; the latter dominates the stored information.Source: E Computing Costs · pp. 33–35

F.1 Designation

  • The multilingual tokenizer merges existing vocabularies into 65,536 byte-organized tokens covering multilingual text, code, and future reserved tokens.Its design includes byte tokens, UTF-8 sequences, words, scripts, accented letters, Chinese characters, Hangul, kana, and emojis.Source: §F.1 · p. 35

F.2 Efficiency Experiments

  • The tokenizer matches GPT4’s cl100k_base overall and surpasses it on three non-European languages despite a smaller 65,536-token vocabulary.Efficiency was evaluated across five natural languages and programming code using token count and average characters per token.Source: §F.2 · pp. 35–36, Table 12 · p. 36

F.3 Speed

  • 90.32 MB/s makes the Rust RWKV tokenizer 9.6× faster than OpenAI’s Tiktoken in the reported comparison.The test used Wikipedia’s 20220301.en corpus on an M2 Mac mini, with speed measured in MB/s; the Python implementation also outperformed Llama2’s tokenizer.Source: §F.3 · p. 36, Table 13 · p. 36

G.1 Alignment Benchmark … M Chat Examples - Comparison with RWKV-4

This section reports Eagle and Finch evaluations across alignment, dialogue, self-learning, zero-shot NLP, ablations, and multilingual chat examples, alongside training and initialization settings. It also presents matched-prompt comparisons with RWKV-4 and illustrates Eagle’s responses, including examples with reasoning errors.

G.1 Alignment Benchmark

  • Eagle and Finch performance on AlignBench improves consistently with model size and generation, with Eagle 7B and Finch 3B surpassing smaller and previous-generation counterparts.AlignBench evaluates Chinese alignment across 683 queries and eight categories using a critique-LLM judging methodology.Source: §G.1 · pp. 36–37, Table 14 · p. 37

G.2 MTBench

  • On MTBench, Eagle 3B has a small advantage over similarly sized Mamba, while Eagle 7B performs similarly to the much larger Raven-14B.MTBench contains 80 multi-turn questions across eight common prompt categories.Source: §G.2 · p. 37, Figure 11 · p. 38

G.3 Self-Learning

  • RWKV-5 outperforms instruction-tuned Mistral-7B on intrinsic self-learning capability but trails a similarly sized DPO-aligned Mistral model, while both methods achieve high SLC scores externally.SLC averages Curiosity and Knowledge-Limit Awareness scores; full results are reported in Table 15.Source: §G.3 · p. 37, Table 15 · p. 39

G.4 Zero-shot evaluation on additional NLP tasks

  • Eagle 7B consistently outperforms Raven 7B across the evaluated zero-shot tasks, although both models remain highly sensitive to the selected prompt template.The evaluation uses subsets of datasets previously used to test ChatGPT, with F1-macro as the metric except accuracy for MathQA.Source: §G.4 · p. 38, Table 16 · p. 39

H Hyperparameters

  • Training used bfloat16 for most parameters, float32 WKV computation, Adam with β1 = 0.9 and β2 = 0.99, 4096-token context, warmup, and cosine learning-rate decay.Weight decay was 0.001 on linear and embedding weights, and time_decay parameters used a special 2x learning-rate multiplier.Source: H Hyperparameters · p. 38

J Architectural Ablations

  • The architectural ablation trained a 170M-parameter RWKV-6 model from scratch on 330B Pile tokens with the GPT-NeoX-20B tokenizer and compared it with similarly sized Mamba, RWKV-4, and Pythia models.All comparison models used exactly the same dataset and tokenizer.Source: J Architectural Ablations · p. 40, Table 18 · p. 40

K DDLerp Ablations

  • The DDLerp ablation trained a six-layer, d_model = 768 Finch model on the 1.6B-token minipile at context length 512, selectively removing the data-dependent token-shift term.Final validation-loss results are presented in Table 19.Source: K DDLerp Ablations · p. 41, Table 19 · p. 41

L Non-English Chat Examples

  • Non-English examples show Eagle 7B producing Japanese tourism guidance, Spanish narrative text, and Japanese-to-English translation, while Eagle-Hermes-7B examples use OpenHermes 2.5 fine-tuning and the exact RWKV-4 comparison requests at temperature 1.0.The Eagle-Hermes system prompt instructs the model to help with user tasks, use markdown where appropriate, and provide its best attempt when uncertain.Source: L Non-English Chat Examples · pp. 41, 43, M Chat Examples - Comparison with RWKV-4 · pp. 43–44

M Chat Examples - Comparison with RWKV-4

  • Eagle provides detailed conversational responses for metaphor explanation, Chinese cooking, recipe follow-up questions, and summarization.The examples cover “Life is like cats,” Chinese meal selection and red braised pork belly, and summaries of RWKV’s architecture.Source: M Chat Examples - Comparison with RWKV-4 · pp. 47, 48, 49, 52
  • The cooking dialogue gives ingredient lists, preparation steps, and advice on seasoning, heat, simmering, sauce thickness, and serving temperature.The responses specify simmering around 30–45 minutes and emphasize maintaining suitable sauce consistency and serving the dish warm.Source: M Chat Examples - Comparison with RWKV-4 · pp. 48–50
  • Several logical answers are incorrect, including claims that penguins can fly and that the day described in the temporal puzzle is Saturday.These responses assert false conclusions rather than resolving the stated premises correctly.Source: M Chat Examples - Comparison with RWKV-4 · p. 51
  • The supplied RWKV background describes parallelizable Transformer-style training, constant-complexity RNN-style inference, scaling to 14 billion parameters, and performance comparable to Transformers.This passage presents the general RWKV architecture rather than a direct quantitative comparison between Eagle and RWKV-4 in the chat examples.Source: M Chat Examples - Comparison with RWKV-4 · p. 52

Related papers

Loading 2404.05892v4…