Source-linked AI summary
Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge
Heydar Soudani, Evangelos Kanoulas, Faegheh Hasibi
TL;DR
Low-frequency factual knowledge is harder for language models, motivating a comparison of RAG and synthetic-data fine-tuning for question answering. The paper evaluates these approaches across language models and introduces Stimulus RAG, which uses a retrieved hint to guide generation. RAG substantially outperforms fine-tuning, and Stimulus RAG outperforms the tested RAG and fine-tuning combinations.
Problem
Language models have constrained memorization for less frequent entities, creating a need to customize them for domain-specific factual question answering.
Source: §1 · p. 1Method
The paper compares fine-tuning and RAG across varied language models and settings, and introduces Stimulus RAG, which guides generation with a hint from retrieved documents.
Source: §1 · p. 2Results
RAG substantially outperforms fine-tuning, and Stimulus RAG outperforms the tested combinations of fine-tuning with and without retrieve-then-generate RAG.
Source: §1 · p. 2Takeaways & Limitations
Stimulus RAG bypasses the need for costly data augmentation and fine-tuning to enrich language models with less popular factual knowledge.
Source: §1 · p. 2The answer evaluation can mark equivalent phrasings incorrect and can mis-score outputs containing multiple entity names, motivating more advanced analysis of answer variations.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Language Models (LMs) memorize a vast amount of factual knowledge, exhibiting strong performance across diverse tasks and domains. However, it has been observed that the performance diminishes when dealing with less-popular or low-frequency concepts and entities, for example in domain specific applications. The two prominent approaches to enhance the performance of LMs on low-frequent topics are: Retrieval Augmented Generation (RAG) and fine-tuning (FT) over synthetic data. This paper explores and evaluates the impact of RAG and FT on customizing LMs in handling low-frequency entities on question answering tasks. We conduct extensive experiments on twelve LMs of varying size and type and different fine tuning, data augmentation, and retrieval models. Our findings indicate that while FT boosts the performance across entities of varying popularity, RAG surpasses FT by a large margin particularly for least popular factual knowledge. Additionally, the success of both RAG and FT approaches is amplified by improving retrieval and data augmentation techniques. Fine tuning, while beneficial for small LMs, requires extensive resources. To address this issue, we propose the new Stimulus RAG approach that surpasses the effectiveness of fine tuning based approaches, thereby eliminating the need for the costly data augmentation and fine tuning step for enriching LMs with less popular factual knowledge. The code is available at \url{https://github.com/informagi/RAGvsFT}.
1 INTRODUCTION
The introduction motivates customization for low-frequency factual knowledge and frames an unresolved comparison between RAG and fine-tuning. It compares the approaches across language models and settings, then introduces Stimulus RAG as a strong alternative to fine-tuning.
Study scope
- The study addresses the lack of a direct RAG–fine-tuning comparison for less popular knowledge, evaluating QA over Wikipedia documents across twelve LMs and varied setups.The comparison assumes that limited textual descriptions are available for a domain and application.Source: §1 · p. 2
Fine-tuning method
- For models under 2 billion parameters, full fine-tuning beats QLoRA on the downstream task, while QLoRA better preserves reasoning and outperforms full tuning when paired with RAG.Prompt-based QA generation also outperforms a fine-tuned QA-generation model for the downstream task.Source: §1 · p. 2
LM type and size
- Decoder-only models outperform similar-sized encoder-decoder models, and smaller fine-tuned models with RAG can match or exceed larger models.Larger models generally do not benefit from fine-tuning, unlike smaller models.Source: §1 · p. 2
Retrieval model
- Higher-performing retrievers improve RAG, while retriever performance decreases as factual knowledge becomes more popular.The introduction cites Figures 1, 7, and 8 for these retrieval-related findings.Source: §1 · p. 2
Stimulus RAG
- Stimulus RAG uses a hint extracted from top retrieved documents to guide response generation, targeting fine-tuning’s substantial resource demands.The introduction frames this as a way to avoid fine-tuning costs while seeking better performance than fine-tuned models with RAG.Source: §1 · p. 2
Fine-tuning vs. RAG
- RAG substantially outperforms fine-tuning, and Stimulus RAG surpasses all tested RAG and fine-tuning setups without costly fine-tuning.The paper reports that fine-tuning combined with RAG is on par with or better than vanilla RAG in all but one case.Source: §1 · p. 2
2 RELATED WORK
Related work covers parametric and retrieved knowledge, prior fine-tuning and RAG comparisons, and approaches to measuring entity popularity. It also motivates a RAG design that highlights likely answer-bearing text.
Parametric and non-parametric knowledge
- Fine-tuning updates parametric knowledge, with synthetic QA generation used to address limited domain data; Figure 2 contrasts this with prompt-injected documents.The cited QA-generation pipeline includes passage selection, answer extraction, question generation, and consistency filtering.Source: §2 · p. 2, Figure 2 · p. 3
Retrieval-augmented generation
Prior comparisons
- Earlier studies compared fine-tuning and RAG in general or specialized domains, while this work focuses on less popular factual knowledge and varies retrieval, augmentation, and tuning methods.The related-work discussion cites comparisons by Mosbach et al., de Luis Balaguer et al., and Ovadia et al. [43].Source: §2 · pp. 2–3
Popularity and retrieval
- Entity popularity is approximated using measures such as Wikipedia pageviews, while RAG accuracy can depend on retriever performance and the relevance, position, and number of prompt passages.Other cited popularity proxies include traffic metrics, content density, and subject–relation co-occurrence.Source: §2 · p. 3
Advanced RAG systems
3 METHODOLOGY
The methodology compares parametric fine-tuning and nonparametric RAG for factual question answering over entities with Wikipedia evidence. It also introduces Stimulus RAG, which prepends a retrieved hint to the generator prompt.
3.1 Task Definition
- Factual QA treats each fact as a subject–relationship–object triplet and uses Wikipedia datasets to pair pageview-based popularity estimates with entity evidence documents.Source: §3.1 · p. 3
3.2 Knowledge Injection with Fine-tuning
- Fine-tuning uses document-derived synthetic QA pairs, generated either end-to-end with Ushio et al. [55]'s T5-large or through Zephyr CoT prompting.Source: §3.2 · pp. 3–4
3.3 Knowledge Injection with RAG
- RAG retrieves and ranks corpus documents, then supplies the query and top-K documents to a generator; the study tests four FT/RAG combinations.Source: §3.3 · p. 4
3.4 Stimulus RAG
- Stimulus RAG splits top-K retrieved documents into sentences, reranks them with the same retriever, and prepends either the top sentence or its containing document as a hint.Source: Figure 4 · p. 4, §3.4 · p. 5
4 EXPERIMENTAL SETUP
The experiments compare long-tail factual QA across datasets, models, fine-tuning strategies, and retrieval setups. They assess answer accuracy while using an ideal-retriever assumption for some RAG comparisons.
4 EXPERIMENTAL SETUP
- The evaluation uses PopQA, WitQA, and EntityQuestion, with entities divided into five popularity buckets using dataset-specific popularity proxies.WitQA reports pageviews here to enable comparison with the other datasets; EntityQuestion contributes about 80% of its questions after filtering for unique Wikidata matches.Source: §4 · p. 5
- Accuracy counts a prediction as correct when a ground-truth response appears as a substring of the generated answer.This metric can reject equivalent phrasings and can count an answer listing multiple entities as correct; answer-variation analysis is left for future work.Source: §4 · pp. 5–6
- The study tests encoder-decoder and decoder-only LMs ranging from 80 million to over 11 billion parameters.The model set includes five FlanT5 sizes and smaller and larger decoder-only models.Source: §4 · p. 6
- Fine-tuning uses synthetic QA pairs generated from Wikipedia summaries and compares full parameter tuning with QLoRA-based PEFT.Table 2 reports accuracy for vanilla and fine-tuned models with and without RAG; Table 3 uses PEFT and prompt-generated QA pairs.Source: Table 2 · p. 5, §4 · p. 6, Table 3 · p. 6
- For RAG, the study tests BM25, Contriever, DPR, and BM25+DPR, and defines the Ideal retriever as returning the page’s summary paragraph first.The authors note that some answers occur outside that paragraph, so the Ideal retriever assumption is imperfect.Source: §4 · p. 6
- Stimulus RAG uses DPR to extract a hint from the top three retrieved documents and tests a top-sentence hint or a document containing that sentence.The two variants are SRAG(S) and SRAG(D), respectively.Source: §4 · p. 6
5 RESULTS
Across the tested setups, retrieval quality and the choice of fine-tuning strategy affect QA accuracy, with effects varying by model size and entity popularity. Stimulus RAG with three documents exceeds fine-tuned RAG in the reported comparisons without adding extra retrieved information.
5.1 Fine Tuning vs. RAG Performance
- PEFT improves accuracy in the fine-tuned-plus-RAG setup, whereas full fine-tuning performs better in most fine-tuned-without-RAG comparisons.The authors select PEFT for subsequent experiments because it better preserves prompt-based reasoning in the RAG setup.Source: §5.1 · p. 7
- Prompt-generated QA pairs yield better downstream performance than E2E-generated pairs, although E2E generates over 12 times more QAs.The authors use prompt-generated QAs in subsequent experiments.Source: §5.1 · p. 7
- Fine-tuning improves answer-generator performance, but its combination with RAG helps models up to 3B parameters and degrades performance for models from 7B to 11.3B.Decoder-only models also outperform similarly sized Flan-T5 encoder-decoder models.Source: §5.1 · p. 7
- DPR and BM25+DPR outperform BM25 and Contriever on retrieval across datasets, and stronger retrieval performance corresponds to higher QA accuracy.DPR and BM25+DPR perform comparably, so DPR is selected for subsequent experiments.Source: §5.1 · pp. 7–8, Table 5 · p. 7
- RAG most improves accuracy for the least popular entities, while fine-tuning raises accuracy across popularity levels.Retriever effectiveness is higher for less popular entities; for the most popular bucket, QA accuracy rises despite weaker retrieval, which the authors relate to LMs relying on internal knowledge.Source: §5.1 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8
5.2 Stimulus RAG performance
- Top-three-document RAG improves accuracy, while using five documents yields negligible gains or decreases despite higher DPR Recall@5.The authors suggest that some LMs struggle to use the additional information effectively.Source: §5.2 · pp. 8–9, Table 6 · p. 9
- Stimulus RAG without fine-tuning achieves higher accuracy than fine-tuned RAG using the same three documents.The hint and its document come from those top-three documents, so the comparison tests the highlighting method rather than additional evidence.Source: §5.2 · p. 9
6 DISCUSSION AND CONCLUSIONS
The paper compares ways to customize LMs for less-resourced domains and concludes that RAG outperforms FT for knowledge injection; SRAG is presented as a cost-effective alternative.
6 DISCUSSION AND CONCLUSIONS
- SRAG’s extracted-hint prompt surpasses other settings; its SRAG(S) and SRAG(D) columns are compared with -FT+RAG and +FT+RAG (Wilcoxon p-value < 0.01).The hint is added to the top of the input prompt.Source: Table 7 · p. 9, §6 · p. 9
- RAG significantly outperforms FT for knowledge injection, and RAG performance improves with higher-performing retrievers.Source: §6 · p. 9
- PEFT enhances downstream performance and preserves LMs’ reasoning abilities while incorporating new knowledge.Source: §6 · p. 9
- Prompt-based QA generation performs better than fine-tuned QA generation models on factual QA tasks.Source: §6 · p. 9
- A small fine-tuned LM with RAG can perform on par with or surpass a larger LM.Source: §6 · p. 9