PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Retrieval-Augmented Generation: A Study of Best Practices

Siran Li, Linus Stenzel, Carsten Eickhoff, Seyed Ali Bahrainian

arXiv:2501.07391v1cs.CLcs.AI

TL;DR

RAG design choices require systematic evaluation because language models’ static knowledge can limit factual and up-to-date responses. This paper uses ablation-based benchmarking across model, prompt, retrieval, query, and knowledge-base configurations. Contrastive In-Context Learning RAG performs best overall, with Focus Mode RAG second, while several other tested factors do not meaningfully improve the reported metrics.

  • Problem

    The study addresses limited evidence about how RAG components and configurations affect response quality, despite RAG’s role in supplying language models with external, up-to-date knowledge.

  • Method

    The paper conducts ablation studies and a custom evaluation across RAG configurations, including Query Expansion, Contrastive In-context Learning, multilingual knowledge bases, and Focus Mode.

  • Results

    Contrastive In-Context Learning RAG outperforms the other tested variants, Focus Mode RAG ranks second, and Query Expansion, multilingual representations, document-size variations, and retrieval stride do not meaningfully improve Table 2 metrics.

  • Takeaways & Limitations

    The findings identify configuration-specific best practices, emphasizing Contrastive In-Context Learning and concise, high-precision retrieved context within the tested settings.

  • Takeaways & Limitations

    The study does not test combinations of its approaches in depth, conducts most experiments with a 7B model, and evaluates multilingual retrieval only with English, French, and German.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) systems have recently shown remarkable advancements by integrating retrieval mechanisms into language models, enhancing their ability to produce more accurate and contextually relevant responses. However, the influence of various components and configurations within RAG systems remains underexplored. A comprehensive understanding of these elements is essential for tailoring RAG systems to complex retrieval tasks and ensuring optimal performance across diverse applications. In this paper, we develop several advanced RAG system designs that incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG. Our study systematically investigates key factors, including language model size, prompt design, document chunk size, knowledge base size, retrieval stride, query expansion techniques, Contrastive In-Context Learning knowledge bases, multilingual knowledge bases, and Focus Mode retrieving relevant context at sentence-level. Through extensive experimentation, we provide a detailed analysis of how these factors influence response quality. Our findings offer actionable insights for developing RAG systems, striking a balance between contextual richness and retrieval-generation efficiency, thereby paving the way for more adaptable and high-performing RAG frameworks in diverse real-world scenarios. Our code and implementation details are publicly available.

1 Introduction

The paper examines how RAG components and configurations affect response quality, motivated by the need to improve the factuality and timeliness of language-model outputs. It uses extensive ablation-based benchmarking to identify best practices across retrieval, prompting, model, and knowledge-base choices.

  • RAG integrates external knowledge during inference to address language models’ static knowledge and factual-reliability concerns.
  • The study investigates nine questions spanning LLM size, prompts, chunk size, knowledge-base size, retrieval stride, query expansion, demonstrations, multilingual documents, and sentence-level retrieval.
  • A custom evaluation framework and ablation studies assess individual RAG components and configurations empirically.
  • The benchmark contributes an extensive analysis of best practices in RAG setups.
  • Query Expansion, Contrastive In-context Learning demonstrations, multilingual knowledge bases, and Focus Mode RAG are presented as novel contributions of the study.

3 Methods

The methods study RAG design choices through targeted research questions and a three-component architecture. The investigated variants cover model, prompt, document, retrieval, query, demonstration, multilingual, and sentence-level context choices.

  • The study evaluates LLM size, prompt wording, document chunk size, knowledge-base size, and retrieval-stride frequency as RAG factors.
  • Query Expansion broadens search coverage with relevant fields before targeted retrieval, enabling evaluation of its impact and efficiency.
  • Contrastive In-context Learning uses evaluation examples as a knowledge base while masking the active query during retrieval.
  • Multilingual experiments test whether context in multiple languages hinders generation or enriches information for more accurate responses.
  • Focus Mode retrieves essential sentences instead of entire documents to balance targeted context against broader retrieval coverage.
  • The RAG system combines query expansion, retrieval, and text generation modules for experiments across these configurations.

A. Query Expansion Module

The Query Expansion module uses a language model to enlarge the original query with relevant keyword phrases before retrieval. This broad-first search design defines a larger search space for subsequent focused retrieval.

  • A Flan-T5 model augments an initial query q by generating N expanded queries q′ containing relevant keyword phrases.
  • The framework’s query-expansion stage feeds the expanded keywords together with the original query into retrieval.
  • The autoregressive T5 process encodes q into hidden state h and predicts each token y_t conditioned on prior tokens y_<t and h.
  • Repeating token generation produces the set of relevant expanded queries used to broaden retrieval.

B. Retrieval Module

The Retrieval Module retrieves document contexts through staged similarity search and can narrow them to relevant sentences. Retrieval Stride additionally updates context during generation, whereas the baseline keeps context fixed.

  • Documents are chunked and indexed as embeddings, enabling similarity-based retrieval of relevant contexts.
  • The staged process first retrieves documents using the original and expanded queries, then reranks with the original query, and finally optionally selects relevant sentences.
  • Focus Mode is the sentence-level third stage, while the baseline performs only direct document retrieval with the original query.
  • Retrieved chunks or sentences form context K, which is supplied with query q to the language model for response generation.
  • Retrieval Stride dynamically updates K at specified intervals using the generated text produced up to time step t_k.
  • Without Retrieval Stride, the baseline keeps K fixed throughout generation.

4 Experimental Setup

The experiments evaluate RAG variants on TruthfulQA and MMLU using a Wikipedia-based knowledge base, multiple generation-quality metrics, and specified retrieval and generation implementations.

  • Evaluation Datasets: TruthfulQA tests commonsense truthfulness, while MMLU tests specialized knowledge across educational and professional subjects.The evaluation uses 817 TruthfulQA questions and 1,824 sampled MMLU questions.
  • Knowledge Base: Wikipedia Vital Articles provide the RAG knowledge base, using Level 3 and Level 4 articles for broad coverage and manageable size.French and German articles are added in the multilingual setting.
  • Evaluation Metrics: The study measures generation quality with ROUGE, embedding cosine similarity, MAUVE, and FActScore.MAUVE is evaluated across the entire dataset because few samples can yield unreliable distribution estimates; FActScore uses GPT-3.5-turbo for factual scoring.
  • Implementation: The implementation uses FLAN-T5-small for query expansion, FAISS with Sentence Transformer embeddings for retrieval, and Mistral-family models for generation.The Instruct7B model is the baseline, with HelpV1 prompting, 64-token chunks, and Level 3 Wikipedia articles.
  • Evaluation Examples: Table 1 illustrates dataset questions together with corresponding correct and incorrect answers.For MMLU, the correct option is treated as the correct answer and all other options as incorrect.

5 Experiments and Results

The experiments compare RAG configurations across relevance metrics and show that prompt design, model size, retrieval stride, and specialized retrieval strategies matter more than several scale-based changes.

  • LLM Size: Instruct45B outperforms Instruct7B particularly on TruthfulQA, while its gains on MMLU are less notable.Subsequent experiments use Instruct7B as the baseline model.
  • Prompt Design: Helpful prompts consistently outperform adversarial prompts, with HelpV2 and HelpV3 achieving the highest scores.The results show that small wording changes can influence performance across relevance metrics.
  • Document Size: Chunk sizes from 48 to 192 tokens produce minimal performance differences, with 192-token chunks slightly better on some metrics.The small variations suggest that increasing chunk size does not significantly affect performance.
  • Knowledge Base Size: Larger knowledge bases and retrieving more documents yield no statistically significant improvement in output quality.Additional documents may be irrelevant or redundant for specific queries.
  • Retrieval Stride: Reducing retrieval stride from 5 to 1 lowers ROUGE, embedding cosine similarity, and MAUVE, indicating that frequent retrievals disrupt context coherence and relevance.The study reports that larger strides preserve context stability better for this task.
  • Query Expansion: Query Expansion produces only marginal gains, with slight metric improvements as the TruthfulQA retrieval filter grows.The study attributes the limited effect to relevant documents often being retrieved without expansion.
  • Contrastive In-context Learning: Contrastive In-context Learning substantially improves relevance, with ICL1Doc+ gaining 3.93% ROUGE-L on TruthfulQA and 2.99% MAUVE on MMLU.The design retrieves correct and incorrect examples as a knowledge base.

8. Multilingual Knowledge Base: This experi-

The study compares multilingual retrieval, sentence-level Focus Mode, and factuality across RAG variants. Multilingual configurations decline relative to baseline, whereas Focus Mode and Contrastive In-context Learning perform strongly.

  • Multilingual Knowledge Base: Multilingual retrieval configurations show lower performance and relevance than the baseline.The paper suggests that synthesizing information across multiple languages is challenging for the model.
  • Focus Mode: Focus Mode ranks retrieved sentences by query relevance and supplies only the most relevant sentences to the generator.For TruthfulQA, 80Doc80S achieves the best results across most metrics, including a 1.65% ROUGE-L gain.
  • Focus Mode: On MMLU, Focus Mode improves selected metrics, with 2Doc1S increasing MAUVE by 0.49% and 120Doc120S increasing embedding cosine similarity by 0.81%.The results indicate benefits from focusing on highly relevant sentences.
  • Factuality Assessment: ICL1D+ achieves the highest factuality scores, reaching 57.00 on TruthfulQA and 74.44 on MMLU.Focus Mode 120Doc120S ranks second on MMLU with 65.87, while 80Doc80S moderately improves TruthfulQA.
  • Factuality Assessment: RAG variants without retrieval underperform, while Contrastive In-context Learning and Focus Mode are the strongest factuality configurations.ExpandL also performs well on TruthfulQA, achieving 55.82.
  • Qualitative Analysis: Generated examples show ICL1D+ and Focus Mode producing concise, factual responses aligned with the intended queries.The reported examples emphasize avoidance of verbose or irrelevant content.

6 Discussion and Key Findings

Across 74 experiments, Contrastive In-Context Learning RAG performed best, while Focus Mode RAG ranked second; several other configuration changes produced limited gains and prompt formulation remained important.

  • Contrastive In-Context Learning RAG outperformed all other variants, with a larger advantage on the specialized-knowledge MMLU dataset.
  • Focus Mode RAG ranked second and significantly outperformed other baselines by using precise, concise retrieved documents.
  • Query Expansion, multilingual representations, document-size variations, and retrieval stride did not meaningfully improve Table 2 metrics.
  • For factuality, Contrastive In-Context Learning RAG and Focus Mode RAG remained the top models, while Query Expansion ranked second on TruthfulQA.
  • Prompt formulation remained crucial even within RAG architectures.

7 Conclusions and Future Work

The study compares existing and four proposed RAG configurations across two datasets and six metrics, identifying strong-performing approaches while outlining unresolved evaluation gaps and future directions.

  • The study compares existing RAG architectures and four new configurations across two datasets and six evaluation metrics.
  • Contrastive In-Context Learning RAG, Focus Mode RAG, and Query Expansion RAG achieved the best overall results.
  • Future work includes dynamically adapting retrieval to prompts and contexts and using AutoML for task-specific retrieval optimization.
  • The experiments did not test combinations of two or more studied approaches.
  • Most experiments used the 7B model, so model-size effects were not studied in depth.

A Appendix

The appendix documents evaluation resources, knowledge-base statistics, prompt variants, and representative adversarial or conversational prompt formulations used in the study.

  • The appendix shares details of the evaluation datasets, knowledge base, and ablation-study settings.
  • Prompt Variants: HelpV1, HelpV2, and HelpV3 prompt variants instruct the model to answer accurately, precisely, concisely, or informatively.
  • Table 4 reports knowledge-base statistics using article counts, sentence ranges and averages, and average words per article.
  • Prompt Variants: AdversV1 frames the model as an imaginative storytelling bot asked to produce a detailed creative tale.
  • Prompt Variants: AdversV2 frames the model as a friendly dog responding with barks, playful sounds, and dog-like communication.

A.3 Contrastive ICL Prompt

The appendix specifies baseline and contrastive in-context prompts, multilingual variants, and representative generation examples used to compare RAG configurations.

  • A.3 Contrastive ICL Prompt: The Contrastive ICL section defines baseline, one- and two-demonstration, and contrastive prompt structures.
  • A.3 Contrastive ICL Prompt: The baseline prompt combines HelpV1, retrieved context, a question, and an answer field.
  • A.3 Contrastive ICL Prompt: ICL1D and ICL2D provide one or two correct-answer demonstrations before the target question.
  • A.3 Contrastive ICL Prompt: ICL1D+ and ICL2D+ add incorrect-answer demonstrations alongside correct-answer demonstrations.
  • A.4 Multilingual Setting: MultiLingo variants randomly replace English documents with French or German documents before embedding.
  • A.5 Generation Examples: Table 5 presents generated examples from TruthfulQA and MMLU, including top-performing configurations for several RAG components.
  • A.4 Multilingual Setting: MultiLingo+ additionally prompts the system to answer in English.
Loading 2501.07391v1…