Source-linked AI summary
Searching for Best Practices in Retrieval-Augmented Generation
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, Xuanjing Huang
TL;DR
RAG systems must address outdated or fabricated model outputs while managing the complexity and latency introduced by multi-step, variable retrieval workflows. This paper systematically evaluates representative module methods and their combinations, finding practices that balance effectiveness and efficiency and extending the analysis to multimodal retrieval and generation. It also identifies scope boundaries around modular evaluation, chunking coverage, and modality coverage.
Problem
RAG workflows contain multiple processing steps with varied implementations, but the optimal combination for the entire workflow had not been systematically established.
Method
The study compares representative methods for each RAG module, evaluates their individual contributions while holding other modules fixed, and sequentially selects methods for an optimized pipeline.
Results
The study recommends optimal RAG practices through extensive experiments and shows that multimodal retrieval improves visual question answering and accelerates multimodal content generation.
Takeaways & Limitations
The findings provide a foundation for implementing RAG systems with module choices informed by performance and efficiency considerations.
Takeaways & Limitations
The evaluation uses a modular design, examines only representative chunking techniques, and leaves joint retriever-generator fine-tuning and other modalities for future work.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) techniques have proven to be effective in integrating up-to-date information, mitigating hallucinations, and enhancing response quality, particularly in specialized domains. While many RAG approaches have been proposed to enhance large language models through query-dependent retrievals, these approaches still suffer from their complex implementation and prolonged response times. Typically, a RAG workflow involves multiple processing steps, each of which can be executed in various ways. Here, we investigate existing RAG approaches and their potential combinations to identify optimal RAG practices. Through extensive experiments, we suggest several strategies for deploying RAG that balance both performance and efficiency. Moreover, we demonstrate that multimodal retrieval techniques can significantly enhance question-answering capabilities about visual inputs and accelerate the generation of multimodal content using a "retrieval as generation" strategy.
1 Introduction
RAG addresses outdated or fabricated outputs by combining pretrained language models with retrieved, domain-specific context, but its multi-step workflow creates implementation and efficiency challenges. This study systematically evaluates module choices and combinations to identify practices that balance performance and efficiency, including multimodal retrieval extensions.
- Motivation: RAG combines pretrained language models with retrieval-based models to provide current, domain-specific context without updating model parameters.This supports rapid deployment when query-related documents are available.
- Workflow complexity: A typical RAG workflow includes query classification, retrieval, reranking, repacking, summarization, vector storage, and generator fine-tuning.The workflow contains multiple intervening processing steps whose implementations vary.
- Research gap: Different implementations and combinations of RAG steps affect both system effectiveness and efficiency, yet no systematic effort had evaluated the entire workflow.Examples include query rewriting, pseudo-response retrieval, and direct embedding-based retrieval.
- Study approach: The study compares representative methods, evaluates each method's contribution while holding other modules fixed, and then uses selected methods in subsequent experiments.Because testing every combination is infeasible, the approach narrows candidates before sequential optimization.
- Contributions: The authors investigate existing RAG approaches and combinations to recommend optimal practices through extensive experimentation.The study frames this investigation as a three-part contribution alongside an evaluation framework and multimodal experiments.
- Contributions: Multimodal retrieval improves question answering about visual inputs and accelerates multimodal content generation through retrieval as generation.The paper presents this as a contribution of its experimental study.
2 Related Work
Related work spans query, document, chunking, reranking, and model fine-tuning transformations for improving RAG retrieval and generation. Despite broad surveys, selecting a practical algorithm and applying RAG methods remains challenging.
- Query transformation: Query transformation methods generate pseudo-documents or decompose queries to improve the information available for retrieval.Examples include Query2Doc, HyDE, and TOC.
- Document transformation: Document-side methods generate pseudo-queries for retrieval documents, while contrastive learning brings query and document embeddings closer in semantic space.These approaches transform retrieval sources or their representations to improve matching.
- Chunking and reranking: Chunking choices trade off sentence fragmentation against irrelevant context, and reranking filters irrelevant documents when retrieved chunks are numerous.The related work identifies chunk length and post-retrieval filtering as important retrieval design choices.
- Fine-tuning: Fine-tuning research separately optimizes generators or retrievers, while holistic approaches jointly fine-tune both despite increased cost.The cited work targets faithful generation, useful passages, or integrated system performance.
- Research gap: Existing surveys cover text generation, LLM integration, multimodal systems, and AI-generated content, but practical algorithm selection remains challenging.This paper focuses on best practices for applying RAG methods in LLMs.
3 RAG Workflow
The RAG workflow combines query classification, chunking, embedding, retrieval, reranking, and summarization modules, each with multiple implementation choices. The study evaluates representative alternatives to identify practices balancing retrieval quality, response efficiency, and system coverage.
- 3.1 Query Classification: Query classification separates queries that can be answered from user-provided information from those for which retrieval may be necessary.The study categorizes 15 tasks by information sufficiency and trains a classifier to automate the decision.
- 3.2 Chunking: Chunking balances context and efficiency: larger chunks improve comprehension but increase processing time, whereas smaller chunks improve recall and reduce time but may lack context.The study uses sentence-level chunking and examines chunk size, overlap, and chunking techniques such as small-to-big and sliding window.
- 3.2 Embedding: LLM-Embedder offers performance comparable to BAAI/bge-large-en while being three times smaller, motivating its selection for balancing performance and model size.The embedding-model comparison uses the MSMARCO dataset and corpus.
- 3.3 Vector Databases: Milvus is identified as the most comprehensive evaluated vector database because it meets all four criteria: multiple index types, billion-scale support, hybrid search, and cloud-native capabilities.The comparison covers Weaviate, Faiss, Chroma, Qdrant, and Milvus.
- 3.4 Retrieval: Hybrid Search with HyDE is recommended as the default retrieval method because it combines strong performance with relatively low latency, while query rewriting and decomposition are less effective.The evaluation uses the TREC DL 2019 and 2020 passage-ranking datasets; supervised methods outperform unsupervised methods.
- 3.7 Summarization: Recomp is recommended for summarization, while LongLLMLingua is retained as an alternative because its generalization is better despite weaker benchmark performance.The methods are evaluated on NQ, TriviaQA, and HotpotQA.
4 Searching for Best RAG Practices
The study evaluates RAG modules across diverse tasks using a comprehensive benchmark, comparing their effects on effectiveness and latency. Query classification, retrieval, reranking, repacking, and summarization each contribute differently to system performance.
- Comprehensive Evaluation: The evaluation covers commonsense reasoning, fact checking, open-domain QA, multihop QA, and medical QA using task-specific and RAG capability metrics.The measured capabilities include Faithfulness, Context Relevancy, Answer Relevancy, Answer Correctness, and Retrieval Similarity.
- Results and Analysis: Query classification improved the overall score from 0.428 to 0.443 while reducing latency from 16.41 to 11.58 seconds per query.
- Results and Analysis: Hybrid with HyDE achieved the highest retrieval-module RAG score of 0.58 but required 11.71 seconds per query, whereas reranking remained necessary for response quality.MonoT5 achieved the highest average reranking score.
- Results and Analysis: Reverse repacking achieved an RAG score of 0.560, indicating that placing more relevant context closer to the query produced the strongest repacking result.
- Results and Analysis: Recomp performed best for summarization, while removing summarization could reduce response time in time-sensitive applications.Recomp was preferred because it addresses the generator’s maximum length constraints.
- Results and Analysis: The experiments show that query classification, retrieval, reranking, repacking, and summarization each contribute uniquely to overall RAG performance.
5 Discussion
The paper proposes separate RAG practices for maximum performance and for balancing efficiency with efficacy. It also extends RAG to multimodal retrieval, using stored content directly when similarity is high and generation otherwise.
- Best Practices for Implementing RAG: The paper recommends two RAG recipes: one maximizing performance and another balancing efficiency and efficacy.
- Best Practices for Implementing RAG: The best-performance practice combines query classification, Hybrid with HyDE retrieval, monoT5 reranking, Reverse repacking, and Recomp summarization, yielding an average score of 0.483.The configuration is computationally intensive.
- Best Practices for Implementing RAG: The balanced-efficiency practice uses query classification, Hybrid retrieval, TILDEv2 reranking, Reverse repacking, and Recomp summarization.The paper identifies retrieval as the majority contributor to processing time and recommends Hybrid retrieval for lower latency.
- Multimodal Extension: Multimodal RAG incorporates text2image and image2text retrieval over paired image-text descriptions, supporting visual question answering and multimodal content generation.
- Multimodal Extension: Retrieval-based methods are presented as more grounded and efficient when suitable stored material exists, while their maintainability can improve by enlarging and improving retrieval sources.
- Multimodal Extension: In text-to-image retrieval, a sufficiently similar stored image is returned directly; otherwise, an image generation model creates the output.
6 Conclusion
The study identifies optimal RAG practices through systematic module comparisons, introduces a comprehensive evaluation benchmark, and uses extensive experiments to support recommendations for RAG implementation.
- Conclusion: The paper systematically assesses alternatives for each RAG module and recommends effective approaches to improve the quality and reliability of generated content.
- Conclusion: It introduces a comprehensive RAG evaluation benchmark and conducts extensive experiments to determine best practices among competing alternatives.
Limitations
The study’s modular design simplifies the search for optimal RAG implementations, but its evaluation leaves several areas for future investigation. These include joint retriever-generator fine-tuning, broader chunking comparisons, and additional modalities.
- Fine-tuning scope: The study evaluates fine-tuning methods for LLM generators but leaves joint retriever-generator training for future work.Previous studies have shown that jointly training the retriever and generator is feasible.
- Chunking scope: Evaluation of chunking is limited to representative techniques within the chunking module because vector-database construction and experiments are costly.The authors suggest studying different chunking techniques across entire RAG systems in future work.
- Modality scope: The study discusses RAG for NLP and extends it to image generation, while speech and video remain future research directions.
A Experimental Details
This section reports that the paper provides detailed experimental settings for each module, including datasets, training parameters, and additional results.
- Experimental settings cover dataset specifics, training parameters, and additional experimental results for each module.
A.1 Query Classification
The query-classification experiments use generated and established question data, with a multilingual BERT classifier trained under specified settings. Results are reported in Table 1.
- Datasets: The dataset combines a subset of Databricks-Dolly-15K with additional data generated using GPT-4.The prompt template for generating questions appears in Table 14.
- Implementation Details: The query classifier is BERT-base-multilingual-cased, trained with batch size 16 and learning rate 1e-5.The evaluation results are showcased in Table 1.
A.2 Experimental Details of Retrieval Methods
The retrieval-method experiments evaluate sparse, unsupervised dense, and supervised dense retrieval alongside query transformations using passage-ranking benchmarks, standard retrieval metrics, and latency.
- Datasets: Retrieval methods are evaluated on the TREC DL 2019 and 2020 passage-ranking datasets.
- Metrics: Evaluation reports mAP, nDCG@10, R@50, R@1k, and average per-query latency.mAP and nDCG@10 are order-aware, whereas R@k is order-unaware.
- Retrieval Methods: Sparse retrieval uses BM25, unsupervised dense retrieval uses Contriever, and supervised dense retrieval uses LLM-Embedder.BM25 and Contriever use Pyserini implementations, while dense indexing uses Faiss with Flat configuration.
- Query Transformations: Query rewriting uses Zephyr-7b-alpha9, while query decomposition uses GPT-3.5-turbo-0125.The models are prompted for query rewriting and decomposition, respectively.
A.3 Experimental Details of Reranking Methods
The reranking experiments evaluate retrieval methods on MS MARCO using standard ranking metrics and compare several reranking implementations. Results show a performance–latency trade-off, with RankLLaMA performing best and TILDEv2 fastest but requiring indexed passages.
- Evaluation setup: The evaluation uses MS MARCO Passage ranking data with MRR@1, MRR@10, MRR@1k, and Hit Rate@10.MRR@10 is the official MS MARCO metric.
- Methods: The reranking implementations include monoT5, monoBERT, RankLLaMA, and TILDEv2, based on T5, BERT, Llama-2, and BERT models respectively.
- Results: All reranking methods improve performance across metrics compared with randomly shuffled ordering and the BM25 retrieval baseline.
- Results: RankLLaMA performs best, while monoT5 and monoBERT achieve approximately equal performance with increasing latency across methods.
- Results: TILDEv2 takes approximately 10 to 20 milliseconds per query, trading lower latency for performance and requiring passages to be present in the indexed collection.Preprocessing must be repeated for unseen passages, which negates its efficiency advantages.
A.4 Experimental Details of Summarization Methods
The experiments assess summarization and RAG methods across multiple QA tasks and datasets using task-specific metrics and RAG capability measures. They compare context settings, compression methods, and RAG modules under controlled evaluation procedures.
- Summarization methods: Summarization methods are evaluated on NQ, TriviaQA, and HotpotQA using F1 score and the number of tokens changed after summarization.Experiments use Llama3-8B-Instruct with a summarization ratio of 0.4.
- RAG evaluation: The broader RAG evaluation covers commonsense reasoning, fact checking, open-domain QA, multi-hop QA, and medical QA across multiple datasets.
- Metrics: Task performance uses accuracy for commonsense reasoning, fact checking, and medical QA, and token-level F1 and Exact Match for open-domain and multi-hop QA.The final RAG score averages the five RAG capabilities.
- Metrics: RAG capability evaluation uses Faithfulness, Context Relevancy, Answer Relevancy, Answer Correctness, and Retrieval Similarity.The RAGAs metrics are evaluated with GPT-4 as judge, while Retrieval Similarity uses embedding-based cosine similarity.
- Evaluation procedure: The evaluation uses zero-shot generation, greedy decoding, response preprocessing, and document truncation for excessively long retrieved documents.