PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Literature Review of Retrieval-Augmented Generation: Techniques, Metrics, and Challenges

Andrew Brown, Muhammad Roman, Barry Devereux

arXiv:2508.06401v3cs.DLcs.AIcs.CLcs.IR

TL;DR

RAG research has become fragmented across retrieval methods, architectures, domains, and evaluation practices, motivating a transparent synthesis of the field. This review applies a citation-weighted PRISMA protocol to 128 highly cited studies, finding broad progress alongside uneven evaluation and persistent resource, security, and coverage limitations.

  • Problem

    RAG research has diversified rapidly, but its results are fragmented and evaluation protocols remain evolving.

  • Method

    The review synthesises 128 highly cited RAG studies from 2020 to May 2025 using a citation-weighted PRISMA protocol and maps datasets, architectures, metrics, and challenges.

  • Results

    RAG evaluation is broad but uneven: overlap metrics dominate, while retrieval-quality measures, human judgements, and LLM-as-judge protocols are increasingly used.

  • Takeaways & Limitations

    Recent progress centres on hybrid retrieval, graph-aware indexing, iterative retrieval, efficiency-oriented compression, memory, and multimodality.

  • Takeaways & Limitations

    Citation thresholds may underrepresent recent breakthroughs and niche domain-specific work because citations accumulate unevenly over time.

Abstract

from arXiv · show

This systematic review of the research literature on retrieval-augmented generation (RAG) provides a focused analysis of the most highly cited studies published between 2020 and May 2025. A total of 128 articles met our inclusion criteria. The records were retrieved from ACM Digital Library, IEEE Xplore, Scopus, ScienceDirect, and the Digital Bibliography and Library Project (DBLP). RAG couples a neural retriever with a generative language model, grounding output in up-to-date, non-parametric memory while retaining the semantic generalisation stored in model weights. Guided by the PRISMA 2020 framework, we (i) specify explicit inclusion and exclusion criteria based on citation count and research questions, (ii) catalogue datasets, architectures, and evaluation practices, and (iii) synthesise empirical evidence on the effectiveness and limitations of RAG. To mitigate citation-lag bias, we applied a lower citation-count threshold to papers published in 2025 so that emerging breakthroughs with naturally fewer citations were still captured. This review clarifies the current research landscape, highlights methodological gaps, and charts priority directions for future research.

I. INTRODUCTION

This review addresses fragmented RAG research and evolving evaluation practices through a transparent, PRISMA 2020 systematic synthesis of influential studies published from 2020 to May 2025. It maps RAG architectures, datasets, evaluation practices, and research challenges while documenting the selection process and its constraints.

  • Motivation: RAG combines external retrieval with generative models to address hallucinations, stale knowledge, and knowledge-intensive queries.The approach combines parametric memory learned during pre-training with non-parametric evidence retrieved at inference time.
  • Limitations: The review’s citation thresholds foreground influential contributions but risk citation-lag bias against recent breakthroughs and niche domain-specific work.The authors identify this as a limitation of the systematic-review methodology.
  • Review objectives: The synthesis catalogues datasets, architectures, evaluation metrics, and deployment challenges for researchers and engineers.The original DPR plus sequence-to-sequence generator architecture is treated as the standard baseline for characterising variants.
  • Review scope: 128 influential RAG studies were included after screening 4,721 records, removing 1,494 duplicates, 158 out-of-range items, and 2,867 below-threshold items.The process screened 202 records and assessed 144 full texts before inclusion.
  • Publication trends: Identified articles increased annually from 2020 to 2023, with a pronounced increase in 2024, while 2025 counts are lower because the year was incomplete.Year-to-year comparisons are affected by staged database indexing, partial 2025 coverage, and the citation thresholds.

C. Domain Characteristics of Included Studies

The 128 included studies span multiple RAG application domains, with knowledge-intensive tasks and open-domain question answering forming the largest reported categories. Their documented system characteristics cover datasets, chunking, retrieval, encoders, and generation models.

  • Coding scheme: Studies were assigned one primary domain for proportional reporting, while secondary tags such as multimodal and conversational were retained without double-counting.The coding rules and examples are reported in Table III.
  • Domain distribution: Knowledge-intensive tasks accounted for 27.34%, followed by open-domain question answering at 15.62%, software engineering at 10.16%, and medical studies at 8.59%.Evaluation comprised 7.03%; multimodal and conversational AI each represented 4.69%.
  • Study characteristics: Table III catalogues datasets, chunking mechanisms, retrieval mechanisms, vector-space encoders, and generation models across the included RAG studies.The table is explicitly described as covering study characteristics by domain.
  • Retrieval mechanisms: Retrieval mechanisms include dense and sparse retrieval, hybrid retrieval with HyDE, learning-based retrieval, subgraph retrieval, and graph database queries.The catalogue also records dynamic retrieval and LLM-driven query reformulation for graph-based systems.
  • Datasets: Knowledge-intensive studies use diverse datasets, including Natural Questions, HotpotQA, FEVER, ASQA, and Wikipedia.The listed resources span question answering, fact verification, and encyclopedic knowledge tasks.

• IIRC [25]

The listed vector-space encoders include several dense embedding models used in the reviewed RAG systems.

  • Vector-space encoders: The encoder list includes sentence-transformers/all-mpnet-base-v2, text-embedding-3-small, text-embedding-ada-002, and thenlper/gte-base.thenlper/gte-small is also listed among the encoders.

• RETRO+ [32]

The reviewed studies document diverse retrieval, chunking, graph, grounding, and generation configurations across datasets and tasks.

  • Datasets: The studies cover datasets such as Natural Questions, HotpotQA, TriviaQA, WebQuestions, WebQSP, and MultimodalQA.These datasets appear alongside open-domain, multi-hop, and multimodal RAG configurations.
  • Adaptive retrieval: Several systems use adaptive or iterative retrieval, including batch grounding, dynamic retrieval, query reformulation, and iterative relation selection.One described grounding process stops once evidence is cited.
  • Generation models: Generation models range from Flan-T5 and GPT variants to Llama, Mistral, Claude, Gemini, and DeepSeek models.The listed configurations include both encoder–decoder and decoder-based language models.
  • Retrieval mechanisms: The catalogue includes dense, sparse, document-level, passage-level, and subgraph retrieval mechanisms.Examples include BM25, Contriever, ColBERTv2, Dense Passage Retriever, and knowledge-graph traversal.
  • Chunking: Chunking strategies include fixed-size chunks, non-overlapping segments, maximum sequence lengths, and grouping short documents into longer units.Reported examples include 256-token maximum sequences and grouping documents from less than 1,000 to around 4,000 tokens.
  • Graph-based RAG: Graph-based approaches retrieve relevant subgraphs, convert graphs into CSV-style representations, and use graph transformers or knowledge-graph traversal.The catalogue also mentions Prize-Collecting Steiner methods and LLM-based graph traversal.

• OK-VQA [74]

The listed open-domain and multimodal RAG configurations combine varied document segmentation, retrieval, encoders, language models, and software-engineering resources.

  • Document preparation: Document processing includes splitting documents into individual passages, fixed-size 512-token chunks, and separate PDF page and table nodes.The catalogue also records passage, node, and corpus-level representations.
  • Datasets: The documented datasets include HotpotQA, Natural Questions, TriviaQA, WebQuestions, WebQSP, and software-engineering resources such as the C Code Summarization Dataset.Multimodal resources include Encyclopedic-VQA, InfoSeek, EntityQuestion, and WitQA.
  • Encoders: The encoder configurations include BERT-based, transformer-based, multimodal T5-and-ViT, CLIP ViT-B/16, and text or image dense encoders.The listed systems therefore include both text and image retrieval components.
  • Retrieval mechanisms: Retrieval methods include BM25, sparse retrieval, dense retrieval, passage-level retrieval with Contriever, and Code Property Graphs.These mechanisms appear across open-domain question answering and software-engineering studies.

• CASIE [75]

CASIE combines code-focused retrieval and generation components, using dense, sparse, lexical, semantic, and hybrid strategies across programming-oriented tasks.

  • Generation back ends include CodeLlama-7B, Repoformer variants, StarCoderBase variants, GPT-3.5-turbo, and transformer decoder architectures.
  • Retrieval targets include code snippets, bug-fix pairs, code diffs, commit messages, and other programming-language data.
  • Chunking strategies include fixed-size sliding windows, punctuation- and paragraph-based heuristics, semantic chunking, and partitioning code files with overlap.
  • CASIE-related methods use dense retrieval with code-oriented encoders, including CodeDiff Encoder, CodeT5, UniXcoder, and SCODE-R.
  • The reviewed systems combine lexical and semantic similarity, including sparse Jaccard token-set, weighted n-gram, and dense embedding-based retrieval.

• RTLLM [85]

RTLLM-related systems apply retrieval-augmented generation to code and clinical settings, combining varied retrievers, language models, chunking schemes, and domain datasets.

  • RTLLM uses code-cushman-001, text-davinci-003, and gpt-3.5-turbo-16k-0613 among its reported language models.
  • Clinical applications cover 14 clinical scenarios, 30 liver-disease guidelines, and 35 preoperative guidelines.
  • The reviewed systems use dense, sparse, and embedding-based retrieval with models including gtr-t5 and Contriever.
  • Reported chunking settings include 1,000 tokens with 100-token overlap and 128 words with 32-word overlap.
  • Applications draw on biomedical instructions, clinical materials, PubMed resources, and full medical reports.

• MEDITRON [91]

MEDITRON-related systems span retrieval-augmented applications in search, legal, disaster, molecular, and biomedical domains, using heterogeneous corpora and retrieval designs.

  • MEDITRON-related evaluations include a 1,000-user benchmark subset, Australian Open Legal QA, CaseHOLD, LEDGAR, FloodBrain, and ClashEval tasks.
  • Retrieval methods include dense retrieval, counterfactual dense retrieval, two independent BERT encoders, Morgan fingerprints, and three-pronged matching.
  • Legal pipelines structure cases into questions, supporting snippets, extracted entities, and answers while using prefiltering to manage token limits.
  • Sources include Bing Search Logs, OpenStreetMap Planet, ReliefWeb flood reports, molecular datasets, and movie-review corpora.
  • Chunking implementations include 1,000-token chunks with 200-token overlap and fixed-size windows of 300 words.

• EN.MC [108]

EN.MC-related systems cover factual, legal, conversational, multimodal, and security applications using diverse benchmarks, retrieval mechanisms, and generative architectures.

  • Evaluations include Factual Recall Questions, False Premise Questions, General Legal Research, Jurisdiction or Time-Specific Research, MultiHop-RAG, Natural Questions, RGB, and Wizard of Wikipedia.
  • Models include Claude-3.5-Sonnet, GPT-3.5-turbo-0125, GPT-4-turbo-2024-04-09, Llama-3-8B-instruct, and BART-Large.
  • Multimodal systems retrieve image-text pairs, uniformly sample video frames or clips, and use CLIP, LXMERT, or temporal deformable convolutional encoders.
  • Conversational systems address emotion-, gender-, and sentiment-specific dialogue using search-engine retrieval, Fusion-in-Decoder, and HybriDial.
  • Security-oriented pipelines split documents into fixed-length contiguous passages and use dense retrieval with models such as Contriever.

• ORQA [127]

The ORQA-related entries list retrieval components, language models, datasets, and preprocessing choices used across reviewed RAG studies.

  • The reviewed entries include dense retrieval components such as Contriever, Dense Passage Retriever, and DPR encoders.
  • Datasets span MS MARCO, Natural Questions, WikiQA, BioChatter Benchmark, and PubMed clinical papers.
  • Listed generation models include Llama, Vicuna, GPT-3.5-turbo, Gemma, and other instruction-tuned systems.
  • The entries also record chunking and retrieval variants, including five-token splitting, dense retrieval, and Weaviate-based retrieval.

• BART [137]

This section surveys RAG retrieval, storage, and chunking choices, emphasizing trade-offs between semantic quality, computational cost, scalability, and generality.

  • Retrieval mechanisms: RAG retrieval mechanisms include sparse, dense, hybrid, query-generating, reclassification, graph, iterative, and specialized domain retrievers.Sparse methods favor efficiency and interpretability, while dense, hybrid, graph, and iterative approaches address semantic recall, multi-hop reasoning, or complex tasks.
  • Vector databases: Vector databases support fast approximate similarity search, but production systems must balance accuracy, latency, memory footprint, synchronization, update throughput, and cost.
  • Vector databases: Adaptive indexing and interoperability across heterogeneous vector stores and embedding formats remain under-explored research gaps.
  • Document chunking: Chunking has evolved from static splits toward semantic, domain-specific, and adaptive strategies that trade scalability and generality against coherence and preprocessing complexity.Static segmentation can fragment meaning, semantic methods add preprocessing cost, domain-specific methods rely on structural priors, and adaptive methods still require validation.

4) Vector encoders:

The section connects encoder and training choices with broader RAG design priorities, including efficiency, security, controllability, and robustness.

  • Vector encoders: RAG encoders span sparse, dense, hybrid, vision-language, graph, and sequence models, reflecting trade-offs among retrieval accuracy, efficiency, and domain adaptability.
  • Training paradigms: Joint end-to-end training aligns retriever and generator components but incurs high computational cost and requires careful loss weighting.
  • Efficiency and results: LongRAG reduces retrieval units 30-fold while increasing answer-recall@1 from 52% to 71% on Natural Questions and answer-recall@2 from 47% to 72% on HotpotQA.It reports EM of 62.7% on NQ and 64.3% on HotpotQA without additional training.
  • Security: Security measures place the retriever at the retrieval interface’s outer boundary through obfuscated identifiers, L2-normalised embeddings, and poison filtering.
  • Prompting and query strategies: Structural prompting and query reformulation can improve relevance, robustness, recall, and latency control, but require governance, fusion, and calibration to avoid brittleness or evidence dilution.
  • Prompting and safety: Prompt policies, fast fusion or reranking, domain schemas, and constrained tool outputs support grounding, precision, reliability, and operational safety.The prompt is also treated as an attack surface requiring sanitisation and output constraints.

3) Hybrid and Specialised Retrievers: No single needlefinder:

The review finds that reliable RAG increasingly depends on heterogeneous retrieval, structure-aware representations, and adaptive retrieval policies rather than a single retriever. These designs improve coverage, faithfulness, efficiency, and provenance, but add latency, governance, and evaluation challenges.

  • Hybrid and specialised retrievers: Hybrid retrieval combines lexical, dense, domain-specific, and graph signals because no single similarity metric surfaces every useful evidence fragment.The review describes hybrid systems as a shift away from single dense retrievers toward configurable combinations of retrieval signals.
  • Hybrid and specialised retrievers: 3 to 6 percentage points gains in top-5 recalls were reported for medical QA when MEDRAG fused BM25 with up to three dense retrievers.The fusion used Reciprocal Rank Fusion and improved recall without requiring the single-index design.
  • Hybrid and specialised retrievers: Hybrid retrieval can improve quality while increasing end-to-end latency by 5-50 ms per query on commodity GPUs.Maintaining multiple indices also requires additional memory and separate refresh cycles, while selective trigger policies can recover benefits at lower cost.
  • Structure-aware and graph-based RAG: Structure-aware RAG represents documents, captions, or code as nodes and edges, making neighbourhoods, paths, and provenance explicit for reasoning.This graph-based approach differs from canonical passage retrieval across indexing, retrieval, and decoding, and extends beyond text to multimodal settings.
  • Structure-aware and graph-based RAG: Graph neighbourhoods reduced prompt length by 40-60% in biomedical QA while supporting faithfulness, error tracing, and inspectable grounding.The review links these gains to denser information carriers and explicit edge or entity provenance.
  • Iterative retrieval and memory: Policy-driven retrieval can preserve most quality gains while reducing unnecessary searches and context use by 15–45%.SELF-RAG, DRAGIN, and TA-ARE use reflection tokens, entropy thresholds, or veto classifiers to trigger retrieval selectively; local revision and fail-safes address iterative pipelines, while memory systems still require privacy and lifecycle governance.

8) Efficiency & Compression—token budgets still matter:

Efficiency-focused RAG research reduces context, indexing, and retrieval costs through compression, overlap, caching, and adaptive retrieval. The review also identifies evaluation and governance gaps that limit dependable comparison and deployment.

  • Compression: 3.53× lower GFLOPs and 1.64× faster CUDA time accompany xRAG’s reduction of retrieved context from roughly 175 tokens to one.xRAG reports task performance comparable to uncompressed RAG.
  • Indexing: 30-fold smaller indices can preserve recall when megabyte-scale PDFs are treated as single long retrieval units.Other approaches refresh FAISS shards asynchronously or expose multiple vector stores and dimensionality choices for speed.
  • Scheduling: RAGCache reportedly halves US dollar cost and reduces 95th-percentile latency by 200 ms through passage reuse, cache warming, and speculative decoding.PipeRAG overlaps CPU retrieval with GPU decoding and cuts roughly a third from end-to-end latency.
  • Adaptive retrieval: Adaptive retrieval varies cadence or triggers retrieval selectively, preserving quality more smoothly than fixed retrieval while respecting latency budgets.RETRO++ adjusts retrieval frequency, while PipeRAG tunes cadence at runtime and other systems precompute dense stores offline.
  • Evaluation and outlook: Benchmarks should report energy and monetary cost alongside EM and BLEU because token budgets, latency, and cost remain deployment constraints.The review also notes that accuracy-focused benchmarks rarely report retrieval counts, GPU minutes, carbon emissions, or memory governance.

4) Other Automated metrics:

Other automated metrics target efficiency, robustness, bias, and domain-specific quality beyond general-purpose scores. Their use remains limited by implementation costs, bespoke data requirements, hardware dependence, and missing standards, motivating hybrid evaluation with human or LLM-based assessment.

  • Metric coverage: Specialised automated metrics cover computational efficiency, robustness, contextual bias, image and code quality, and baseline comparisons.Examples include latency, hallucination and rejection rates, contextual bias, CIDEr, SPICE, edit similarity, Pass@k, and composite comparison scores.
  • Adoption barriers: Specialised metrics appear sporadically because they require bespoke datasets or annotations, depend on hardware and execution environments, and lack universally accepted standards.Composite measures can also obscure which components drive gains and presuppose strong domain baselines.
  • Human evaluation: Human evaluation captures correctness, relevance, fluency, factuality, hallucination, coherence, and user trust that automated measures can miss.It is nevertheless time-intensive, costly, and susceptible to annotator bias, with interannotator agreement frequently below κ = 0.7.
  • Recommendations: Future evaluations should combine automated, semantic, and human or LLM-mediated measures while reporting annotation scales, rater qualifications, agreement, prompts, and model versions transparently.Open guidelines and code can improve reproducibility and broader adoption.
  • LLM-as-judge: G-EVAL uses GPT-4 with a 1–5 rubric for coherence, consistency, and fluency, outperforming traditional overlap metrics in correlation with human judgments.It has also been applied to domain-specific reports such as flood incident summaries.
  • Evaluation frameworks: ARES offers human-aligned assessment but depends on annotated data, whereas RAGAS is operationally efficient yet potentially prompt-sensitive.The review presents their combination with calibrated human oversight as a balanced evaluation strategy.

8) Holistic Evaluation of RAG Benchmarks:

RAG evaluation is shifting from one-size-fits-all scores toward multidimensional and domain-specific benchmark suites. The review also identifies persistent trade-offs involving data quality, retrieval noise, computational cost, architectural interfaces, and reproducibility.

  • Domain-specific evaluation: Specialised benchmarks such as AlphaFin and MIRAGE capture domain nuances that general-purpose evaluation may miss.MIRAGE reports accuracy gains of up to 18% with retrieval augmentation, while also exposing context-overload effects.
  • Retrieval bottlenecks: 56% accuracy with real retrieval versus 89% with ground-truth contexts shows that retrieval remains a bottleneck even for GPT-4.The comparison comes from MultiHop-RAG and isolates the gap between retrieved and ideal evidence.
  • Metric coverage: RAG benchmarks use complementary metrics—including accuracy, rejection rates, ARR, ROUGE, MAP@K, and Sharpe ratios—because no single number captures performance.The review combines quantitative metrics with qualitative assessments to reflect both objective performance and domain-specific nuance.
  • Dataset practices: Approximately 343 datasets have been used to evaluate RAG, but inconsistent segmentation and versioning limit reproducibility and comparability.Open-domain datasets support consistency, whereas specialised datasets reveal domain-specific challenges.
  • Efficiency trade-offs: Dynamic retrieval, extended context, and end-to-end tuning improve relevance but increase latency, memory use, and hardware demands.Full retriever–generator tuning can require days of multi-GPU time and hundreds of gigabytes of RAM.
  • Reliability constraints: Noisy or heterogeneous inputs, incompatible retrieval scores, graph-pruning choices, stale corpora, and undocumented hyperparameters can undermine RAG reliability.Cross-encoder correction adds 2–5 times the latency, while chunking and k choices can shift accuracy–latency curves by double-digit margins.
  • Pipeline interfaces: Splitting retrieval, reranking, and generation into processing stages can make an early misranked passage irreversibly bias the generator.Deep cross-encoders improve ranking fidelity but remain costly, requiring approximate first-pass filters.
  • Memory mechanisms: Memory augmentation can reduce latency by up to 40% without hurting precision, but evidence remains limited to small-scale experiments.Cached errors may also be re-retrieved in later turns, creating staleness and snowballing risks.

APPENDIX

The appendix catalogues the datasets used across the reviewed RAG studies, spanning language, multimodal, biomedical, legal, financial, conversational, adversarial, and other application settings. The table is intended to show their characteristics, origins, and research uses.

  • Dataset overview: Table IV summarises the datasets used in the included RAG studies and records their characteristics and origins.Its purpose is to support analysis of dataset trends and methodologies across the reviewed literature.
  • General and multimodal resources: The appendix includes large pretraining corpora such as C4 and CCNet, alongside image-text resources including COCO, COYO, LAION, and Conceptual Caption.These resources support unsupervised pretraining, vision-language modelling, image captioning, and related tasks.
  • Question answering: Question-answering resources span open-domain, commonsense, biomedical, legal, financial, conversational, multimodal, and current-events settings.Examples include IIRC, CommonsenseQA, Biomedical MCQ, CaseHOLD, ConvFinQA, QAConv, MultimodalQA, and RealTimeQA.
  • Biomedical and clinical data: Biomedical and clinical datasets cover literature, guidelines, adverse drug effects, drug dosage, liver disease, Alzheimer’s knowledge, and chest X-ray applications.The listed resources support retrieval, diagnosis, clinical decision support, biomedical extraction, and health-related question answering.
  • Other task families: The appendix also covers classification, entity linking, relation extraction, code completion, dialogue, legal reasoning, fact checking, and adversarial robustness.Examples include AGNews, AIDA CoNLL-YAGO, ChemProt, CrossCodeEval, DailyDialog, CREAK, AdvBench, and ANLI.
  • Application-specific resources: Specialised collections include course materials, flood-event listings, and driving episodes for education, disaster-response interfaces, and safe-driving planning.These datasets illustrate the breadth of deployment-oriented RAG applications represented in the review.
Loading 2508.06401v3…