A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions

Agada Joseph Oche; Ademola Glory Folashade; Tirthankar Ghosal; Arpan Biswas

Source-linked AI summary

A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions

Agada Joseph Oche, Ademola Glory Folashade, Tirthankar Ghosal, Arpan Biswas

arXiv:2507.18910v1Jul 25, 2025cs.CLcs.LG

TL;DR

RAG addresses the difficulty of keeping language-model knowledge current and verifiable. This paper systematically reviews its architectures, milestones, applications, and challenges, concluding that RAG has become a foundational approach for grounding AI systems while requiring further work on efficiency, fusion, and evaluation.

Problem
Language models struggle to access up-to-date information and provide verifiable sources, motivating retrieval augmentation for knowledge-intensive tasks.
Method
The paper conducts a systematic review of RAG’s technical components, chronological development, applications, implementation challenges, and proposed solutions.
Results
RAG evolved from early retrieval-and-generation systems into architectures with demonstrated benefits across open-domain question answering, enterprise applications, and AI-powered search.
Takeaways & Limitations
The review identifies retrieval efficiency, document fusion, and robust evaluation metrics as priorities for advancing RAG systems.
Takeaways & Limitations
Future RAG systems still face challenges involving ultra-fast retrieval at trillion-token scale, faithfulness verification for multi-hop reasoning, and energy-efficient multimodal indexing.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) represents a major advancement in natural language processing (NLP), combining large language models (LLMs) with information retrieval systems to enhance factual grounding, accuracy, and contextual relevance. This paper presents a comprehensive systematic review of RAG, tracing its evolution from early developments in open domain question answering to recent state-of-the-art implementations across diverse applications. The review begins by outlining the motivations behind RAG, particularly its ability to mitigate hallucinations and outdated knowledge in parametric models. Core technical components-retrieval mechanisms, sequence-to-sequence generation models, and fusion strategies are examined in detail. A year-by-year analysis highlights key milestones and research trends, providing insight into RAG's rapid growth. The paper further explores the deployment of RAG in enterprise systems, addressing practical challenges related to retrieval of proprietary data, security, and scalability. A comparative evaluation of RAG implementations is conducted, benchmarking performance on retrieval accuracy, generation fluency, latency, and computational efficiency. Persistent challenges such as retrieval quality, privacy concerns, and integration overhead are critically assessed. Finally, the review highlights emerging solutions, including hybrid retrieval approaches, privacy-preserving techniques, optimized fusion strategies, and agentic RAG architectures. These innovations point toward a future of more reliable, efficient, and context-aware knowledge-intensive NLP systems.

1 Introduction

This review traces RAG’s rapid evolution from foundational retrieval-and-generation methods to broad research and industry adoption, with particular attention to proprietary data, system capabilities, limitations, and future work.

Background and motivation: RAG combines retrieval with generative models to ground outputs in external documents and address limitations involving outdated or unverifiable parametric knowledge.The review presents this combination as the response to challenges in accessing current information and providing verifiable sources.
Field development and adoption: RAG has expanded from open-domain question answering into search, customer support, enterprise question answering, and systems that provide supporting evidence.The paper describes substantial growth in both academic literature and commercial adoption.
Scope and objectives: The review examines RAG’s yearly technical milestones, deployment in enterprise settings, and integration with proprietary or private data sources.It specifically considers efficient retrieval on private corpora, sensitive information handling, security, and privacy gaps.
Review objectives: The review evaluates current capabilities and limitations while identifying objectives for future research and development.Its stated objectives include charting RAG’s evolution, assessing its capabilities and limitations, and identifying remaining research areas.
Review coverage: The paper surveys RAG’s core components and architectures, including retrievers, indexes, sequence-to-sequence generators, and fusion strategies.It also organizes the field’s development chronologically and synthesizes technical progress across the paper’s sections.

2 Methodology

The review uses a systematic process to identify, screen, extract, and synthesize RAG research from academic and industry sources published between 2017 and mid-2025.

Review design: The methodology comprises search strategy, inclusion and exclusion criteria, and data extraction and synthesis.These steps were designed to analyze and consolidate findings across the surveyed RAG literature.
Search strategy: Searches covered ACL Anthology, IEEE Xplore, ACM Digital Library, and Google Scholar, using keywords spanning RAG, retrieval methods, proprietary data, and web search.The resulting pool included journal articles, conference papers, technical reports, and white papers.
Corpus scope: The review included academic and industry-focused works published from 2017 through mid-2025, including early retrieve-and-generate approaches.Its scope was not restricted to peer-reviewed journal and conference articles.
Screening criteria: Screening removed duplicates and excluded works focused solely on retrieval or generation without explicit retrieval-augmented integration.Works with peripheral RAG discussion or without verifiable methodology were also excluded.
Corpus finalization: The screening process produced a finalized corpus of documents considered pertinent to the state of the art in RAG.This finalized set followed the stated search, eligibility, and exclusion procedures.
Eligibility criteria: Included studies were required to address RAG or closely related integrated baselines and knowledge-intensive tasks, with preprints and reputable industrial sources permitted.Only English-language texts were included, and non-English papers were excluded for feasibility reasons.
Data extraction and synthesis: Extracted information covered bibliographic details, retrieval methods, generator architectures, tasks, datasets, performance, latency, and implementation challenges.The collected data were stored centrally for cross-study comparison and chronological synthesis.
Reliability procedures: Review disagreements were resolved through discussion or consultation with a third reviewer to support consistent criteria application and reliable data extraction.The resulting dataset supported analyses of yearly progress, enterprise applications, and proposed solutions.

3 Foundations of RAG

RAG foundations combine neural retrieval with sequence-to-sequence generation, using external documents as non-parametric memory to ground knowledge-intensive responses. Its pipeline retrieves and reranks evidence, fuses multiple documents, and generates an answer that may include provenance.

Definition and Key Concepts: RAG combines a neural text retriever with a text generator, augmenting a sequence-to-sequence model with external corpus access.The retriever selects a small subset Z from corpus C, with K ≪ |C|, and the generator conditions on the query and retrieved documents.
Definition and Key Concepts: RAG grounds generation in retrieved documents, improving factual specificity and enabling knowledge updates by modifying the document index rather than retraining the generator.Retrieved text supplies current evidence, addressing fixed knowledge in models whose training data ends at a cutoff date.
Technical Components of RAG: A typical pipeline chunks documents, embeds them for vector search, retrieves candidate passages, optionally reranks them, and generates from the refined evidence.Reranking uses query-document joint encoding to refine coarse embedding-similarity results before generation.
Technical Components of RAG: RAG can fuse retrieved evidence through marginalization, direct concatenation, or learned weighting over documents during decoding.Marginalization models retrieved documents as latent variables, concatenation exposes all retrieved texts to the generator, and weighting assigns differing importance to passages.
Technical Components of RAG: Joint training adjusts retrieval and generation through the output likelihood, indirectly increasing probabilities for documents that help produce the correct answer.The top-K approximation supplies a tractable training signal, and initializing retrieval with DPR reportedly outperforms training the retriever from scratch.
Historical Context: Introduced in 2020, RAG generalized retrieval-augmented architectures beyond question answering and outperformed parametric-only models and earlier retrieve-and-read pipelines on knowledge-intensive tasks.The architecture was also associated with provenance outputs and fast retrieval and generation using efficient vector databases and modern transformers.

4 Year-by-Year Progress in RAG

RAG progressed from dense retrieval and jointly trained retriever–generator systems in 2020 to broader, more efficient, and more robust applications across tasks and domains. By 2022, retrieval, multi-document fusion, few-shot learning, dialogue, and evidence grounding had become central directions, while latency, reasoning, and faithfulness remained open challenges.

2020: DPR improved top 20 recall by 9–19 pp over BM25, establishing dense neural retrieval as a foundation for subsequent RAG systems.DPR used dual BERT encoders to embed questions and passages into a shared space.
2020: RAG and FiD jointly advanced retrieval–generation integration, achieving state-of-the-art question-answering performance by conditioning generation on multiple retrieved passages.RAG used a BART generator with jointly trained DPR retrieval, while FiD used a T5 decoder attending over dozens of passages.
2020: By year-end 2020, relatively small RAG systems surpassed 11-billion-parameter closed-book language models, while retrieval latency, multi-hop reasoning, and faithfulness evaluation remained unresolved.The review presents this as evidence for the efficiency of combining parametric and non-parametric memory.
2021: In 2021, FiD-style multi-document fusion and improved training strategies produced further open-domain QA gains while expanding retrieval-based methods to fact-checking, dialogue, and entity-rich tasks.The reviewed works included EM-like training improvements and architectures that combine evidence from many passages.
2021: KILT unified 11 knowledge-intensive tasks, showing that shared retrievers and generative models performed strongly across QA, fact-checking, dialogue, and slot filling.The benchmark helped standardize evaluation and consolidate RAG progress by the end of 2021.
2022: RAG expanded into dialogue, fact-checking, summarization, and few-shot learning, with retrieval improving factual consistency, evidence-based verdict generation, and performance under limited supervision.The review also describes live-search dialogue systems, unsupervised dense retrieval, and retrieval-based gains on FEVER and summarization.
2023: By 2023, hybrid strategies and retrieval-or-context selection positioned RAG as a practical approach for up-to-date, interpretable language systems despite continuing efficiency considerations.The review notes that RAG remained more cost-efficient and better at exposing citations than relying solely on large context windows.
2022: ATLAS and RETRO demonstrated retrieval’s scaling and few-shot efficiency, including 42.4% accuracy for 11B ATLAS on Natural Questions with 64 examples and a 3% advantage over a 540B closed-book model.ATLAS also achieved new results on Natural Questions and TriviaQA, while RETRO showed that retrieved text could substitute for much larger parameter counts.

5 RAG for Proprietary Data - Industry Implementation

RAG is deployed with proprietary data through secure retrieval pipelines, enabling domain-specific access while raising privacy, performance, and integration concerns. Industry implementations emphasize accuracy, timeliness, citation, efficiency, and compliance.

Secure proprietary-data retrieval: Organizations use secure pipelines, often with on-premise or VPC-hosted vector databases, to retrieve proprietary documents without sending embeddings beyond the corporate firewall.Access-control layers can filter documents according to user permissions.
Industry applications: Enterprise case studies apply RAG to information retrieval, agriculture, mortgage processing, data organization, source citation, and domain-specific AI applications.Examples include PGA Tour, Bayer, Rocket Companies, Shorenstein Properties, Cohere, NVIDIA, and IBM.
Reliability and currency: RAG supports current, domain-specific responses and can reduce hallucinations by connecting models to proprietary or authoritative information and enabling source citation.The cited implementations emphasize more accurate, relevant, accessible, or verifiable outputs.
Open concerns: Remaining enterprise concerns include data privacy, intellectual property, computational overhead, and the non-trivial performance cost of secure retrieval techniques.Secure enclaves and homomorphic encryption are identified as possible protections, while fine-tuning may memorize private text.
Operational trade-offs: Industry systems balance retrieval accuracy against latency, with accuracy-oriented models incurring higher latency while applications such as NVIDIA RAG target sub-second responses.Enterprise systems also prioritize database integration and data privacy compliance over benchmark performance alone.
Scaling and freshness: RAG extends beyond Wikipedia-scale retrieval to proprietary databases and dynamic web sources, providing more up-to-date responses than static knowledge settings.WebGPT and IBM Watsonx RAG are cited as examples of dynamic knowledge-base retrieval.

6 Evaluation of RAG Systems

RAG evaluation must jointly assess retrieval, generation, factual grounding, efficiency, scalability, and benchmark coverage. The review highlights RAGAS and architectural choices such as chunking, embeddings, and re-ranking as tools for evaluating and improving grounded performance.

Evaluation dimensions: RAG evaluation covers retrieval accuracy, answer quality, factuality, latency, scalability, memory, and computational requirements.This framework evaluates both the retrieval pipeline and the generative model.
Metrics: Recall@k, Mean Reciprocal Rank, and Mean Average Precision measure ranked retrieval quality, while Exact Match, F1, BLEU, and ROUGE assess generated outputs.Faithfulness and hallucination rate evaluate factual consistency with retrieved sources.
Benchmarks: Benchmarks span open-domain QA, fact-checking, multi-task knowledge access, zero-shot retrieval, multi-turn reasoning, and unified retrieval-generation-support evaluation.Examples include Natural Questions, FEVER, KILT, BEIR, MTRAG, and the TREC RAG Track.
RAGAS: RAGAS evaluates factuality, grounding, and attribution by measuring how well generated content is supported by retrieved evidence.Its design targets unsupported statements and hallucinations rather than only surface-level linguistic overlap.
Architectural factors: Chunking, embedding, and re-ranking directly affect retrieval and generation performance across tasks and benchmarks.Semantic chunking improves retrieval precision and answer accuracy, while embedding and re-ranking choices influence Recall@k, EM, and F1.
Interpretation: Retrieval fidelity correlates strongly with factual answer quality, making optimization of retrieval stages important for competitive QA and generation results.The review treats retrieval design as a central determinant of downstream answer quality.

7 Challenges of RAG

This section examines RAG challenges, their manifestations in applications, and existing solutions and future directions.

The section discusses RAG challenges, application manifestations, existing solutions, and the way forward.
Its scope includes both practical cases and proposed responses to the identified challenges.
The section also considers future directions for addressing RAG challenges.

7.1 Technical Challenges in RAG

Technical RAG challenges center on retrieval quality, computational overhead, evidence integration, conflicting information, context limits, and the complexity of task-specific adaptation.

Retrieval quality and relevance: Poor retrieval can produce incorrect or irrelevant answers because RAG answer accuracy depends strongly on document quality.BM25 may miss relevant texts or return noise, while neural retrievers face vocabulary mismatch, ambiguity, and domain terminology issues.
Retrieval quality and relevance: Choosing the number of retrieved passages is difficult: too few limit evidence, whereas too many introduce irrelevant context and increase complexity and latency.Ranking and iterative query reformulation can improve precision but add overhead.
Latency and efficiency: RAG increases computational complexity and latency through retrieval overhead, vector searches, and expanded context processing.Approximate-nearest-neighbor indices, caching, distillation, and lightweight retrievers can reduce latency at an accuracy cost.
Latency and efficiency: Retrieval can reduce model-size requirements but shifts complexity toward maintaining external knowledge bases and infrastructure.This trade-off is illustrated by retrieval-augmented systems matching larger models without retrieval.
Evidence integration: Models may ignore retrieved evidence when it conflicts with internal knowledge, while contradictory documents can create confusion or confirmation bias.Transformer context limits may force truncation or summarization that omits essential information.
Evidence integration: Fine-tuning for retrieval-augmented tasks can improve evidence integration but adds complexity, especially for non-differentiable or API-based models.

7.2 System-Level Challenges

System-level deployment challenges span scale, knowledge freshness, hallucination control, and component coordination. These constraints require substantial infrastructure and ongoing maintenance to preserve reliable operation.

Scalability: Millions or billions of documents require efficient indexing, distributed computing, caching, multi-tier retrieval, and cost management.These measures are presented as essential for large deployments such as web search engines.
Knowledge freshness: Real-time domains require incremental data pipelines, possible document re-encoding, and synchronized retrieval indices to maintain current knowledge.Update delays or inconsistencies can produce outdated or contradictory answers.
Hallucination control: RAG reduces hallucinations but cannot eliminate fabricated or misattributed information when retrieved context is incomplete or contradictory.Legal studies still report errors, including invented citations, motivating source verification and confidence calibration.
System integration: Multiple retrievers, rerankers, indexes, and LLMs increase complexity and potential failure points across updates, access controls, prompts, and dialogue management.Evaluation must therefore cover end-to-end performance, retrieval quality, and faithfulness to evidence.

7.3 Ethical and Societal Concerns

Ethical and societal concerns center on bias, misinformation, privacy, and accountability. Responsible deployment requires trustworthy sources, security controls, transparent attribution, and continuing oversight.

Bias and fairness: RAG can inherit or amplify biases from language models and retrieved data, particularly in historically biased legal or medical domains.Suggested mitigations include inclusive curation, diversified retrieval, and prompts encouraging balanced responses.
Information reliability: Misinformation or outdated evidence can be synthesized into false or misleading answers, making source quality, citation transparency, and verification essential safeguards.Even credible sources may become outdated or be misapplied contextually.
Privacy and security: Sensitive-data RAG systems face risks of confidential-information exposure and prompt injection, requiring access controls, regulatory compliance, and security testing.The cited examples include GDPR and HIPAA obligations.
Accountability and transparency: Sourced retrieval supports traceability, but inaccurate citations or improper synthesis can mislead users in critical decisions.Ethical deployment therefore requires attribution, explanations, expectation management, oversight, and user correction mechanisms.

7.4 Application Domains and Case Studies

RAG applications differ by domain: legal systems emphasize precision and human oversight, medical systems require authoritative evidence and rigorous validation, and customer support systems depend on current knowledge bases and effective interaction management.

Legal applications: Legal RAG must handle jurisdiction-specific retrieval, long documents, standardized citations, novel legal reasoning, and continuously changing law.Current systems primarily assist with information rather than replacing nuanced legal judgment.
Legal applications: Legal RAG can reduce research time and broaden access to legal information, but requires explicit limitations and human oversight.The paper distinguishes assistance to lawyers from unauthorized legal practice.
Medical applications: Medical RAG requires authoritative source curation, dynamic evidence updates, privacy protections, bias mitigation, and regulatory positioning as decision support.Open-ended diagnostic reasoning remains challenging compared with factual inquiries.
Medical applications: Medical advice systems require explicit disclaimers and positioning as decision-support tools rather than autonomous advisors under potential regulatory oversight.The medical domain also emphasizes factual accuracy, ethical transparency, and validation comparable to clinical trials.
Customer support: Customer-support RAG depends on flexible query interpretation, multi-turn context, secure personalization, empathetic interaction, escalation, and useful evaluation metrics.Support success is assessed through resolution rates and customer satisfaction as well as factual accuracy.
Customer support: Customer-support utility is directly tied to the quality and currency of internal documentation, while user interactions can improve both system performance and documentation quality.This creates a feedback connection between deployment and continual knowledge-base improvement.

7.5 Existing and Potential Solutions

Proposed solutions target retrieval quality, latency, integration, hallucination, scalability, freshness, bias, and misinformation. The approaches combine improved retrieval and infrastructure with verification and source-governance mechanisms.

Retrieval quality: Domain-adaptive training, advanced encoders, query reformulation, and reranking improve retrieval relevance, with reranking adding computation.Iterative retrieval and chain-of-thought reasoning are identified as future directions.
Efficiency: HNSW search structures, multi-level caches, approximate embedding caches, and adaptive retrieval reduce retrieval latency by reusing or simplifying searches.Adaptive methods balance retrieval complexity according to query difficulty.
Retrieval-generation integration: Joint retriever-generator training and architectural mechanisms such as RETRO cross-attention improve integration between retrieval and generation.Selective retrieval and reinforcement learning are proposed for optimizing when external knowledge enters generation.
Factual reliability: Penalizing ungrounded assertions, iterative retrieval, self-checking, source citation, and external verification target factual hallucinations.Self-RAG critiques and revises outputs against retrieval results.
Scalability: Advanced indexing, distributed retrieval, approximate nearest-neighbor methods, selective indexing, caching, and parallel retrieval support massive knowledge repositories.These approaches combine corpus curation with infrastructure improvements to address scale.
Freshness and trustworthiness: Incremental updates, selective retrieval, live search APIs, hybrid retrieval, user feedback, balanced sources, diversification, and credibility checks address freshness, bias, and misinformation.Continuous monitoring and cross-verification further strengthen robustness against adversarial information.

8 Discussion and Future Direction

RAG systems combine retrieval and generation to improve factual, current, and domain-grounded responses, while expanding into enterprise, multimodal, agentic, structured, and streaming settings. Persistent concerns include retrieval failures, conflicting evidence, latency, privacy, security, and scalability.

Synthesis of Findings: RAG combines dynamic external-knowledge retrieval with generation, outperforming purely parametric models on knowledge-intensive NLP tasks.The reviewed paradigm evolved from sparse lexical retrieval toward differentiable dense retrievers and evidence-conditioned generation.
Challenges: Retrieval failures can produce incorrect outputs, while fusion of multiple passages may introduce contradictory evidence or increased latency.Improving retrieval robustness and document-fusion mechanisms remains a critical research direction.
Enterprise Applications: Enterprise RAG accesses proprietary data without embedding sensitive information in model parameters, while independently refreshable retrieval supports current knowledge.Deployments still require strict access controls, authentication, and privacy-preserving techniques such as encrypted indices or federated retrieval.
Future Directions: Emerging architectures extend RAG through multi-hop retrieval, multimodal inputs, autonomous agents, structured knowledge, and real-time streaming retrieval.These directions target multi-document reasoning, richer context, dynamic planning, semantic relationships, and timely updates.
Applications: RAG supports enterprise search, real-time assistants, fact-checking, conversational agents, and low-resource question answering.Reported applications emphasize private-knowledge synthesis, current information, evidence retrieval, informed responses, and viable QA under limited resources.

9 Conclusion

The review presents RAG as a bridge between retrieval and generation that improves factuality and adaptability across open-domain question answering, enterprise systems, and AI-powered search. It concludes that retrieval quality, evidence conflicts, scalability, and privacy remain central challenges requiring continued research.

Conclusion: RAG bridges retrieval and generation, supporting improved factuality and adaptability across open-domain QA, enterprise applications, and AI-powered search.The review characterizes its development from early retrieve-and-read systems to architectures with neural retrievers and sequence-to-sequence generators.
Conclusion: High-quality retrieval, conflicting evidence, and scaling retrieval across large knowledge bases remain unresolved challenges.These issues are identified as requiring further research.
Conclusion: Proprietary RAG deployments require privacy-preserving retrieval strategies because of security concerns.Security is a continuing constraint when systems operate over sensitive organizational knowledge.
Future Directions: Future work should optimize retrieval efficiency, refine document fusion, and develop robust evaluation metrics for RAG.The review links these directions to the continued convergence of information retrieval and AI-powered text generation.