PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K. Qiu, Lili Qiu

arXiv:2409.14924v1cs.CLcs.AI

TL;DR

Building data-augmented LLM applications remains challenging across specialized fields, from retrieving relevant evidence to handling complex reasoning. This survey classifies query types and synthesizes approaches for them; it concludes that different query types call for different retrieval methods and that practical systems may combine methods.

  • Problem

    Developers face substantial effort and challenges building data-augmented LLM applications, including retrieval, data interpretation, and complex reasoning in specialized fields.

    Source: §1 · p. 2

  • Method

    The survey defines four query levels, summarizes their challenges and techniques, and discusses context, small-model, and fine-tuning routes for integrating external data.

    Source: §1 · p. 2, §7 · p. 17

  • Results

    For explicit facts, basic RAG is preferred; implicit facts favor iterative RAG or graph/tree RAG that retrieve and connect multiple related facts.

    Source: §7 · p. 17

  • Takeaways & Limitations

    Because practical applications combine query types, developers may need routing pipelines that integrate multiple methods.

    Source: §7 · p. 18

    Fine-tuning with new factual data can deteriorate accurate-response performance, increase hallucinations, and leave learned facts vulnerable to rephrasing.

    Source: §4.6 · p. 11

Abstract

from arXiv · show

Large language models (LLMs) augmented with external data have demonstrated remarkable capabilities in completing real-world tasks. Techniques for integrating external data into LLMs, such as Retrieval-Augmented Generation (RAG) and fine-tuning, are gaining increasing attention and widespread application. Nonetheless, the effective deployment of data-augmented LLMs across various specialized fields presents substantial challenges. These challenges encompass a wide range of issues, from retrieving relevant data and accurately interpreting user intent to fully harnessing the reasoning capabilities of LLMs for complex tasks. We believe that there is no one-size-fits-all solution for data-augmented LLM applications. In practice, underperformance often arises from a failure to correctly identify the core focus of a task or because the task inherently requires a blend of multiple capabilities that must be disentangled for better resolution. In this survey, we propose a RAG task categorization method, classifying user queries into four levels based on the type of external data required and primary focus of the task: explicit fact queries, implicit fact queries, interpretable rationale queries, and hidden rationale queries. We define these levels of queries, provide relevant datasets, and summarize the key challenges and most effective techniques for addressing these challenges. Finally, we discuss three main forms of integrating external data into LLMs: context, small model, and fine-tuning, highlighting their respective strengths, limitations, and the types of problems they are suited to solve. This work aims to help readers thoroughly understand and decompose the data requirements and key bottlenecks in building LLM applications, offering solutions to the different challenges and serving as a guide to systematically developing such applications.

1 Introduction

Data-augmented LLMs can address gaps in generic models, but building effective applications in specialized fields remains difficult. The survey proposes distinguishing query types and their challenges to support systematic development.

1 Introduction

  • Generic LLMs may lack timely, comprehensive, or proprietary domain knowledge, while data augmentation can support more detailed answers and customization.The passage also describes potential benefits including domain-expert alignment, reduced hallucination, and improved controllability and explainability.Source: §1 · pp. 1–2
  • Developers invest substantial human effort to meet expectations for data-augmented LLM applications, including high question-answering success rates.Studies report challenges in specialized domains such as law, healthcare, and manufacturing [1] [2] [4] [5].Source: §1 · p. 2
  • Challenges range from data-pipeline construction to interpreting varied domain data, handling long-distance dependencies, and meeting interpretability and consistency demands.Finance may require high-dimensional time series, healthcare may rely on images or records, and legal or mathematical tasks can involve long-distance dependencies.Source: §1 · p. 2
  • The survey identifies query levels and their distinct challenges because prior surveys often focus on only one level or technology topic [8] [9] [10] [11] [12] [13].It aims to provide a broad view and a handbook for systematic application development.Source: §1 · p. 2

2 Problem Definition

A data-augmented application maps a query to an answer using given external data. The section classifies queries by the data interaction and reasoning they require, from direct fact retrieval to applying or inferring rationales.

2 Problem Definition

  • A data-augmented application maps user input Q to expected answer A based on given data D.The paper frames this as the application task f establishing a mapping from Q to A based on D.Source: §2 · p. 2

2.1 Stratification of Queries

  • Level-1 explicit-fact queries locate information directly present in the data and require no additional reasoning.The example asks where the 2024 Summer Olympics will be held.Source: §2.1 · p. 3
  • Level-2 implicit-fact queries require common-sense reasoning or simple deductions, often by combining information spread across segments.The Canberra example combines the city’s country with information about that country’s current majority party.Source: §2.1 · p. 3
  • Level-3 interpretable-rationale queries apply domain-specific reasoning that is explicitly available in external resources, such as regulatory guidance or diagnostic criteria.Examples include following a customer-support workflow or evaluating a drug application against FDA guidance.Source: §2.1 · p. 3
  • Level-4 hidden-rationale queries infer unrecorded strategies from patterns and outcomes in external data, such as incident or debugging histories.The LLM must extract tacit decision-making and synthesize it into responses reflecting experienced professionals’ problem-solving.Source: §2.1 · pp. 3–4
  • The four query levels form a complexity gradient: the first two focus on factual retrieval, while the latter two require understanding or applying rationales.The classification distinguishes directly stated facts, implicit facts, interpretable rationales, and hidden rationales.Source: §2.1 · pp. 2–4

3.1 Overview

Explicit fact queries depend directly on specific external data segments, with answers often stated in documents and requiring little reasoning. The section formalizes the needed segment subset and the retrieval-and-generation conditions for this query type.

3.1 Overview

  • Explicit fact queries answer questions by retrieving directly accessible facts from specific data segments, with minimal reasoning or inference beyond those segments.Examples include asking what method a paper used or a company’s AI strategy, given relevant documents.Source: §3.1 · p. 4, §3.1.2 · p. 6

3.1.1 Data Dependency

  • Data dependency is the subset of dataset segments indispensable for answering a query, since not every segment is required.The dataset may be segmented into documents or shorter, focused segments; the footnote also allows segments as large as a whole document or larger.Source: §3.1.1 · p. 5, §3.1.2 · p. 5

3.1.2 Definition

  • An explicit fact query requires retrieval to identify segments closely matching the query’s necessary data, then a generator to answer from the retrieved information alone.The generated answer should equal or approximate the answer supported by those directly accessible facts.Source: §3.1.2 · pp. 5–6

3.2 Challenges and Solutions

For explicit fact queries, accurate responses depend primarily on retrieving the right data, yet building robust RAG systems remains challenging. The section identifies processing, retrieval, and evaluation as key difficulties.

3.2 Challenges and Solutions

  • Explicit fact queries depend primarily on correct data retrieval for accurate responses, but robust, high-quality RAG systems remain difficult to construct.The survey also points to literature and tools addressing these challenges and discusses technical alternatives beyond RAG.Source: §3.2 · p. 6
  • Unstructured external data, multimodal content, and chunking can make it difficult to preserve the original context and meaning during data processing.Relevant source content may include tables, images, and videos.Source: §3.2 · p. 6
  • Retrieving relevant segments from large, unstructured datasets can be computationally intensive and error-prone, demanding efficient and accurate retrieval mechanisms.Source: §3.2 · p. 6
  • Component-level RAG evaluation is complex and requires robust metrics for assessing both retrieval quality and response generation.Source: §3.2 · p. 6

3.3 Retrieval-augmented Generation (RAG)

RAG typically constructs data indexes, retrieves information, and generates answers. The section surveys enhancements to processing, retrieval, and response generation, including approaches for multimodal data, alignment, and noisy retrieved context.

3.3.1 Data Processing Enhancement

  • Multimodal content can be converted into text or descriptions for LLM processing, or represented with multimodal embeddings used as soft prompts.Table-to-Text methods and visual-content descriptions [35] [36] exemplify textual conversion; multimodal embedding approaches include [37] [38] [39].Source: §3.3.1 · p. 6
  • Chunking strategies trade semantic coherence against noise: larger chunks preserve more context but tend to contain more noise, while other methods adapt chunk size or segment by document structure.Strategies include fixed-size, recursive, sliding-window, paragraph-based, and semantic chunking; query-aware granularity methods [43, 44], refinement into complete smaller segments [45], and vision-based structural segmentation [46] are also described.Source: §3.3.1 · p. 6

3.3.2 Data Retrieval Enhancement

  • Retrieval enhancement spans indexing and query-document alignment: sparse indexing uses words, dense indexing uses vectors, and hybrid indexing combines both.The survey also describes retrieval as involving indexing, query processing, matching, reranking, and evaluation.Source: §3.3.2 · p. 7
  • Sparse methods such as TF-IDF and BM25 [47, 48] can miss synonyms, while dense retrieval maps text and queries into vector spaces and includes encoder-based and LLM-based approaches.LLM feedback can guide retriever training [61] [62] [63]; LLM2vec [65] and Llama2Vec are examples of LLM-based dense retrieval methods.Source: §3.3.2 · p. 7
  • Hybrid retrieval can combine sparse and dense indexing; other approaches determine a query’s knowledge domain or fine-tune an LLM to generate indices and text segments.Feng et al. (2023) describe domain-first dense retrieval [68]; studies also blend indexing methods [69] [70] [71], while Tang et al. (2024) fine-tune LLMs for indexing and retrieval [72].Source: §3.3.2 · p. 7
  • Query-document alignment can use a shared encoding space, synthetic answers to retrieve documents, or synthetic questions to map documents into query space.Query rewriting can address vague or inaccurate terminology [74]; HyDE is a document-domain alignment example [75], while query-domain alignment is described in [76]. Figure 3 depicts the three alignment types.Source: §3.3.2 · p. 7, Figure 3 · p. 8
  • After retrieval, systems rerank or filter text blocks using relevance scores, other ranking criteria, LLM assessments, or fact-verification models; iterative retrieval can address omissions from a single attempt.Recursive retrieval can break ambiguous questions into disambiguated ones [82], while SEATER iteratively recalls nodes in a hierarchical tree [83].Source: §3.3.2 · pp. 7–8

3.3.3 Response Generation Enhancement

  • Response generation must assess whether retrieved information is sufficient, handle conflicts with model priors, and avoid being misled by irrelevant or erroneous context.Fine-tuning or pretraining on designed data can help models recognize noisy contexts [87] [88] [89]; RAAT [89] reports progressively greater detrimental effects from irrelevant, relevant, and counterfactual retrieval noise, and some work jointly trains retriever and generator [90] [91] [92].Source: §3.3.3 · p. 8

4 Implicit Fact Queries (L2)

Implicit fact queries require combining information that is not stated in any single passage, often by decomposing the query into simpler retrieval steps. The section covers iterative retrieval, graph and SQL methods, and the importance of distinguishing implicit from explicit fact queries.

4.1 Overview

  • Implicit fact queries combine information across documents, often through basic inference, and may require decomposing the original question into simpler explicit fact queries.A response generator aggregates answers to those subqueries to derive an answer not explicitly stated in the data.Source: §4.1 · p. 9

4.2 Challenges and Solutions

  • Adaptive retrieval volumes and coordination between reasoning and retrieval are central challenges; fixed retrieval counts can yield noise or insufficient information.Methods named include iterative RAG, graph/tree RAG, and RAG with SQL.Source: §4.2 · p. 9

4.3 Iterative RAG

  • Iterative RAG methods use plans or information-gap filling to refine retrieval across steps, with some approaches deciding when to search and when to answer.ReAct, IRCoT, and RAT guide retrieval targets; ITRG and FLARE revisit incomplete answers, while Self-RAG learns when to search or stop.Source: §4.3 · pp. 9–10

4.4 Graph/ Tree Question Answering

  • Graph and tree methods organize entities or text chunks into relational structures, supporting retrieval and synthesis across multiple references.Approaches include knowledge-graph exploration and chunk graphs or hierarchies that retrieve or summarize relevant subgraphs.Source: §4.4 · p. 10

4.5 Natural Language to SQL Queries

  • For structured data, natural-language-to-SQL methods translate queries into database operations that provide external information for LLM responses.The section describes text-to-SQL tools as a way to retrieve structured database content.Source: §4.5 · pp. 10–11

4.6 Discussion on Fact Queries

  • Fine-tuning new facts may impair accuracy, increase hallucinations, or leave knowledge vulnerable to paraphrase; query-level misclassification can also lead to unhelpful or insufficient retrieval.Implicit fact queries often need dynamically integrated information, whereas explicit fact queries generally need only one data snippet.Source: §4.6 · p. 11

5 Interpretable Rationale Queries (L3)

Interpretable rationale queries require using domain-specific reasoning that is explicitly available in external data, such as text or structured instructions. The section reviews prompt optimization and reasoning-chain approaches for integrating and applying those rationales.

5.1 Overview

  • Interpretable rationale queries use explicit explanations or decision paths, expressed as plain texts, structured instructions, workflows, decision trees, or pseudocode.The section distinguishes Text-Conditioned Moore and Mealy Machines as forms of structured instructions.Source: §5.1 · p. 11

5.2 Challenges and Solutions

  • Key challenges include the time and computational cost of tailoring prompts and limited transparency into how prompts affect LLM responses.Restricted access to model parameters complicates understanding and verifying response interpretability.Source: §5.2 · p. 12

5.3 Prompt Tuning

  • Prompt-tuning methods optimize instructions through reinforcement-learning feedback, prompt edits, or LLM-generated candidates and evaluations.Examples include TEMPERA, Rlprompt, Directional Stimulus Prompting, GrIPS, OPRO, and Reflexion.Source: §5.3 · pp. 12–13

5.4 CoT Prompting

  • CoT approaches include manually designed reasoning prompts, automated prompt generation, and agent workflows that incorporate feedback and evaluate reasoning-chain quality.Automate-CoT uses a minimally labeled dataset to generate rational chains and select prompt combinations.Source: §5.4 · p. 13

6 Hidden Rationale Queries (L4)

Hidden rationale queries depend on implicit domain expertise embedded in data, making relevant evidence difficult to retrieve and synthesize. The section surveys offline rationale extraction, example-based in-context learning, and fine-tuning as approaches to these tasks.

6.1 Overview

  • Hidden rationales are implicit, varied, and often too numerous or extensive for a context window; they may be embedded in in-domain examples or dispersed prior knowledge.Examples include historical question-answer records, legal codes, intermediate mathematical conclusions, and accumulated experience.Source: §6.1 · p. 14

6.2 Challenges and Solutions

  • Hidden-rationale tasks challenge retrieval based on logical or thematic alignment and require synthesizing answers when relevant guidance is dispersed or not stated explicitly.The section identifies logical retrieval and data insufficiency as central difficulties.Source: §6.2 · p. 14

6.3 Offline Learning

  • Offline methods extract rules, guidelines, or reusable rationales from examples, errors, or generated reasoning; the survey reports that learning from cases can benefit reasoning tasks.Methods include rationale generation, guideline formation, principle extraction, and retrieval of task- or question-level insights.Source: §6.3 · pp. 14–15

6.4 In Context Learning (ICL)

  • In-context learning retrieves examples to support few-shot reasoning, but irrelevant prompt content can distract LLMs and lead to incorrect responses.The section also describes methods for more targeted or diverse example selection and notes challenges extending reasoning beyond trained domains.Source: §6.4 · p. 15

6.5 Fine-tuning

  • Fine-tuning is presented as a way to leverage pretrained knowledge while learning domain rationales; instruction tuning uses paired instruction-output data but remains costly in time and computation.The section also surveys parameter-efficient approaches and domain-specific supervised fine-tuning work.Source: §6.5 · pp. 15–16

7 Conclusion

The conclusion maps query focus to tailored techniques and compares three ways to inject external knowledge into LLMs. Because practical applications combine query types, developers may need a routing pipeline that integrates multiple methods.

7 Conclusion

  • Static knowledge suits CoT, explicit facts basic RAG, implicit facts iterative or graph/tree RAG, extensive linkage text-to-SQL, interpretable rationales prompt tuning or CoT, and hidden rationales offline/in-context learning or fine-tuning.Each query category has distinct challenges and calls for tailored methods.Source: §7 · p. 17
  • Context injection offers interpretability and stability but is limited by the context window and retrieval demands; small models reduce training time, while fine-tuning uses large-model capacity but depends on data design.Out-of-domain factual fine-tuning may increase erroneous outputs and risk losing previously known domain knowledge.Source: §7 · p. 17
  • Choosing an injection strategy requires understanding the data sources and making a decision based on that understanding.Source: §7 · p. 18
  • When practical applications combine query types, developers may need a routing pipeline that integrates multiple methodologies.Source: §7 · p. 18

Related papers

Loading 2409.14924v1…