PaperlayerView exact revision on arXiv

Source-linked AI summary

Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks

Yunfan Gao, Yun Xiong, Meng Wang, Haofen Wang

arXiv:2407.21059v1cs.CLcs.AIcs.IR

TL;DR

RAG systems are becoming more complex as applications integrate advanced retrievers, LLMs, and complementary technologies, while the retrieve-then-generate paradigm remains difficult to unify. This paper introduces Modular RAG, decomposing systems into coordinated modules and operators that support reconfigurable flows beyond linear retrieval and generation. It characterizes common flow patterns and discusses how modular designs support future RAG development.

  • Problem

    RAG applications increasingly require complex workflows and diverse capabilities, but existing paradigms struggle to unify evolving methods beyond retrieve-then-generate.

    Source: §I · pp. 1–2

  • Method

    The paper proposes Modular RAG, a three-tier architecture that decomposes systems into modules, sub-modules, and operators coordinated through RAG Flows.

    Source: §I · p. 3

  • Results

    Modular RAG provides a structured, reconfigurable framework for expressing current RAG methods and supporting linear, conditional, branching, and looping workflows.

    Source: §I · p. 3, §V · p. 9, §VII · p. 16

  • Takeaways & Limitations

    The modular design supports scenario-based customization, tracking, debugging, maintainability, scalability, and exploration of new operators and workflows.

    Source: §I · p. 3, §VII · p. 16

Abstract

from arXiv · show

Retrieval-augmented Generation (RAG) has markedly enhanced the capabilities of Large Language Models (LLMs) in tackling knowledge-intensive tasks. The increasing demands of application scenarios have driven the evolution of RAG, leading to the integration of advanced retrievers, LLMs and other complementary technologies, which in turn has amplified the intricacy of RAG systems. However, the rapid advancements are outpacing the foundational RAG paradigm, with many methods struggling to be unified under the process of "retrieve-then-generate". In this context, this paper examines the limitations of the existing RAG paradigm and introduces the modular RAG framework. By decomposing complex RAG systems into independent modules and specialized operators, it facilitates a highly reconfigurable framework. Modular RAG transcends the traditional linear architecture, embracing a more advanced design that integrates routing, scheduling, and fusion mechanisms. Drawing on extensive research, this paper further identifies prevalent RAG patterns-linear, conditional, branching, and looping-and offers a comprehensive analysis of their respective implementation nuances. Modular RAG presents innovative opportunities for the conceptualization and deployment of RAG systems. Finally, the paper explores the potential emergence of new operators and paradigms, establishing a solid theoretical foundation and a practical roadmap for the continued evolution and practical deployment of RAG technologies.

I. INTRODUCTION

RAG has evolved beyond simple retrieve-then-generate pipelines, creating demands for heterogeneous data integration, controllability, maintainability, and workflow orchestration. The paper proposes Modular RAG as a reconfigurable architecture of coordinated modules and operators.

Limitations of Existing RAG

  • Naive RAG relies on chunk similarity and can perform poorly on complex queries while exposing systems to retrieval redundancy and noise.Retrieved noise can interfere with identifying key information and increase the risk of erroneous or hallucinated responses.Source: §I · p. 1
  • Advanced RAG improves retrieval through pre-retrieval and post-retrieval processing, but its capabilities remain behind evolving real-world application requirements.Examples include query rewriting and reranking, while newer settings require heterogeneous data and greater transparency, control, and maintainability.Source: §I · p. 1

Motivation

  • Modular RAG addresses increasingly complex RAG systems by targeting design, management, maintenance, and diverse application demands.The framework is motivated by heterogeneous data, interpretability and controllability needs, component coordination, and workflow scheduling.Source: §I · pp. 2–3

Modular RAG Architecture

  • The proposed architecture organizes RAG into independent modules, sub-modules, and operators coordinated through orchestration and represented as computational graphs.Advanced RAG is treated as a special case of Modular RAG, and Naive RAG as a special case of Advanced RAG.Source: §I · p. 3

Implications

  • Modular RAG emphasizes flexibility, extensibility, maintainability, and comprehensibility through independently designed components and adaptable workflows.The paper also discusses new methods, operators, and application-specific adaptations as directions for future exploration.Source: §I · p. 3

RAG Flow

  • RAG Flow composes modules and operators into six typical flow patterns, allowing current RAG methods to be expressed across practical scenarios.The contribution frames workflow orchestration as a way to analyze the universality of Modular RAG.Source: §I · p. 3

II. RELATED WORK

Prior RAG research has organized methods around retrieval, generation, augmentation, training, and applications, but existing syntheses largely assume fixed processes. The paper responds by formalizing RAG as modular components and computational flows.

II. RELATED WORK

  • RAG research has progressed from retrieval-augmented pretraining and inference methods toward hallucination reduction, token reduction, graph integration, fine-tuning, and adaptive retrieval.Examples include Retro, Atlas [22], IRCOT [23], RRR [24], LLMlingua, and adaptive retrieval [14], [28].Source: §II · p. 3
  • Prior syntheses classify RAG by enhancement stage, retrieval-generation-augmentation processes, method categories, model components, training strategies, and application domains.These perspectives include pre-training, inference, fine-tuning, pre-retrieval, retrieval, post-retrieval, generation, and multiple RALM training methods.Source: §II · pp. 3–4
  • Existing RAG reviews primarily optimize retrieval and generation within fixed processes, leaving process scheduling and functional componentization insufficiently analyzed.The paper identifies this gap as a reason that RAG paradigms lag behind the technology’s development.Source: §II · p. 4

III. FRAMEWORK AND NOTATION

  • The framework defines indexing, retrieval, and generation as core RAG components, with indexing embedding document chunks and retrieval selecting the top k similar chunks.Generation then supplies the query and retrieved chunks jointly to the LLM.Source: §III · p. 4
  • Modular RAG represents systems through three levels: modules for core processes, sub-modules for functional groupings, and operators for specific implementations.Operators are denoted Op = {fθi}, while modules and sub-modules organize them hierarchically.Source: §III · p. 5
  • Modules and operators form a RAG Flow that can be decomposed into a graph of sub-functions, with the simplest instance being a linear chain.The notation expresses the flow as F = (Mϕ1, . . . , Mϕn), where ϕ denotes module parameters.Source: §III · p. 5

IV. MODULE AND OPERATOR

The Modular RAG framework organizes RAG into six modules and specialized operators, covering indexing, pre-retrieval, retrieval, post-retrieval, generation, and orchestration. The indexed and pre-retrieval stages address fragmented document structure and poorly formulated or ambiguous queries through chunk organization, metadata, query expansion, and decomposition.

IV. MODULE AND OPERATOR

  • Modular RAG defines six modules: Indexing, Pre-retrieval, Retrieval, Post-retrieval, Generation, and Orchestration.Source: §IV · p. 5

A. Indexing

  • Indexing must address incomplete chunk representations, inaccurate similarity search, and unclear reference trajectories across documents.Source: A. Indexing · p. 5

1) Chunk Optimization:

  • Chunk optimization balances context and noise: larger chunks provide more context but increase noise, processing time, and cost, whereas smaller chunks may omit necessary context.Source: 1) Chunk Optimization · p. 5

2) Structure Organization:

  • Hierarchical indexes and knowledge graphs organize document relationships to speed traversal, preserve concept connections, and improve retrieval coherence.Source: 2) Structure Organization · p. 5

B. Pre-retrieval

  • Pre-retrieval must handle poorly worded, complex, or ambiguous queries, including abbreviations such as LLM with multiple meanings.Source: B. Pre-retrieval · p. 5

1) Query Expansion :

  • Query expansion creates diverse parallel queries or decomposes complex problems into sub-queries, but multi-query expansion can dilute the original intent.Source: 1) Query Expansion · pp. 5–6

2) Query Transformation:

Query transformation and construction adapt user questions for retrieval across textual and structured sources, while retriever selection and fine-tuning trade off efficiency, semantic coverage, and domain alignment. The section describes rewrites, hypothetical documents, structured query languages, hybrid retrieval, supervised adaptation, language-model supervision, and adapters.

2) Query Transformation:

  • Query transformation rewrites or abstracts the original question, and methods such as HyDE retrieve through hypothetical answers or queries rather than the original wording.Source: 2) Query Transformation · p. 6

3) Query Construction:

  • Query construction converts user questions into languages such as SQL or Cypher and can combine structured queries with semantic information and metadata.Source: 3) Query Construction · p. 6

C. Retrieval

  • Retriever selection balances efficiency, retrieval quality, and task-data-model alignment: sparse methods are efficient, dense methods capture richer semantics, and hybrid methods combine both.Source: C. Retrieval · p. 6, 1) Retriever Selection · p. 6

2) Retriever Fine-tuning:

  • Domain-specific fine-tuning improves retrieval efficiency and alignment when contexts diverge from pre-trained corpora, including specialized healthcare and legal terminology.Source: 2) Retriever Fine-tuning · p. 6
  • Supervised fine-tuning uses labeled positives and negatives, whereas language-model-supervised retrieval uses model-generated results as supervisory signals during RAG.Source: 2) Retriever Fine-tuning · pp. 6–7
  • Adapters reduce the cost of fine-tuning large retrievers and can improve alignment with specific downstream tasks.Source: 2) Retriever Fine-tuning · p. 7

D. Post-retrieval

Post-retrieval processing improves the usefulness of retrieved context by reranking, compressing, or selecting chunks before generation. These operators address lost-in-the-middle effects, contradictory or noisy documents, context-window limits, and the risk that excessive context obscures key information.

D. Post-retrieval

  • Post-retrieval processing is needed because long contexts can lose middle content, include noisy or contradictory chunks, and exceed the model’s context window.Source: D. Post-retrieval · p. 7

1) Rerank:

  • Reranking reorders retrieved chunks without changing their content or length, using rule-based relevance and diversity criteria or language models, including for tables and images.Source: 1) Rerank · p. 7

2) Compression:

  • Excessive context can add noise and diminish the LLM’s perception of key information, motivating compression and selection of retrieved content.Source: 2) Compression · p. 7
  • LLMLingua compresses prompts by removing unimportant tokens with aligned small language models, balancing compression ratio and language integrity without additional LLM training.Source: 2) Compression · p. 7

3) Selection:

  • Selection removes irrelevant chunks rather than compressing their contents, using predicates, self-information, or LLM critique to filter retrieved material.Source: 3) Selection · p. 7

E. Generation

Generation combines answer production with fine-tuning and verification, while orchestration routes queries through suitable modules and flows.

1) Generator Fine-tuning:

  • Fine-tuning can adapt an LLM to domain knowledge, specific input formats, output styles, or human and retriever preferences.Source: 1) Generator Fine-tuning · pp. 7–8
  • Generator fine-tuning can align generators and retrievers through dual fine-tuning, with RA-DIT aligning their scoring functions using KL divergence.Source: 1) Generator Fine-tuning · p. 8

2) Verification :

  • Verification filters responses that fail required standards through knowledge-base or model-based checking, with model-based verification enabling iterative regeneration.Source: 2) Verification · p. 8

F. Orchestration

  • Modular RAG uses orchestration to make decisions at pivotal junctures and dynamically select subsequent steps based on previous outcomes.Source: F. Orchestration · p. 8

1) Routing:

  • Routing assigns queries to tailored RAG flows using functions that map queries or intents to active modules.Source: 1) Routing · p. 8

2) Scheduling:

Scheduling governs whether modular RAG should stop, retrieve again, or follow another action based on rules, LLM judgments, or knowledge-guided reasoning.

2) Scheduling:

  • Scheduling identifies retrieval needs, assesses response adequacy, and decides when to cease generation or initiate another retrieval loop.Source: 2) Scheduling · pp. 8–9
  • Rule-based scheduling accepts tentative answers only when all token probabilities meet a threshold; otherwise, it generates a new answer.Source: 2) Scheduling · p. 9
  • LLM judges choose subsequent actions through prompting or fine-tuned action tokens, with the latter requiring many compliant instruction sets.Source: 2) Scheduling · p. 9
  • Knowledge-guided scheduling uses knowledge-graph reasoning chains to steer retrieval and generation while improving problem-solving precision and explanation clarity.Source: 2) Scheduling · p. 9

3) Fusion:

Fusion integrates outputs from multiple branches, while modular RAG flows organize operators into graphs that support linear execution and common linear implementations.

3) Fusion:

  • Fusion combines branch information into coherent outputs, using LLM summarization, weighted ensembles, or Reciprocal Rank Fusion for heterogeneous rankings.Source: 3) Fusion · p. 9
  • Weighted ensemble assigns document-query similarity weights through softmax so the normalized weights sum to one.Source: 3) Fusion · p. 9

V. RAG FLOW AND FLOW PATTERN

  • A modular RAG flow is a graph of sub-functions whose operators can execute linearly or conditionally, branch, or loop through control logic.Source: §V · p. 9

A. Linear Pattern

  • The full linear pattern proceeds through indexing, pre-retrieval, retrieval, post-retrieval, and generation, while omitting both retrieval-processing stages yields Naive RAG.Source: A. Linear Pattern · p. 10
  • In RRR [24], a trainable query rewriter precedes BM25 retrieval and is optimized with reinforcement learning using the LLM’s final output as reward.Source: A. Linear Pattern · p. 10, Figure 5 · p. 10

B. Conditional Pattern

Conditional flows select one pipeline based on routing conditions, whereas branching flows run multiple parallel retrieval or generation paths before merging their outputs.

B. Conditional Pattern

  • Conditional RAG uses a routing module to select the next module or pipeline according to the query and retrieved information.Source: B. Conditional Pattern · p. 10

C. Branching

B. Conditional Pattern

  • Conditional routes can vary retrieval sources, processes, configurations, models, and prompts to suit different question scenarios.Source: B. Conditional Pattern · p. 10

C. Branching

  • Post-retrieval branching performs one retrieval, generates independently for each document chunk, and merges the resulting outputs.Source: Figure 8 · p. 11, C. Branching · p. 11
  • Branching differs from conditional routing by executing multiple parallel branches instead of selecting one among alternative branches.Source: C. Branching · p. 11
  • REPLUG [55] exemplifies post-retrieval branching by weighting token probabilities across branches and using the ensemble result to fine-tune Contriever.Source: C. Branching · pp. 11–12, Figure 9 · p. 12

D. Loop Pattern

Loop-pattern Modular RAG systems organize retrieval and generation into iterative, recursive, or adaptive flows controlled by scheduling and judgment mechanisms. These patterns support repeated retrieval, query deepening, or model-directed decisions about retrieval and termination.

D. Loop Pattern

  • Loop flows model RAG as a directed graph whose modules can return to earlier modules through a Judge, forming iterative, recursive, or adaptive patterns.The loop ends when the control logic does not return to a predecessor or when a predefined limit is reached.Source: D. Loop Pattern · pp. 12–13

Iterative retrieval

  • Iterative retrieval repeats retrieval and generation for up to T iterations, using previous outputs and the current query to retrieve documents and generate updated outputs.The Judge module decides whether to continue, and the final output synthesizes the generated sequence.Source: D. Loop Pattern · p. 12
  • ITER-RETGEN [56] exemplifies iterative flow by using each generation as context for the next retrieval, enabling progressively more relevant knowledge retrieval.Its figure depicts multiple retrieval-generation rounds bounded by a maximum iteration count.Source: D. Loop Pattern · p. 12, Figure 11 · p. 13

Recursive retrieval

  • Recursive retrieval derives new queries from prior retrieval and generation steps, expanding a tree-like process until its termination condition or maximum recursive depth is reached.ToC [13] illustrates this pattern by recursively deepening analysis through newly generated queries and progressively expanding a clarification tree.Source: D. Loop Pattern · p. 13, Figure 12 · p. 13

Adaptive (Active) retrieval

  • Adaptive retrieval lets RAG systems decide when to retrieve and when to stop, using prompt-based or tuning-based criteria.FLARE [14] retrieves when provisional generation contains low-probability tokens, while Self-RAG [28] uses specialized tokens to control retrieval, critique, and final-output selection.Source: D. Loop Pattern · pp. 13–14, Figure 13 · p. 14, Figure 14 · p. 14

E. Tuning Pattern

Modular RAG includes fine-tuning patterns that optimize trainable components and their compatibility with the overall flow. The main categories are retriever fine-tuning, generator fine-tuning, and dual fine-tuning.

E. Tuning Pattern

  • Fine-tuning in Modular RAG optimizes component performance and compatibility with the overall flow through retriever, generator, and dual fine-tuning stages.The framework treats fine-tuning as a way to improve trainable language-model components within RAG workflows.Source: E. Tuning Pattern · p. 14
  • Retriever fine-tuning includes direct supervised tuning, trainable adapters, language-model-supervised retrieval, and LLM-reward reinforcement learning.These methods respectively use retrieval datasets, adapters for otherwise inaccessible embedding models, LLM-generated supervision, or reinforcement learning to align retrieval with generation.Source: Figure 15 · p. 14

1) Retriever FT:

Retriever fine-tuning adapts retrieval components to specialized data, downstream tasks, and generator preferences. The listed methods range from supervised retriever updates and adapters to LLM-based supervision and reinforcement learning.

1) Retriever FT:

  • Retriever fine-tuning uses direct supervised datasets to adapt dense retrievers to open-source or domain-specific retrieval data.This approach constructs specialized retrieval data and fine-tunes the dense retriever directly.Source: 1) Retriever FT · p. 14
  • Trainable adapters provide an alternative when API-based embedding models cannot be directly fine-tuned and can align representations with task-specific or general downstream needs.The passage names PRCA [42] as a task-specific example and AAR [58] as a general-purpose example.Source: 1) Retriever FT · p. 14
  • LM-supervised retrieval fine-tunes the retriever using results generated by an LLM as the supervisory signal.The method transfers language-model-generated outcomes into retriever training.Source: 1) Retriever FT · p. 14
  • LLM-reward reinforcement learning uses LLM outputs as supervision and aligns the retriever with the generator through a generative Markov-chain formulation.The retrieval process is decomposed into a generative Markov chain for reinforcement learning.Source: 1) Retriever FT · p. 14

3) Dual FT:

  • Dual fine-tuning updates both the language model and retriever so their preferences remain coordinated.RA-DIT [27] updates the LLM toward correct answers under retrieval-augmented instructions and adjusts the retriever to reduce divergence from LLM preferences.Source: 3) Dual FT · p. 14

2) Generator FT:

Generator fine-tuning and broader Modular RAG opportunities extend the framework from component optimization to customizable flows and new operators. The paper presents modularity as a basis for scalable research, application-specific design, and continued RAG innovation.

2) Generator FT:

  • Generator fine-tuning includes supervised tuning, distillation, and reinforcement learning from LLM or human feedback.These methods add knowledge or format control, transfer capabilities from stronger models, and evaluate final answers through human or LLM feedback.Source: Figure 16 · p. 15
  • Dual fine-tuning coordinates retriever and generator updates so retrieval preferences align with language-model objectives.The retriever is updated against LLM preference distributions while the language model is optimized for correct retrieval-augmented answers.Source: Figure 17 · p. 15, 2) Generator FT · p. 15

VI. DISCUSSION

  • Modular RAG is presented as scalable and compatible with evolving methods, supporting new modules, operators, and workflows.The discussion frames this compatibility as support for model innovation and adaptation to changing application requirements.Source: §VI · p. 15, B. Compatibility with new methods · p. 15

A. Opportunities in Modular RAG

  • For researchers, modularity enables the introduction of innovative modules and operators and supports exploration of new theoretical and practical directions.The framework is described as a way to organize and extend RAG technologies while preserving flexibility.Source: A. Opportunities in Modular RAG · p. 15
  • For applications, users can tailor RAG flows to specific data, use cases, and downstream tasks, while developers can define new domain-specific flow patterns.This customization is described as simplifying design and implementation and enriching application functionality.Source: A. Opportunities in Modular RAG · p. 15

1) Recombination of the current modules:

  • Existing modules can be recombined into new processes, as in DR-RAG [59] and PlanRAG [18], without introducing new modules.DR-RAG uses branching retrieval and classifier filtering, while PlanRAG adds planning, query expansion, targeted retrieval, and replanning for complex decision-making.Source: 1) Recombination of the current modules · p. 15

3) New flow derived from new operators.:

  • New operators can create new flow designs, as Multi-Head RAG [60] does by retrieving multifaceted documents through multi-head attention activations.Different attention heads represent different data and query aspects, improving retrieval for complex queries whose relevant documents may be far apart in embedding space.Source: 3) New flow derived from new operators · p. 15

VII. CONCLUSION

  • The conclusion characterizes Modular RAG as a LEGO-like system of discrete modules and operators that supports structured design, scenario-based customization, tracking, debugging, maintainability, and scalability.It also identifies novel modules and workflows as avenues for future RAG development.Source: §VII · p. 16

Related papers

Loading 2407.21059v1…