PaperlayerView exact revision on arXiv

Source-linked AI summary

MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval

Qi Luo, Xiaonan Li, Yuxin Wang, Tingshuo Fan, Yuan Li, Xinchi Chen, Xipeng Qiu

arXiv:2510.27569v1cs.CL

TL;DR

LLM reasoning is constrained by static pretraining knowledge, while single-retriever RAG limits access to corpus-wide evidence. MARAG-R1 coordinates four retrieval tools through supervised fine-tuning and reinforcement learning, interleaving retrieval with reasoning. It substantially outperforms baselines and achieves state-of-the-art global retrieval performance.

  • Problem

    Single-retriever RAG relies on fixed top-k selection, restricting access to corpus-wide evidence needed for comprehensive reasoning.

  • Method

    MARAG-R1 trains an LLM to coordinate semantic search, keyword search, filtering, and aggregation through supervised fine-tuning and reinforcement learning.

  • Results

    MARAG-R1 substantially outperforms existing baselines and establishes state-of-the-art performance on global retrieval tasks.

  • Takeaways & Limitations

    Multi-tool coordination enables broader and more efficient external information acquisition for global reasoning.

  • Takeaways & Limitations

    Existing retrieval approaches remain constrained by top-k paradigms or graph-traversal locality, limiting exhaustive corpus-wide retrieval for aggregation tasks.

Abstract

from arXiv · show

Large Language Models (LLMs) excel at reasoning and generation but are inherently limited by static pretraining data, resulting in factual inaccuracies and weak adaptability to new information. Retrieval-Augmented Generation (RAG) addresses this issue by grounding LLMs in external knowledge; However, the effectiveness of RAG critically depends on whether the model can adequately access relevant information. Existing RAG systems rely on a single retriever with fixed top-k selection, restricting access to a narrow and static subset of the corpus. As a result, this single-retriever paradigm has become the primary bottleneck for comprehensive external information acquisition, especially in tasks requiring corpus-level reasoning. To overcome this limitation, we propose MARAG-R1, a reinforcement-learned multi-tool RAG framework that enables LLMs to dynamically coordinate multiple retrieval mechanisms for broader and more precise information access. MARAG-R1 equips the model with four retrieval tools -- semantic search, keyword search, filtering, and aggregation -- and learns both how and when to use them through a two-stage training process: supervised fine-tuning followed by reinforcement learning. This design allows the model to interleave reasoning and retrieval, progressively gathering sufficient evidence for corpus-level synthesis. Experiments on GlobalQA, HotpotQA, and 2WikiMultiHopQA demonstrate that MARAG-R1 substantially outperforms strong baselines and achieves new state-of-the-art results in corpus-level reasoning tasks.

1 Introduction

LLMs need external information to improve factual, adaptable reasoning, but single-retriever RAG restricts access to narrow static evidence. MARAG-R1 addresses this bottleneck by reinforcement-learning-based coordination of multiple retrieval tools and achieves state-of-the-art global retrieval results.

  • 1 Introduction: LLMs are limited by static pretraining knowledge, which can produce factual inaccuracies and reduced adaptability to new information.Accessing and using external information is presented as important for factuality, interpretability, and trustworthiness.
  • 1 Introduction: Single-retriever RAG confines models to a narrow, static corpus subset, creating a major bottleneck for comprehensive external information acquisition.Fixed top-k workflows cannot refine or expand their evidence after retrieval, limiting tasks requiring understanding across all documents.
  • 1 Introduction: MARAG-R1 coordinates semantic search, keyword search, filtering, and aggregation to acquire and integrate broader external information.The framework interleaves reasoning and retrieval, progressively gathering and consolidating evidence until sufficient information is obtained.
  • 1 Introduction: The model learns tool usage through supervised fine-tuning followed by reinforcement learning that optimizes tool sequencing and reasoning strategies for coverage.This training design lets the agent learn both when and how to invoke retrieval tools.
  • 1 Introduction: MARAG-R1 significantly outperforms all baselines and establishes new state-of-the-art performance on global retrieval tasks.Further analyses report that its components contribute to overall improvements and transfer strongly to multi-hop QA tasks.

2 Related Work

RAG research has progressed from dense semantic retrieval toward adaptive, multi-hop, structured, and tool-using approaches. However, existing methods remain limited in corpus-wide coverage by top-k selection and graph locality, motivating broader retrieval strategies.

  • Foundations of RAG: Early RAG systems used dense dual-encoder retrievers to find semantically similar documents and improve open-domain question answering.Later work explored contrastive learning and tighter retrieval-generation integration.
  • Adaptive and multi-hop retrieval: Top-k retrieval limits complete document coverage for corpus-wide aggregation tasks, even as recent methods improve reasoning over small document sets.Self-RAG, FLARE, and IRCoT dynamically control or interleave retrieval and reasoning, but remain constrained by top-k paradigms.
  • Structured retrieval: Graph-based RAG preserves structured relationships but can lose information during graph construction and restrict retrieval through local graph traversal.These limitations hinder exhaustive corpus-wide retrieval required for global aggregation tasks.
  • Tool-using retrieval agents: Tool-using agents enable language models to interleave reasoning and actions with external tools, while LLatrieval iteratively verifies and updates retrieval results.This line of work opens a path toward more interactive retrieval than fixed retrieval pipelines.

3 Method

MARAG-R1 replaces single-retriever access with reinforcement-learned coordination of four complementary retrieval tools. Its three-stage pipeline collects expert trajectories, applies supervised fine-tuning, and uses reward-driven reinforcement learning to improve evidence acquisition and tool coordination.

  • 3.1 Expert Trajectory Collection: MARAG-R1 combines semantic search, keyword search, filtering, and aggregation to broaden and synthesize retrieved evidence.The tools target contextual exploration, precise matching, constraint-based selection, and statistical or structural synthesis, respectively.
  • 3 Method: The framework follows expert trajectory collection, supervised fine-tuning, and reinforcement learning for tool coordination.Expert traces are generated and filtered for consistency and factual correctness before the cold-start model is refined through RL.
  • 3.1.2 Trajectory Structure and Collection: Each trajectory interleaves intermediate reasoning, a tool invocation, and resulting documents before ending with a predicted answer.This structure represents multi-step retrieval and reasoning as successive states.
  • 3.3.1 Reward Design: The composite reward combines answer correctness, document coverage, and tool exploration to balance accurate answers, comprehensive evidence, and efficient retrieval.Answer reward uses token-level F1; document coverage rewards necessary documents while minimizing irrelevant retrievals; exploration discourages redundant or unproductive calls.
  • 3.3.2 Policy Optimization: RLOO refines multi-tool coordination by comparing each trajectory’s reward with the mean reward of other trajectories in its batch.The leave-one-out baseline reduces gradient variance without an explicit reference model or KL regularization.

4 Experimental Setup

The experiments evaluate MARAG-R1 against representative RAG systems on corpus-level and multi-hop reasoning benchmarks using answer and document-retrieval metrics. Configurations are unified across methods to support fair comparison.

  • Baselines: The comparison includes single-round, iterative, graph-based, and reinforcement-learning RAG baselines.Named systems include StandardRAG, ITER-RETGEN, IRCoT, HyperGraphRAG, Search-R1, and ReCall.
  • Benchmarks: Evaluation spans GlobalQA corpus-level reasoning and multi-hop benchmarks that test evidence synthesis across multiple documents.GlobalQA includes TopK, Count, Sort, and MinMax tasks requiring corpus-wide information integration.
  • Evaluation Metrics: Performance is measured with token-level answer F1 and Document F1@k for retrieved-document coverage.The two metrics assess final-answer quality and overlap between retrieved and gold document sets.
  • Implementation: All methods use BGE retrieval, Qwen3-4B filtering, TopK 20, identical hardware and hyperparameters, and chunk-free retrieval.These choices standardize evidence granularity and experimental conditions across systems.

5 Main Results

MARAG-R1 achieves state-of-the-art GlobalQA performance across model scales, outperforming retrieval and reasoning baselines through reinforcement-learned coordination of multiple retrieval tools. Its gains extend beyond local retrieval by supporting broader evidence acquisition and corpus-level reasoning.

  • Overall Performance: MARAG-R1 achieves the highest F1 scores across Qwen2.5-3B, 7B, and 14B, increasing from 26.4 to 31.22, while also attaining the best D-F1@20 scores from 37.05 to 42.11.F1 measures final-answer accuracy, whereas D-F1@20 measures intermediate reasoning-step correctness.
  • Comparison with Graph-based Methods: Compared with HyperGraphRAG, MARAG-R1 preserves fine-grained evidence and achieves 32.65 F1 on TopK and 39.2 on MinMax, whereas HyperGraphRAG lacks meaningful F1 on most tasks.Static graph abstractions lose factual and numerical detail needed for corpus-level aggregation.
  • Comparison with Local RAG Methods: MARAG-R1 expands beyond local top-ranked retrieval by dynamically coordinating semantic search, keyword search, filtering, and aggregation for corpus-level reasoning.These complementary tools broaden the effective retrieval horizon and support aggregation across dispersed evidence.
  • Effect of Reinforcement Learning: Reinforcement learning improves Qwen2.5-14B over MARAG-CS from 28.92 to 31.22 average F1 and from 39.83 to 42.11 D-F1@20.The largest gains occur on MinMax, where MARAG-R1 reaches 39.2 / 45.81 versus 36.53 / 42.45 for MARAG-CS.
  • Scaling Behavior: MARAG-R1 scales better than competing methods: its F1 rises from 26.4 to 31.22 between 3B and 14B, while ReCall increases only from 6.41 to 14.25.The comparison indicates that larger models benefit more when they learn appropriate strategies for coordinating multiple tools.

6 Analysis

Ablations show that supervised initialization, answer supervision, document coverage, and tool-use rewards contribute complementary benefits. Additional tool calls, retrieval-tool diversity, and learned coordination support stronger performance and transfer to unseen multi-hop tasks.

  • Impact of External Information Acquisition: MARAG-R1 averages 6.32 tool calls per query and reaches 31.22 F1 and 42.11 D-F1@20, whereas MARAG makes 2.81 calls and scores 8.63 / 13.58.MARAG-CS makes 6.27 calls but remains below MARAG-R1 by +1.6 F1 and +2.3 D-F1@20, indicating that call quality and sequencing also improve.
  • Impact of Different Retrieval Tools: Removing the Aggregation Function causes the largest retrieval-tool ablation drop, decreasing F1 by 16.9 and D-F1@20 by 18.7.Aggregation is especially important for corpus-level statistical reasoning in Count, Sort, and MinMax tasks.
  • Generalization to Multi-Hop Reasoning Tasks: On unseen 2WikiMultiHopQA and HotpotQA, MARAG-R1 achieves 22.00 EM / 26.93 F1 and 31.00 EM / 39.16 F1, respectively, surpassing IRCoT.Averaged across datasets, it improves EM by +18.5 and F1 by +11.8, showing transfer from global retrieval training to multi-hop reasoning.
  • Training Dynamics: During RL, reward growth coincides with rising test F1 and D-F1@20, while both evaluation metrics continue improving after rewards plateau.The authors interpret this pattern as evidence that the reward formulation aligns with reasoning quality and supports generalization.

7 Conclusion

MARAG-R1 addresses the information-access bottleneck by coordinating multiple retrieval tools and iteratively gathering evidence for reliable reasoning. Its combination of supervised fine-tuning and reinforcement learning improves retrieval completeness and reasoning accuracy, achieving state-of-the-art global retrieval performance.

  • 7 Conclusion: MARAG-R1 coordinates multiple retrieval tools and iteratively gathers external knowledge until evidence is sufficient for reliable reasoning.The framework combines supervised fine-tuning with reinforcement learning to learn strategies for comprehensive and efficient information acquisition.

A Effect of Different Retrieval Steps

Increasing retrieval steps improves MARAG-R1’s ability to gather evidence and construct coherent global knowledge. Performance rises rapidly at low step counts and remains stable beyond ten steps, although a case study shows that missing document IDs can still cause reasoning errors.

  • A Effect of Different Retrieval Steps: At 2–5 retrieval steps, MARAG-R1 and ReCall show rapid gains, indicating that iterative retrieval enriches the evidence used for reasoning.The step count directly influences the model’s ability to construct a coherent global knowledge graph.
  • A Effect of Different Retrieval Steps: A four-step case study returns document ID 1 instead of the expected 21 because document IDs were missing and the model equated coherence with the smallest ID.The failure occurred after searches for three entity or topic descriptions followed by a reasoning step.
  • A Effect of Different Retrieval Steps: MARAG-R1’s F1 increases from 0 to 30.65 and D-F1@20 from 16.59 to 40.35 as retrieval steps increase, while ReCall reaches 13.33 and 19.54.Beyond 10 steps, MARAG-R1 maintains stable performance, suggesting a balance between retrieval depth and noise control.

B Effect of Different Filters

MARAG-R1’s reported performance improves as the Qwen3 retriever scale increases from 0.6B to 8B parameters. The smallest retriever remains competitive, while larger retrievers provide incremental gains attributed to improved evidence coverage and precision.

  • B Effect of Different Filters: F1 and D-F1@20 improve from 31.22/42.11 to 32.31/43.06 as the Qwen3 retriever grows from 0.6B to 8B parameters.The results indicate consistent gains from larger retrieval backbones.
  • B Effect of Different Filters: The 0.6B retriever remains competitive, showing that MARAG-R1 can operate effectively with a lightweight retrieval backbone.Larger retrievers add incremental improvements rather than eliminating the usefulness of the smallest configuration.

C Effect of Different Retrievers

MARAG-R1 generalizes across Qwen3 retrievers of different capacities, with larger retrievers yielding steady but marginal performance improvements. Even the 0.6B retriever performs competitively, while stronger retrievers improve evidence coverage and precision.

  • C Effect of Different Retrievers: F1 and D-F1@20 improve from 29.73/40.17 to 30.57/41.26 as the Qwen3 retriever grows from 0.6B to 8B parameters.The trend supports consistent benefits from larger retrieval capacity.
  • C Effect of Different Retrievers: The 0.6B retriever already performs competitively, while larger retrievers provide marginal yet consistent gains in coverage and retrieval precision.MARAG-R1 therefore operates across retrievers of varying capacities and can leverage stronger backbones for higher reasoning fidelity.
Loading 2510.27569v1…