PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Benchmarking and Improving Retrieval for Deep Research Agents

Tiansheng Hu, Yilun Zhao, Canyu Zhang, Arman Cohan, Chen Zhao

arXiv:2602.05975v2cs.IRcs.CL

TL;DR

The paper asks whether LLM-based retrievers can effectively support deep research agents, a question complicated by reasoning-intensive scientific literature retrieval. It introduces SAGE and evaluates agents and alternative retrievers, finding that BM25 outperforms LLM-based retrievers by approximately 30%; corpus-level augmentation then yields 8% and 2% gains on short-form and open-ended questions.

  • Problem

    Whether LLM-based retrievers can effectively contribute to deep research agent workflows remains an open question, especially for scientific queries requiring reasoning across scholarly articles.

  • Method

    SAGE benchmarks deep research retrieval with 1,200 queries over 200,000 papers, while a corpus-level test-time scaling framework uses LLMs to add metadata and keywords to papers.

  • Results

    BM25 outperforms LLM-based retrievers by approximately 30%, while corpus augmentation yields 8% gains on short-form questions and 2% on open-ended questions.

  • Takeaways & Limitations

    Effective collaboration between retrievers and deep research agents requires adaptation because existing agents generate keyword-oriented sub-queries.

  • Takeaways & Limitations

    The study does not assess retriever-aware agent training, and behavioral findings centered on DR Tulu may not generalize to agents with different training recipes or architectures.

Abstract

from arXiv · show

Deep research agents have emerged as powerful systems for addressing complex queries. Meanwhile, LLM-based retrievers have demonstrated strong capability in following instructions or reasoning. This raises a critical question: can LLM-based retrievers effectively contribute to deep research agent workflows? To investigate this, we introduce SAGE, a benchmark for scientific literature retrieval comprising 1,200 queries across four scientific domains, with a 200,000 paper retrieval corpus. We evaluate six deep research agents and find that all systems struggle with reasoning-intensive retrieval. Using DR Tulu as backbone, we further compare BM25 and LLM-based retrievers (i.e., ReasonIR and gte-Qwen2-7B-instruct) as alternative search tools. Surprisingly, BM25 significantly outperforms LLM-based retrievers by approximately 30%, as existing agents generate keyword-oriented sub-queries. To improve performance, we propose a corpus-level test-time scaling framework that uses LLMs to augment documents with metadata and keywords, making retrieval easier for off-the-shelf retrievers. This yields 8% and 2% gains on short-form and open-ended questions, respectively.

1 Introduction

SAGE examines whether LLM-based retrievers can support deep research agents on reasoning-intensive scientific literature retrieval. It benchmarks agents and retrievers, finding that keyword-oriented agent sub-queries favor BM25, while corpus-level document augmentation improves retrieval.

  • Benchmark: SAGE introduces 1,200 scientific-literature queries over 200,000 papers spanning four domains, including short-form and open-ended questions.Short-form questions have verifiable answers and often require intensive reasoning; open-ended questions reflect practical research tasks such as searching related work.
  • Agent evaluation: All six evaluated deep research agents struggle with reasoning-intensive retrieval requiring metadata synthesis and inter-paper relationships.The evaluation includes proprietary systems such as GPT-5 and Gemini-2.5-Pro, alongside the open-source DR Tulu.
  • Retriever comparison: BM25 outperforms LLM-based retrievers by about 30% in DR Tulu workflows.Existing agents generate keyword-oriented sub-queries, which align with surface-form matching but mismatch the semantic capabilities of LLM-based retrievers.
  • Test-time scaling: The proposed corpus-level test-time scaling framework augments papers with informative metadata and keywords to make retrieval easier for off-the-shelf retrievers.LLMs reason over each paper before adding retrieval-oriented signals to the corpus.
  • Results: 8% gains on short-form questions and 2% on open-ended questions result from corpus-level document augmentation.The framework addresses the reasoning-intensive retrieval challenge without changing the retriever itself.

2 Related Work

Prior work has advanced deep research agents and LLM-based retrievers, but their plug-and-play combination over closed-domain corpora remains insufficiently studied. Test-time retrieval research has focused on reranking, query expansion, and rewriting, whereas SAGE investigates corpus adaptation.

  • Deep Research Agents: Deep research agents autonomously plan multi-step information-seeking tasks but primarily rely on web search or proprietary retrieval backends.The open question is whether they function as plug-and-play systems with LLM-based retrievers over closed-domain corpora.
  • LLM-based Retrievers: Decoder-based retrievers such as LLM2Vec and GritLM repurpose generative LLMs for embedding tasks.Recent retriever research also explores instruction following and reasoning-oriented retrieval capabilities.
  • Test-time Scaling for Retrieval: Test-time retrieval scaling includes reranking models trained to use additional inference-time computation, alongside query expansion and query rewriting.SAGE extends this direction by studying corpus-level adaptation rather than only modifying queries or ranking.

3 SAGE Benchmark

SAGE benchmarks deep research retrieval over a controlled scientific-literature corpus using reasoning-intensive short-form and open-ended questions. It defines agentic retrieval as iterative sub-querying and evidence accumulation, with distinct construction and evaluation procedures for each question type.

  • Problem formulation: Deep research agents iteratively choose to reason, issue sub-queries, or answer while accumulating retrieved evidence across multiple rounds.This formulation distinguishes agentic retrieval from one-shot retrieval-augmented generation.
  • Benchmark scope: SAGE contains 1,200 questions spanning four scientific domains and a large, controlled paper corpus.Each domain is represented by a curated corpus of papers, enabling precise retriever comparisons.
  • Short-form Questions: Short-form questions require synthesizing metadata, paper details, and inter-paper relationships to produce one verifiable answer.Questions are generated from metadata, figures, tables, and reference-overlap relationships; the answer is the seed paper.
  • Evaluation: SAGE evaluates short-form outputs with Exact Match and open-ended retrieval with Weighted Recall over retrieved text and citations.Weighted Recall uses linear gain g(r) = r, where relevance distinguishes most relevant, relevant, and non-relevant papers.
  • Open-Ended Questions: Open-ended questions model real-world literature-review scenarios and assign multiple ground-truth papers weighted by relevance.Their ground truth prioritizes two seed papers and then papers cited by both, with relevance scores of 2, 1, or 0.

4 Experiment

The experiments evaluate deep research agents on SAGE and compare retrievers within DR Tulu. Results show that retrieval quality depends more on query decomposition and retriever compatibility than on search volume alone.

  • 4.2 Web-Search Results: GPT-5 achieves the best short-form Exact Match performance at 71.69%, while open-ended performance varies across domains and models.
  • 4.2 Web-Search Results: DR Tulu can match or exceed proprietary systems in precise, retrieval-heavy settings, outperforming Gemini-2.5-series agents on short-form questions.
  • 4.2 Web-Search Results: Search quantity is insufficient for precise retrieval: Gemini-2.5-flash makes nearly twice as many searches as GPT-5, while DR Tulu averages 37.32 references yet trails GPT-5.
  • 4.2 Web-Search Results: More structured, phrasal query decomposition corresponds to fewer, higher-yield searches and improved retrieval precision compared with DR Tulu’s keyword-like sub-queries.
  • 4.4 Corpus-Search Results: BM25 outperforms LLM-based retrievers by roughly 30% on short-form questions, while the gap narrows on open-ended questions; ReasonIR ranks last on both types.
  • 4.4 Corpus-Search Results: Increasing per-search top-k consistently improves performance, with the largest benefit for ReasonIR, suggesting larger candidate sets offset weaker first-page ranking.
  • 4.4 Corpus-Search Results: LLM-based retrievers underuse their semantic capabilities because agents generate keyword-like sub-queries that mismatch the retrievers’ natural-language-query training distribution.

5 Test-Time Corpus Scaling

The paper introduces corpus-level test-time scaling to make scientific papers easier for existing retrievers to find. It augments documents with metadata and LLM-extracted keywords, producing stronger short-form gains than open-ended gains.

  • 5 Test-Time Corpus Scaling: Corpus-level test-time scaling adds reasoning-derived information to documents instead of increasing query complexity.
  • 5 Test-Time Corpus Scaling: Each document is augmented with bibliographic metadata and eight LLM-extracted keywords summarizing its core contributions.
  • 5 Test-Time Corpus Scaling: BM25 gains 8.18% absolute on short-form questions after corpus scaling, while LLM-based retrievers improve only modestly.
  • 5 Test-Time Corpus Scaling: All three retrievers show only marginal improvements on open-ended questions because limited query-decomposition diversity restricts retrieval breadth.

6 Conclusion

SAGE reveals that LLM-based retrievers underperform BM25 by approximately 30% in deep research agent workflows, while corpus-level test-time scaling improves retrieval. The conclusion also points to direct paper editing or rewriting as an approach considered in this context.

  • SAGE exposes reasoning-intensive scientific literature retrieval as a central challenge for deep research agents.
  • Approximately 30%: LLM-based retrievers underperform BM25 because agents generate keyword-oriented sub-queries.
  • Corpus-level test-time scaling enriches papers with metadata and LLM-generated keywords to improve retrieval.
  • Directly editing or rewriting each paper is identified as an approach in the paper’s discussion.

Limitations and Future Work

The study does not fine-tune or align open-source deep-research agents, limiting assessment of retriever-aware query generation. Behavioral findings are also concentrated on DR Tulu and may not generalize across training recipes or architectures.

  • The study does not perform instruction fine-tuning or alignment on open-source deep-research agents.
  • Without retriever-aware agent training, the study cannot assess whether query-generation strategies could adapt to retriever type.
  • Behavioral analysis is conducted mostly on DR Tulu, whose post-training may influence observed agent behaviors.
  • Findings may not fully generalize to agents using different training recipes or base-model architectures.

Appendix Contents

The appendix contains query-answer examples, query-decomposition analysis, document-length analysis, retriever-behavior comparisons, further SearchR1 experiments, and prompt templates.

  • Appendix A.1 contains a query-answer example.
  • Appendix A.2 presents a query-decomposition case study.
  • Appendix A.3 reports document length distribution.
  • Appendix A.4 compares retriever behavior with BrowseComp-Plus.
  • Appendix A.5 provides further experiments with SearchR1-32B, while A.6 contains prompt templates.

A.1 Query-Answer Example

The appendix includes examples of both short-form and open-ended questions.

  • Figure 5 gives an example of a short-form question.
  • Together, Figures 5 and 6 illustrate two question formats used in the benchmark.
  • Figure 6 gives an example of an open-ended question.

A.2 Query-Decomposition Case Study

This case study contrasts query decomposition and retriever behavior across deep-research settings. It highlights stronger BM25 performance on short-form questions, narrower open-ended gaps, and limited iterative retrieval by SearchR1-32B.

  • Query-decomposition case study: The appendix includes GPT-5 and Dr-Tulu query-decomposition examples alongside prompts for keyword generation and shared-reference analysis.
  • Comparison with BrowseComp-Plus: Longer documents and dispersed evidence reduce the effectiveness of limited-window dense encoding compared with settings where answers are concentrated near document prefixes.BrowseComp-Plus documents average 6,733 tokens versus 13,376 here, and prefix truncation preserves a gold answer for 86.5% of its queries.
  • Comparison with BrowseComp-Plus: Prefix-only dense encoding can favor dense retrievers when answers are front-loaded, whereas SAGE permits up to 32,000 tokens per document and removes that short-prefix advantage.
  • SearchR1-32B: SearchR1-32B issues only 1.1–1.2 searches per question, so end-to-end performance is primarily determined by the initial query and base retriever.
  • Retriever behavior: BM25 markedly outperforms LLM-based retrievers on short-form questions, while BM25 and gte-Qwen are closer on open-ended questions.ReasonIR remains substantially worse on both query types.
Loading 2602.05975v2…