PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedback

Fangyuan Xu, Rujun Han, Yanfei Chen, Zifeng Wang, I-Hung Hsu, Jun Yan, Vishy Tirumalashetty, Eunsol Choi, Tomas Pfister, Chen-Yu Lee

arXiv:2601.18202v1cs.AI

TL;DR

Deep-search agents need complex, high-quality QA data, but long exploration trajectories make human annotation expensive and existing datasets difficult to scale. SAGE generates difficulty-controlled QA pairs and iteratively refines them using search-agent execution feedback. The resulting data improves generated-data quality and yields up to 23% relative downstream improvement.

  • Problem

    High-quality deep-search QA data is expensive to annotate because solving complex questions requires long, multi-document exploration trajectories.

  • Method

    SAGE uses a data generator and search agent in iterative rounds to generate, verify, and refine corpus-grounded QA pairs toward a target search-step difficulty.

  • Results

    Up to 23% relative improvement is obtained on out-of-domain evaluation data, while SAGE also increases generated-data correctness and difficulty.

  • Takeaways & Limitations

    SAGE provides synthetic data that supports training search agents with more challenging and diverse reasoning requirements than existing benchmarks.

  • Takeaways & Limitations

    Experiments use Wikipedia, do not evaluate model scales beyond 7B or alternative reinforcement-learning algorithms, and use pass@K=1 as an imperfect correctness approximation.

Abstract

from arXiv · show

Deep search agents, which aim to answer complex questions requiring reasoning across multiple documents, can significantly speed up the information-seeking process. Collecting human annotations for this application is prohibitively expensive due to long and complex exploration trajectories. We propose an agentic pipeline that automatically generates high quality, difficulty-controlled deep search question-answer pairs for a given corpus and a target difficulty level. Our pipeline, SAGE, consists of a data generator which proposes QA pairs and a search agent which attempts to solve the generated question and provide execution feedback for the data generator. The two components interact over multiple rounds to iteratively refine the question-answer pairs until they satisfy the target difficulty level. Our intrinsic evaluation shows SAGE generates questions that require diverse reasoning strategies, while significantly increases the correctness and difficulty of the generated data. Our extrinsic evaluation demonstrates up to 23% relative performance gain on popular deep search benchmarks by training deep search agents with our synthetic data. Additional experiments show that agents trained on our data can adapt from fixed-corpus retrieval to Google Search at inference time, without further training.

1. Introduction

SAGE addresses the scarcity and limited scalability of complex deep-search QA data by generating question–answer pairs with controlled difficulty and refining them through search-agent execution feedback. The resulting data improves correctness and difficulty, and training on it improves downstream search-agent performance.

  • High-quality, complex QA pairs are costly for humans to annotate, limiting accessible training and evaluation data for capable search agents.
  • Existing multi-hop benchmarks generally require no more than four retrieval and reasoning steps and depend on extensive annotation or structural information.
  • SAGE uses reverse formulation to generate QA pairs from sampled corpus documents under a target difficulty level, grounding questions and answers in retrieved evidence.
  • SAGE combines a data generator with a search agent whose execution traces provide feedback for iterative QA-pair regeneration.
  • 27% relative improvement is achieved on in-domain evaluation, with up to 23% relative improvement on out-of-domain data across 3B and 7B models.

2. Background

Deep-search agents alternate reasoning with tool queries, but existing training resources remain limited by short reasoning chains, expensive trajectory annotation, and costly or unreleased browsing-based data construction.

  • Search agents issue multi-turn queries and alternate reasoning traces with retrieval-tool calls before producing a final answer.
  • Gold trajectories for supervised training include sub-queries, reasoning traces, and retrieved information, making them expensive to collect.
  • Widely used datasets contain many questions requiring only a small number of search steps, while multi-hop datasets are often automatic or partially automatic.
  • Recent automatic pipelines use browsing tools but are difficult to reproduce, expensive to train because of API costs, or not publicly available at scale.

3. Generating Synthetic Data for Deep Search Agent: SAGE

SAGE generates deep-search QA pairs from corpus documents using a target search-step count, verifies them with sampled search-agent traces, and iteratively regenerates them using execution feedback. It evaluates correctness and difficulty jointly to filter and improve the synthetic data.

  • Framework: SAGE targets QA pairs that require multiple search calls and reasoning over retrieved information, using a data generator, search agent, retrieval model, and corpus.
  • Generation procedure: The algorithm generates an initial QA pair from a document and target step count, then reruns the generator with accumulated generator and search-agent traces.
  • Verification: The verification procedure samples K search-agent traces, checks answer agreement for correctness, and measures difficulty by the minimum steps among correct traces.
  • Quality criteria: Difficulty is defined by whether the minimum correct-trace search length reaches the target S, while correctness uses pass@K against the generated answer.
  • Motivation: 18% of generator-only data satisfies both correctness and the targeted 3–7 search-step difficulty constraint.
  • Execution feedback: Execution feedback reconciles discrepancies between generator and search-agent trajectories by feeding both traces back for iterative QA regeneration and filtering pass@K=0 pairs.
  • Evaluation: Table 2 reports correctness, successful generation requiring at least S steps, and difficulty among correct data using Avg@4 and search-step counts.

4. Experiments

The experiments evaluate SAGE’s generated data intrinsically, as training data for search agents, and for transfer from Wikipedia retrieval to Google Search. Across these settings, execution feedback improves data quality and downstream performance, including substantial gains on in-domain, out-of-domain, and Google Search benchmarks.

  • Experimental design: SAGE evaluates generated-data correctness and difficulty, then measures downstream search-agent performance and transfer to Google Search.The intrinsic evaluation uses correctness and difficulty metrics; downstream experiments train search agents on generated data and test fixed-corpus training with Google Search at inference.
  • Intrinsic evaluation: 18% pass rate is achieved by the initial generator, while resampling and execution feedback improve correctness and difficulty; feedback consistently outperforms resampling.The pass metric counts data that is correct and requires at least the target number of search steps; Figure 2 reports larger feedback benefits at higher target steps.
  • Downstream evaluation: 27% relative improvement is achieved by QWEN-3B trained on SAGE data in in-domain evaluation, increasing average accuracy to 28.5%.The comparison is against the reported NQ + HotpotQA and Musique training baselines.
  • Downstream evaluation: 29% relative improvement is achieved by QWEN-7B trained on SAGE data in in-domain evaluation, increasing average accuracy to 38.1%.Musique training reaches 29.6% versus 29.1% for NQ + HotpotQA, while SAGE data reaches 38.1%.
  • Downstream evaluation: On FRAMES, SAGE training yields 11% and 23% relative improvements for QWEN-3B and QWEN-7B, respectively.FRAMES is treated as an out-of-domain dataset in the downstream evaluation.
  • Google Search transfer: Training on SAGE data improves transfer to Google Search, producing 36% and 50% relative improvements on GAIA for QWEN-3B and QWEN-7B.Browsecomp also improves for QWEN-7B; gains on HLE are more modest, likely because of its specialized scientific domain.

5. Analysis

The analysis examines feedback-round ablations, reasoning strategies, and generator failure patterns. Execution feedback improves downstream performance, while SAGE broadens reasoning coverage and exposes difficulty and correctness mismatches.

  • 5.1. Ablation on feedback rounds: Increasing feedback rounds from 0 to 2 improves both in-domain and out-of-domain performance, but a third round adds no further improvement.Three-round data is more difficult than two-round data, suggesting difficulty alone is insufficient without balancing learnability.
  • 5.2. Reasoning strategy analysis: SAGE questions span broader reasoning types than Musique, including conflict resolution, hypothesis generation, self-correction, calculation, and temporal reasoning.Calculation occurs in 35% of SAGE data versus 5% in Musique, while temporal reasoning occurs in 32% versus 8%.
  • 5.3. Analyzing data generator’s error: 35% of generated questions exhibit information co-location, while 21% exhibit multi-query collapse, reducing the search steps actually required.These patterns create mismatches between intended and actual difficulty and are discoverable through execution feedback.
  • 5.3. Analyzing data generator’s error: Incorrect data most commonly reflects search-agent retrieval or reasoning failures, while approximately 20% stems from generator errors and 7% from ambiguous questions.The pipeline does not distinguish search-agent failures from truly incorrect data and filters such examples from training.

6. Related Work

Prior deep-search datasets cover increasingly multi-hop questions but remain limited in search depth, scalability, structural requirements, or generation cost. SAGE instead generates data on a fixed corpus and demonstrates transfer to Google Search at inference time.

  • Deep Search: Earlier RAG systems typically use one retrieval step, whereas deep-search agents interleave retrieval and reasoning across multiple steps.Existing benchmarks typically require no more than four retrieval and reasoning steps.
  • Synthetic data generation for deep search: Concurrent synthetic-data methods use reverse QA, difficulty filtering, knowledge graphs, entity relations, or other structured signals to construct complex questions.The cited methods include WebDancer, WebShaper, WebSailor, WebPuzzle, and WebExplorer.
  • Synthetic data generation for deep search: Many concurrent methods depend on commercial retrieval APIs during generation, increasing costs because of numerous intermediate queries.SAGE is grounded in a fixed corpus, with experiments demonstrating transfer to Google Search only at inference time.
  • Synthetic data generation for deep search: Existing public training datasets rely on human annotation, Wikipedia structure, or both, and are grounded in Wikipedia.The passage identifies Kwiatkowski et al., Trivedi et al., and Yang et al. as baseline datasets.

7. Conclusion

The paper introduces SAGE, an agentic pipeline for generating deep-search data from a corpus. A data generator proposes questions and answers, while a search agent attempts solutions and supplies execution feedback; evaluations cover data quality and downstream training.

  • 7. Conclusion: SAGE automatically generates deep-search data for a given corpus through a dual-agent pipeline.The data generator creates complicated questions and answers, while the search agent attempts to solve them and provides execution feedback.
  • 7. Conclusion: Evaluation covers both intrinsic data quality and downstream performance when search agents are trained on SAGE-generated data.The reported evaluations include in-domain and out-of-domain settings.

Limitations

The paper’s limitations include fixed-agent feedback, approximate correctness verification, an RL-focused data format, restricted model and algorithm coverage, and evaluation on only Wikipedia.

  • Proposed method: SAGE uses a fixed search agent for execution feedback, and pass@K=1 is only a practical approximation of generated-data correctness.The paper notes that this criterion may admit hallucinated or incorrect content and suggests co-evolving agents and stronger verification.
  • Experimental setting: The experiments do not cover alternative reinforcement-learning algorithms, models beyond 7B parameters, or domain-specific corpora beyond Wikipedia.The paper suggests future evaluation on legal or scientific corpora.

A.1. Use of large language models (LLMs)

The authors used LLMs to edit the manuscript for grammatical correctness, clarity, and flow, while stating that the scientific content and research ideas remained their own.

  • LLMs such as ChatGPT were used to correct grammatical errors and improve the text’s clarity and flow.
  • The authors state that all core scientific content and research ideas were authored solely by them.

A.2. Implementation details for SAGE

The appendix details SAGE’s generation, search, judging, feedback, reasoning-analysis, and PPO-training configurations, including prompts, update procedures, and an example generated pair.

  • Generation and evaluation settings: SAGE uses temperature 1 with thinking disabled for gemini-2.5-flash, and forces QA generation after the maximum search-call budget is exhausted.
  • Generation and evaluation settings: The appendix provides prompts for the initial data generator, execution-feedback updates, search agent, reference-based judge, and reasoning-strategy analysis.
  • Search-agent training: PPO trains the search agent by maximizing a clipped policy objective using current and previous policies, advantage estimates, and loss masking for retrieved documents.
  • Search-agent training: Training uses learning rates of 1e-6 for the policy LLM and 1e-5 for the value LLM over 500 steps on 8 H100 GPUs.
  • Search-agent training: The configuration limits sequences to 8,192 tokens, responses to 1024 tokens, retrieved content to 1000 tokens, and retrieval calls to 8.
  • Examples and comparisons: The appendix includes an example question generated and updated by SAGE and compares the pipeline with concurrent synthetic-data-generation work.
  • Execution-feedback updates: The feedback procedure updates easy QA pairs to require more search steps, while restricting factual content to context retrieved by the generator and search agent.
Loading 2601.18202v1…