PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent

Bince Qu, Wanli Li, Bo Pan, Jianyu Zhang, Zheng Liu, Pan Zhang, Wei Chen, Bo Zhang

arXiv:2604.17931v5cs.AI

TL;DR

Deep-research agentic RL is limited by synthetic data that misses real search capabilities and live-web training that is unstable and costly. LiteResearcher constructs an isolated virtual world with realistic local search tools and curriculum RL, and LiteResearcher-4B achieves 71.3% on GAIA and 78.0% on Xbench while competing with larger systems. The reported limitation is that long browsing chains can exhaust a small model’s context window.

  • Problem

    Agentic RL for deep research is constrained by synthetic data that fails to elicit genuine search capabilities and by unstable, costly dependence on real-world search.

  • Method

    LiteResearcher constructs an isolated virtual world mirroring real-web architecture, with locally simulated search tools and curriculum RL for sustained training.

  • Results

    71.3% on GAIA and 78.0% on Xbench were achieved by LiteResearcher-4B, with results surpassing larger open-source agents and matching or exceeding several proprietary systems.

  • Takeaways & Limitations

    The results indicate that scalable RL training can produce a powerful on-device deep-research agent and that data–environment design matters alongside model scale.

  • Takeaways & Limitations

    On BrowseComp, deep browsing chains can exhaust a 4B model’s 128K context window, reducing performance without memory summarization.

Abstract

from arXiv · show

Reinforcement Learning (RL) has emerged as a powerful training paradigm for LLM-based agents. However, scaling agentic RL for deep research remains constrained by two coupled challenges: hand-crafted synthetic data fails to elicit genuine real-world search capabilities, and real-world search dependency during RL training introduces instability and prohibitive cost, which limits the scalability of Agentic RL. LiteResearcher is a training framework that makes Agentic RL scalable: by constructing a lite virtual world that mirrors real-world search dynamics, we enable a continuously improving training recipe that empowers a tiny search agent to outperform large-scale open-source and commercial models (e.g., Tongyi DeepResearch and Claude-4.5 Sonnet). Specifically, on common benchmarks such as GAIA and Xbench, our LiteResearcher-4B achieves open-source state-of-the-art results of 71.3% and 78.0% respectively, demonstrating that scalable RL training is a key enabler for Deep Research Agents.

1 Introduction

LiteResearcher addresses the instability, cost, and limited diversity that hinder scalable agentic RL for deep research by building an isolated virtual world mirroring real-web search. Its synthetic data, local tools, and difficulty-aware curriculum support sustained improvement, while LiteResearcher-4B achieves strong benchmark results against larger systems.

  • Motivation: Real-world online RL introduces variance and prohibitive costs, while local retrieval systems cannot reproduce diverse internet search dynamics.These constraints have prevented deep-research agents from achieving sustained scalable RL improvements.
  • Approach: LiteResearcher constructs an isolated virtual world that mirrors real-world web architecture, combining realistic search dynamics with controlled execution.The twin architecture is intended to support generalization to the open web while shielding training from environmental noise.
  • Approach: Scaling information sources produces realistic training-task distributions even with simple data synthesis.The framework prioritizes broad information-source coverage to capture diverse information-seeking behavior.
  • Approach: Targeted expansion of a local webpage corpus creates search and browse tools whose dynamics resemble the open web while remaining controlled.Each synthetic task triggers retrieval of related real-world webpages for the local environment.
  • Approach: Difficulty-aware curriculum RL progressively increases task difficulty and context length while retaining tasks the current model can partially solve.Discarding trivially easy and impossibly hard tasks provides a consistent training signal and helps avoid saturation.
  • Results: 71.3% on GAIA and 78.0% on Xbench were achieved by LiteResearcher-4B, surpassing larger open-source agents and matching or exceeding several proprietary systems.The reported comparisons include Tongyi DeepResearch 30B at 70.9%/75.0% and Claude-4.5-Sonnet at 71.2%/66.0%.

2 Related Work

Prior work has established RL for reasoning but faces distinctive environment-design challenges in agentic deep research. Existing systems trade off realistic live-web interaction against cost and nondeterminism, or efficiency against the narrowness of local corpora.

  • RL for Reasoning: RL with verifiable rewards demonstrated that reasoning capabilities can be internalized directly into language-model weights.This reasoning-oriented progress motivates extending RL to agents that also use tools.
  • Agentic Deep Research: Agentic deep-research systems interact with external search environments through iterative query formulation and page navigation.The literature is organized by how these environments are designed.
  • Agentic Deep Research: Online systems provide realistic interactions but incur high cost and nondeterministic reward signals, whereas local retrieval systems offer narrower corpus-based environments.These are contrasting environment-design trade-offs in prior work.

3 Preliminaries: Agentic Framework

The paper models deep research as sequential ReAct decision-making: an agent alternates reasoning, executable actions, and observations across multiple interaction steps before producing a final answer.

  • Sequential Decision Process: At each step, the agent generates a reasoning thought conditioned on prior history, then selects an action using that history and thought.The interaction history records thoughts, actions, and environmental observations.
  • Action Space: The action space contains two primitives: Search(q′), which returns ranked snippets and URLs, and Browse(u,q′), which returns a query-conditioned page summary.Search formulates retrieval while Browse inspects a selected webpage.
  • Termination: After collecting sufficient information, the agent terminates environment interaction and generates a final response to the original query.

4 Methods: Towards Scalable Training

LiteResearcher scales agentic RL by co-evolving synthetic training data with an enriched local corpus, providing stable search tools and difficulty-aware curriculum learning. The framework targets diverse search capabilities while filtering tasks and enforcing on-policy optimization to sustain useful learning signals.

  • Framework Overview: Unstable online interactions bottleneck scalable training, motivating a framework that decouples agent evolution from the open web while preserving real-web dynamics.
  • Framework Overview: The framework has three pillars: co-constructed training data and corpus, stable local tools, and difficulty-aware curriculum learning.Together they target data diversity, environment stability, and continued capability growth.
  • Search Capabilities: Complex deep-research trajectories are decomposed into five atomic capabilities: direct information, aggregation, enumeration, cross-verification, and statistics.
  • Training Data and Corpus: The data pipeline scales information sources first, then filters the resulting pool into solvable, high-quality tasks.Seed QA pairs, source masking, strict quality checks, difficulty filtering, and iterative webpage expansion support this construction process.
  • Stable Local Tool Environment: A local environment built from an enriched corpus replaces online RL interaction with page-level indexing and low-latency search and browse tools.The implementation targets concurrent rollouts while preserving search dynamics analogous to the real web.
  • Difficulty-Aware Curriculum: Training saturation occurs when the model solves easy tasks but fails complex ones, eliminating useful learning gradients.Difficulty-aware filtering retains queries with 1 ≤ c ≤ 7 correct responses across K=8 rollouts, discarding trivial and impossible cases.
  • Difficulty-Aware Curriculum: The method uses GRPO with group-normalized advantages and a strictly on-policy clipped surrogate update.The probability ratio compares training-engine and rollout-engine probabilities, while rollout batches are not reused through replay-style mini-batch updates.

5 Experiments

Experiments evaluate LiteResearcher-4B across agentic search benchmarks and analyze how local environments, synthetic data, on-policy updates, curriculum training, and RL contribute to performance. The results show strong benchmark performance, lower-cost local training, and improvements from the framework’s training design.

  • Main Results: LiteResearcher-4B is evaluated on eight complex reasoning and agentic search benchmarks using online APIs and comparisons with leading closed-source and open-source agents.BrowseComp evaluation uses 400 randomly sampled examples, with the same tool setup as prior work.
  • Main Results: 71.3% on GAIA-Text matches Claude-4.5-Sonnet at 71.2% and surpasses Tongyi DeepResearch 30B at 70.9%, while 78.0% on Xbench-DS surpasses OpenAI-GPT-5-high at 77.8%.LiteResearcher also scores 83.1% on Frames, exceeding Claude-4-Sonnet at 80.7% and DeepSeek-V3.2 at 80.2%.
  • Limitations: BrowseComp exposes a context-length limitation: without memory, performance is 20.3%/28.4%, improving to 27.5%/32.5% with summaries at 64K.The task often requires browsing chains exceeding 20 pages, which can exhaust a 128K context window for a 4B model.
  • Local Environment: 73.2M local tool calls incur zero marginal cost and provide a 10–46× latency advantage over online alternatives, increasing rollout throughput for on-policy training.The corresponding online-search cost is estimated at $59K–$243K.
  • Synthetic Data: 66.8% on GAIA and 71.0% on Xbench with synthetic data fall to 58.7% and 66.3% when that synthetic data is removed.The ablation supports the role of the synthesis pipeline in covering diverse search patterns.
  • Training Strategy: 68.9% GAIA validation accuracy for on-policy training exceeds 66.8% for off-policy training, whose reward rises earlier but later declines.On-policy training continues improving more steadily, while repeated updates over reused rollout batches are hypothesized to create policy lag in long-horizon search.
  • Training Strategy: 68.3% GAIA accuracy after Stage 2 improves 3.6 points over the Stage 1 plateau at 64.7%, showing continued gains after deliberate difficulty escalation.The two-stage strategy adjusts the data distribution and targets greater complexity.
  • RL Contribution: RL raises GAIA accuracy from 55.6% with SFT alone to 71.3%, exceeding the teacher’s 70.9% and producing consistent gains across all eight benchmarks.The reported comparison attributes the primary performance driver to the RL training framework rather than teacher distillation.

6 Conclusions

LiteResearcher addresses data diversity and environment stability by constructing an isolated virtual world for agentic RL. The resulting local training framework supports sustained improvement and strong performance from a compact model.

  • LiteResearcher constructs an isolated virtual world to resolve data diversity and environment stability bottlenecks in agentic RL.
  • The framework identifies sustained capability improvement, scalable synthetic data, and local training as key insights for deep research agents.
  • 71.3% on GAIA and 78.0% on Xbench-DS are achieved by LiteResearcher-4B trained entirely locally at zero marginal cost.
  • The results suggest scalable RL training is a critical enabler for strong deep research capabilities beyond model scale alone.

A Synthetic Data

LiteResearcher synthesizes search tasks from webpage information and filters them for factual, specific, self-contained, and answerable QA pairs. The pipeline emphasizes verifiability and rejects ambiguous, open-ended, oversimple, or temporally vague questions.

  • The extraction prompt converts raw webpages into factual question–answer pairs with concise, verifiable answers.
  • The generation procedure requires questions to be context-specific, independently understandable, and focused on extractable factual data.
  • The pipeline outputs structured JSON containing each question, answer, and total pair count.
  • The rubric filters QA pairs using independence, answer specificity, unambiguity, answerability, non-open-endedness, non-oversimplicity, and time specificity.
  • Examples reject descriptive or open-ended answers, ambiguous wording, and vague temporal references such as “latest,” while accepting precise factual questions.
  • A question about a specific 2025 finals record passes because it is self-contained, unambiguous, answerable, and temporally constrained.

B.1 Data Composition

The data pipeline builds a large, diverse search corpus and generates multi-hop and factual training data from it. Processed trajectories combine synthesized, multi-hop, and open-source QA sources, with long contexts and broad domain coverage.

  • The SFT dataset contains 68,231 search trajectories, with 52.8% from the synthesis pipeline, 7.1% from multi-hop QA, and 40.2% distilled from open-source QA pairs.
  • The multi-hop synthesis pipeline searches entities, extracts factual features, discovers related entities, grows a subgraph, and formats a six-node graph.
  • It selects a target entity, converts graph edges into relationship constraints, optionally adds entity constraints, and composes a natural-language multi-hop question.
  • Corpus expansion starts from 10M pages and adds approximately 22M real-web pages through approximately 220K Serper calls costing $220.
  • The processing pipeline applies seven filtering rules and four cleaning steps before retaining trajectories for training.
  • The processed trajectories have a mean token length of 12.4K and mean interaction count of 8.7, with a long tail extending to approximately 45K tokens.
  • The final corpus spans 18 domain categories and more than 1M unique domains, with Academic, Regional, and Encyclopedia sources largest.

B.3 Training and Baseline Performance

Training uses long-context supervised initialization followed by staged RL with filtered data and correctness-based rewards. RL improves performance while reducing repetitive search behavior and maintaining a distributed rollout-training pipeline.

  • SFT uses 64K-token sequences, a learning rate of 2 × 10^-5, one epoch, 8×H100 GPUs, and effective batch size 128.
  • SFT provides a strong RL initialization but leaves substantial room for improvement on benchmarks requiring complex multi-step reasoning.
  • Stage 2 diversifies RL data with wiki-sourced queries, multi-hop subgraph reasoning, and science-domain data after difficulty-aware filtering.
  • Each rollout receives a binary correctness reward, with Qwen3-30B-A3B-Instruct serving as the semantic reward judge.
  • Repetitive action loops are a dominant post-SFT failure mode, causing repeated searches or URL revisits that consume the token budget without progress.
  • Mean reward rises from approximately 0.42 to 0.70 over 420 steps while response length falls from approximately 18K to 12K tokens and interaction turns from approximately 30 to 24.
  • The behavioral correction emerges from outcome-based rewards and GRPO clipping without explicit length or repetition penalties.

C.5 Training Dynamics

LiteResearcher uses a two-stage RL process in which Stage 1 establishes an initial gain over SFT and Stage 2 further improves accuracy while action selection becomes increasingly confident.

  • Training stages: Stage 1 trains on local-RAG-only data for steps 0–220, while Stage 2 continues from its checkpoint with the full data mixture.The sampling temperature changes from 0.7 in Stage 1 to 1.0 in Stage 2.
  • Accuracy: GAIA validation accuracy improves across the two stages, reaching a peak of ∼68%.Stage 1 establishes the initial gain over the SFT baseline, and Stage 2 further improves accuracy.
  • Policy behavior: Policy entropy decreases steadily in both stages, indicating increasingly confident action selection.Stage 2 has higher absolute entropy because it uses a higher sampling temperature.

D Infrastructure Details

The infrastructure section describes local search and browsing services, while RL suppresses repetitive inherited actions without explicit length or repetition penalties.

  • Local services: LiteResearcher’s local search and browse services are summarized in the infrastructure configuration table.
  • Action efficiency: RL reduces repetitive action behavior inherited from SFT without using an explicit length or repetition penalty.Mean reward rises from ∼0.42 to ∼0.70, while response length, interaction turns, and length clip ratio all decrease.
  • Training monitoring: Figure 9 tracks GAIA validation accuracy, policy entropy, tool calls per sample, and trajectory token length across training.The dashed vertical line marks the Stage 1→2 transition at step 220.

E Example Trajectories

The example trajectories illustrate multi-source verification, manual arithmetic, exhaustive enumeration, and self-derived problem solving across diverse deep-research tasks.

  • Overview: The presented rollouts were automatically judged correct and manually verified to contain no answer leakage.Raw trajectories span tens to over a hundred interaction steps and are rendered with thoughts, actions, and observations.
  • Xbench: The Xbench trajectory chains four domains and ends in date subtraction after online calculators fail to provide usable results.The agent computes the date difference manually using leap-year rules and cross-checks the result.
  • Xbench: 2965 days is obtained by combining leap-year arithmetic with a self-check that rejects an inconsistent calculator result.The agent computes 2922 days across eight years and adds 43 days to reach 2965.
  • GAIA: The GAIA trajectory resolves a 1002 versus 1037 source discrepancy before applying exact arithmetic.The agent selects the article-type count of 1002 and obtains the reference answer 41 by applying the ceiling.
  • Seal-0: The Seal-0 trajectory enumerates plausible former presidents, verifies them against primary sources, and identifies Van Buren and Fillmore as the only two satisfying all conditions.The answer is 2, with near-misses ruled out under the precise compound condition.
Loading 2604.17931v5…