PaperlayerView exact revision on arXiv

Source-linked AI summary

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh

arXiv:2511.19399v3cs.CLcs.AIcs.LG

TL;DR

Open deep research agents are often trained on constrained short-form tasks, while long-form research poses challenges for reliable reward signals amid under-specified criteria and evolving knowledge. DR Tulu uses rubrics that co-evolve with the policy during reinforcement learning; across four long-form benchmarks, it outperforms open agents and matches or exceeds proprietary systems.

  • Problem

    Training long-form deep research agents is challenging because questions are under-specified and knowledge-intensive, making fixed criteria inadequate and static rubrics liable to miss evidence.

    Source: §1 · p. 1, §3 · pp. 2–3

  • Method

    Reinforcement Learning with Evolving Rubrics updates rubrics during policy training using sampled responses and search traces to provide on-policy, discriminative feedback.

    Source: §1 · p. 2

  • Results

    By 4.8–41.8 percentage points across four long-form benchmarks, DR Tulu-8B outperforms the strongest open 8–32B models and matches or exceeds proprietary systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DR Tulu-8B and RLER provide a released model, data, rubrics, and training infrastructure to support reproducibility and future research on deep research agents.

    Source: §8 · p. 9

    Models with the highest training reward did not necessarily achieve the highest downstream evaluation performance.

    Source: §6.2 · p. 8, A. Discussion and Future Work · p. 15

Abstract

from arXiv · show

Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards, which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), where rubrics are constructed and maintained to co-evolve with the policy model during training. This allows the rubrics to incorporate newly explored information from search and contrasting model responses, enabling better fact checking and more discriminative on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first fully open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare, and general domains, DR Tulu substantially outperforms existing open deep research agents (by 15.6% over Tongyi DR on average) and matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average), while being significantly smaller and cheaper per query (1000x cheaper than OpenAI DR per query).

1. Introduction

Open-ended long-form deep research is difficult to train because response criteria are underspecified and evaluation requires extensive, up-to-date external information. DR Tulu-8B addresses this with rubrics that co-evolve with the policy during training.

Method

  • RLER constructs and updates rubrics from sampled model responses and search traces, incorporating newly discovered information to provide on-policy, discriminative feedback.DR Tulu-8B is first finetuned on high-quality, naturally occurring user data, then trained with RLER.Source: §1 · pp. 1–2

Results

  • DR Tulu-8B outperforms the strongest open 8–32B models, including Tongyi DR 30B (Team et al., 2025), by 4.8–41.8 percentage points across four long-form benchmarks.The benchmarks are AstaBench-ScholarQA-CS2 (SQAv2), DeepResearchBench, ResearchQA, and HealthBench.Source: §1 · p. 2
  • DR Tulu-8B matches or exceeds OpenAI DR, Perplexity DR, and Gemini3 Pro + Search, while costing USD 0.0019 per SQAv2 query versus USD 1.8 for OpenAI DR.The paper describes DR Tulu-8B as almost three orders of magnitude cheaper on SQAv2.Source: §1 · p. 2
  • On GeneticDiseasesQA, DR Tulu-8B exceeds or competes with proprietary deep research agents; other open agents cannot tackle the task because they lack reliable, verifiable citations.The paper introduces this clinical dataset to test whether models can search for and synthesize evidence to assess therapeutic eligibility of disease-causing genetic variants.Source: §1 · p. 2

Analysis

  • RLER adds 6.4–16.0 points over the finetuned model across four benchmarks, improving the comprehensiveness, depth, and citation accuracy of long-form responses.DR Tulu-8B selects search tools by task: it uses paper search 90% of the time on SQAv2 and web search and browsing about 55% of the time on DeepResearchBench.Source: §1 · p. 2

Contribution

  • The authors release data, code, models, dr-agent-lib, and an evaluation suite, providing an end-to-end training stack for scalable RL over long-horizon tool-use trajectories.The stack includes data and infrastructure for asynchronous tool calls, addressing a barrier created by the limited availability of deep research training resources.Source: §1 · p. 2

2. Preliminaries

The section formalizes deep research as a tool-using language-model process that ends in a final answer and introduces sample-wise rubrics as criteria for evaluating long-form responses.

Problem formulation

  • A deep research model samples think, tool, answer, or cite actions, appending generated content or tool observations to context until it produces its final answer.Tool calls take queries and arguments and return textual resources that can be cited.Source: §2 · p. 2

Rubrics as rewards

  • Sample-wise rubrics specify per-question natural-language criteria, with each rubric item assigned a weight (Viswanathan et al., 2025; Gunjal et al., 2025).Source: §2 · p. 2
  • A judge LM scores each rubric item as 0, 0.5, or 1, and reinforcement learning optimizes expected rubric scores across training questions.The paper says concrete, well-defined items reduce susceptibility to judge bias and support consistent, comparable scores across judge-model runs.Source: §2 · p. 2

3. RLER: Reinforcement Learning with Evolving Rubrics

RLER addresses the limits of fixed or closed-book rubrics for long-form deep research with instance-specific rubrics grounded in external knowledge and co-evolving with the policy. Its framework initializes rubrics from search, evolves them using policy rollouts, and manages the rubric buffer to retain discriminative criteria.

3. RLER: Reinforcement Learning with Evolving Rubrics

  • Long-form research questions admit many plausible quality criteria and require evolving external knowledge, so fixed rubrics and closed-book LM-generated rubrics can miss evidence and subtle errors or invite reward hacking.Source: §3 · pp. 2–3
  • RLER uses instance-specific, externally grounded rubrics that co-evolve with the policy; the rubric LM proposes rubrics from rollouts and the buffer, scores responses, and informs policy updates.Source: §3 · p. 3, Figure 2 · p. 3

3.1. Search-Augmented Evolving Rubrics

  • Rubric generation gains a generation–verification gap from search results for fact checking and multiple independently sampled responses for contrastive quality signals.Source: §3.1 · p. 3
  • Before training, each prompt receives persistent initial rubrics generated by an LM from the question and documents fetched through web search.Source: §3.1 · p. 3
  • During training, the rubric LM uses the prompt, sampled responses and search context, and existing rubric pool to generate positive rubrics for new strengths or knowledge and negative rubrics for undesirable behaviors such as copying retrieved text.Source: §3.1 · p. 3
  • After each GRPO rollout, RLER removes rubrics with zero reward variance and retains up to Kmax rubrics ranked by descending standard deviation to preserve discriminative value while limiting evaluation cost.Source: §3.1 · p. 3
  • RLER adds format, search, and citation rewards to encourage correct formatting, effective search use, and high-quality citations supporting relevant claims.Source: §3.1 · pp. 3–4

4. DR Tulu with Open Infrastructure and Training Recipe

DR Tulu-8B combines an extensible research-agent infrastructure with an SFT-then-RL recipe. After SFT cold-starts search behavior, customized GRPO trains the agent with real tool calls and evolving-rubric scoring.

4.1. DR Tulu Agent Infrastructure: dr-agent-lib

  • dr-agent-lib unifies local and API search and browsing through MCP, with cached high-concurrency execution and composable prompts; training uses search, browse, and paper-search workflows.Its backend uses asynchronous process locking for rate-limit-aware execution.Source: §4.1 · p. 4

4.2. Supervised Fine-Tuning for Cold Start

  • SFT cold-starts the model by distilling common search patterns from a teacher, improving early rollout quality and accelerating later RL training.Long-form prompts come from SearchArena (Miroyan et al., 2025) and OpenScholar (Asai et al., 2024), with LM-based 1–5 quality filtering; short-form prompts combine existing datasets and challenging synthetic prompts.Source: §4.2 · p. 4
  • GPT5 generates tool-use trajectories for SFT prompts; format and, for short-form prompts, gold-answer rejection filters yield 16K trajectories.Short-form answer filtering follows prior work (Jin et al., 2025; Li et al., 2025a).Source: §4.2 · p. 4

4.3. Online RL with Asynchronous Tool Calls

  • RL iteratively generates agentic rollouts with real tool calls and scores final answers against evolving rubrics using a customized GRPO variant.The RL stage focuses exclusively on long-form questions.Source: §4.3 · p. 4
  • RL uses approximately 5K filtered SearchArena and OpenScholar prompts plus 4K RaR prompts, which remain partially out-of-distribution from evaluation data.The prompts are long-form questions.Source: §4.3 · p. 4
  • GRPO training adds token-level loss, one-step asynchronous training, tool-output masking, sample packing, and immediate asynchronous tool dispatch during rollout generation.The implementation is based on Open-Instruct.Source: §4.3 · p. 4

5. Experimental Results

Across four long-form deep-research benchmarks, DR Tulu-8B outperforms open systems, matches or exceeds proprietary systems, and costs substantially less. On expert-curated genetic-variant research, it improves over Qwen3-8B + search and remains competitive with stronger proprietary systems on evidence metrics.

5.1. Experimental Settings

  • Evaluation uses four long-form, open-ended benchmarks spanning healthcare, scientific literature synthesis, and general-domain research, with human-written or human-verified rubrics and open, fixed-pipeline, and proprietary baselines.SQAv2 and DRB also report fine-grained metrics such as relevance, instruction-following, and citation precision and recall.Source: §5.1 · pp. 4–5

5.2. Main Results

  • 65.6 average, 15.6 points above Tongyi Deep Research-30B: DR Tulu-8B achieves the strongest performance among open deep-research models across four long-form benchmarks.Models trained primarily for constrained short-form tasks perform poorly on report-length generation; existing open baselines also lack citations, hurting SQAv2 scores where citation quality matters.Source: Table 1 · p. 5, §5.2 · p. 6
  • DR Tulu-8B achieves the strongest performance on SQAv2 and matches or exceeds proprietary systems on the other long-form benchmarks, while remaining competitive with OpenAI Deep Research overall.It outperforms Claude Sonnet Search, Perplexity Sonar (high-reasoning), and Perplexity Deep Research; it is built on an 8B open model.Source: §5.2 · p. 6
  • $0.00008/query for tool APIs and $0.0018/query including LM inference: DR Tulu-8B costs less than OpenAI Deep Research ($1.80/query) and Tongyi Deep Research ($0.03/query).Its adaptive tool use includes relying primarily on free paper search on SQAv2; on DRB it remains over 10× cheaper than Tongyi DR.Source: §5.2 · p. 6

5.3. Application: Researching Pathogenic Gene Variants

  • GeneticDiseasesQA contains 47 expert-curated questions on 24 pathogenic gene variants, requiring long-form citation-backed reports that synthesize databases, literature, and case reports.Evaluation covers Final Answer, Evidence Support, Evidence Quality, and Evidence Synthesis.Source: §5.3 · p. 6, Figure 3 · p. 7
  • On GeneticDiseasesQA, DR Tulu-8B improves over Qwen3-8B + search across all metrics and beats Ai2 ScholarQA on Final Answer correctness, while remaining competitive on evidence metrics against GPT-5 and Gemini systems.GPT-5 and Gemini-based systems achieve higher Final Answer scores; Evidence Support is computed from cited snippets, and baselines without traceable citations are excluded.Source: §5.3 · p. 6, Figure 3 · p. 7

6. Analysis

The analyses show that evolving rubrics improve RL performance, while SFT composition, stronger cold starts, and longer RL training shape downstream results. DR Tulu also generalizes to short-form QA and can use an open model as its rubric judge and generator.

6.1. Evaluation on Short-form QA Tasks

  • RL raises short-form QA average accuracy by 3.3 points over SFT, following SFT gains over the Qwen3-8B + Our Search baseline.Evaluation used GPT-4.1 as the LLM judge on SimpleQA, WebWalkerQA, and 2Wiki; SimpleQA and 2Wiki each used 1,000 randomly sampled questions.Source: §6.1 · p. 7, Table 2 · p. 7

6.2. Analysis on Training and Inference

  • Search-based static rubrics outperform general and closed-book rubrics, while a single shared general rubric underperforms the SFT baseline.The static-rubric RL ablation ran for 500 steps from an intermediate SFT checkpoint and used OpenScholar samples.Source: Table 3 · p. 7, §6.2 · pp. 8–9
  • Long-form SFT data is important for long-form benchmarks, while short-form data helps 2Wiki; long-form results improve with little data and largely saturate beyond 50%, whereas short-form tasks benefit up to the full dataset.Full-data SFT slightly reduces SQAv2 citation scores, but subsequent RL recovers citation performance.Source: §6.2 · p. 7, Figure 4 · p. 8
  • Even 5% high-quality SFT data gives a better RL cold start than no SFT, the full SFT mix improves further, and extended RL training can improve initially flat benchmarks.Training remains robust to persistent tool errors, and higher training reward does not necessarily imply higher downstream reward. Paper search dominates on SQAv2, while web search is primary for HealthBench, DeepResearchBench, and SimpleQA.Source: §6.2 · pp. 7–9, Figure 5 · p. 8
  • Removing evolving rubrics drops average performance by up to 2 points, with the gap widening during training; rubric-based rewards outperform random rewards, and citation reward is not the source of comparable gains.The widening gap reflects evolving rubrics capturing knowledge explored by the model. Runs with and without citation reward yield comparable performance.Source: Figure 6 · p. 8, §6.2 · pp. 8–9
  • Using Qwen3-8B as both judge and rubric generator gains 4.4 average points over SFT, 1.3 points below the GPT-judge configuration’s 5.7-point gain.The open-judge variant was evaluated after 1,000 RL steps; Qwen3-8B was also DR Tulu’s starting model.Source: §6.2 · pp. 8–9, Table 4 · p. 9
  • Tool use adapts to task information needs: paper search dominates SQAv2, whereas web search is primary for HealthBench, DeepResearchBench, and SimpleQA.The pattern is consistent with SQAv2’s focus on literature understanding and the broader open-web needs of the other tasks.Source: §6.2 · p. 9

7. Related Works

Prior deep-research agents often target short-form QA, while long-form systems commonly have workflow, tool-support, or release limitations; existing rubric methods are costly or weakly adaptive, unlike retrieval-grounded rubrics that co-evolve with the policy.

Deep research agents

  • Recent DR agents often target short-form QA (Jin et al., 2025; Liu et al., 2025; Team et al., 2025; Gao et al., 2025), while long-form systems often have static or proprietary workflows, limited tools, or incomplete releases (Li et al., 2025a;b; Prabhakar et al., 2025; Singh et al., 2025).The paper positions its work against these limitations.Source: §7 · p. 9

Rubric design for long-form generation

8. Conclusion

DR Tulu-8B and RLER provide an end-to-end framework for long-form deep research, released with resources to support reproducibility and further research.

8. Conclusion

  • DR Tulu-8B and RLER introduce an end-to-end training framework for long-form deep research tasks.The release includes the model, data, rubrics, and training infrastructure.Source: §8 · p. 9
  • The released model, data, rubrics, and training infrastructure are intended to support reproducibility and future research on deep research agents.Source: §8 · p. 9
  • Future directions include adaptive verifier design, scaling privileged information for judges, aligning training rewards with downstream evaluations, and extending agents to specialized scientific workflows.Source: §8 · p. 9

Impact Statement · Appendix

The paper introduces RLER and DR Tulu-8B as an open deep research agent and frames the model primarily as a research artifact. It highlights potential benefits while calling for safeguards against errors, selective citation, bias, and misuse.

Impact Statement

  • DR Tulu-8B is an open deep research agent trained using Reinforcement Learning with Evolving Rubrics (RLER).The paper introduces both RLER and the model.Source: Impact Statement · p. 10
  • The model, training data, evaluation rubrics, and agent infrastructure are fully open-sourced to support reproducibility and further research.Source: Impact Statement · p. 10
  • Potential benefits include broader access to long-form research capabilities, improved reproducibility, and more rigorous evaluation across science, healthcare, and general domains.Source: Impact Statement · p. 10
  • Research assistants may generate plausible but incorrect claims, selectively cite sources, or produce biased synthesis.Source: Impact Statement · p. 10
  • Without safeguards, such systems could scale misinformation or low-quality research outputs.Source: Impact Statement · p. 10
  • The paper views DR Tulu primarily as a research artifact and recommends careful evaluation, transparency about uncertainty and sources, and domain-appropriate human oversight, especially in high-stakes settings.Source: Impact Statement · p. 10

A. Discussion and Future Work … G.1. SFT Hyperparameters

This section discusses the evolving-rubric approach, its relation to prior work, formulation, analyses, limitations, and future directions. It also provides supporting details on SFT and RL data construction, filtering, a toy study, and SFT hyperparameters.

A. Discussion and Future Work

  • RLER scales verifier compute by enriching its information with contrasting responses, searches, and policy reasoning, while evolving rubrics incorporate evidence discovered during training.The authors argue that more information can extend verifier capabilities beyond additional reasoning tokens alone, though it increases context length and compute costs.Source: A. Discussion and Future Work · pp. 14–15
  • The authors identify train-test mismatch as a limitation: higher training reward did not always mean better downstream performance, potentially because judges, rubrics, and evaluation setups differ.They also report generalization to GeneticDiseasesQA without task-specific training and propose integrating specialized data sources or enabling flexible use of unseen tools.Source: A. Discussion and Future Work · p. 15

B. Related Work

  • The authors position their framework against short-form agents and fixed long-form pipelines, emphasizing open code and data, multi-tool search, citations, and end-to-end training for realistic long-form tasks.They distinguish their externally grounded, co-evolving rubrics from static reference-based rubrics and closed-book online rubric approaches.Source: B. Related Work · pp. 15–16

C. Problem Formulation for Deep Research

  • The deep research policy acts through think, tool, answer, and cite actions, continuing tool-mediated state updates until it produces a final answer.The RLER overview describes training on rubrics that co-evolve with the policy and are grounded in searched knowledge.Source: C. Problem Formulation for Deep Research · p. 16, §D.1 · p. 17

D.2. How Do Evolving Rubrics Work?

  • Over 50% of initial and evolving search-based rubric claims are assertive, versus 22% for closed-book instance-wise rubrics, with search grounding supporting specificity and factuality.The study measures assertiveness and factuality using an LM; general rubrics lack specific evaluation criteria.Source: Table 6 · p. 18, §D.2 · p. 18
  • Evolving rubrics adapt criteria to policy behavior: in a toy case, they penalized irrelevant Python code in responses and reduced its occurrence during training.Static initial rubrics did not anticipate this behavior, an artifact also reported for the Qwen model (Shao et al., 2025).Source: §D.2 · p. 18, Figure 9 · p. 18

D.5. Citation, Search, and Format Rewards

  • Auxiliary rewards assess citation support and relevance, cap search-call rewards at three, and reward well-formed answer, citation, and query formats.The format reward is weighted 0.5 for answer format, 0.3 for citation format, and 0.2 for query format.Source: §D.5.1 · p. 21, §D.5.2 · p. 22, §D.5.3 · p. 22

E.1. Rubric Specificity Analysis

  • Rubric specificity analysis used a search-enabled GPT-4O-SEARCH-PREVIEW model to classify assertiveness and check factuality, because the task requires factual knowledge.The authors describe this model as useful for simple fact checking, though less competitive than a deep research model.Source: §E.1 · p. 22

E.2. Toy Case Study on Evolving Rubrics

  • For the toy study, the authors trained Qwen3-8B for extended epochs on one closed-book RAG survey query, comparing evolving rubrics with an initial-only rubric that asked for the first paper proposing RAG.The setup was intended to echo how static rubrics can be under-specified; the supplied passage does not report comparative outcomes.Source: §E.2 · p. 24

F.1.1. PROMPT CURATION

F.1.2. TRAJECTORY GENERATION

  • SFT trajectories contained reasoning traces, tool calls, and cited answers; GPT-5 generations used Google search, web browsing, and paper search, with a 15-tool-call cap and workflow-format filtering.Short-form verifiable QA examples also required answer agreement by F1 overlap above 0.9 or an LLM judge; GPT-5 generated a few hundred interleaved search-and-think BrowseComp-style traces. Ai2 ScholarQA traces were transformed into iterative-search data by adding GPT-4.1-generated section sub-queries and interleaving retrieved papers with reasoning and final cited answers.Source: §F.1.2 · pp. 24, 28

F.2. RL Data Construction

  • Initial RL rubrics were generated with GPT-4.1-mini from the top 10 search results for each prompt, using paper search for OpenScholar and Google search plus web browsing for SearchArena.Source: §F.2 · p. 28

F.3. Onpolicy SFT Data Construction

  • The paper reports that DR Tulu SFT trajectories were sampled from the original SFT checkpoint, then filtered; the final RL run instead initialized from that original checkpoint because it performed better than on-policy SFT initialization.For long-form prompts, trajectories were retained only if rubric coverage and citation precision each exceeded 0.6, in addition to lightweight filtering.Source: §F.3 · p. 28

G.1. SFT Hyperparameters

  • The SFT training hyperparameters are provided in Table 8, but their values are not specified in the supplied passages.Source: §G.1 · p. 28, Table 8 · p. 30

G.2. RL Training Details and Hyperparameters … H.1. Prompts for DR Tulu

The appendix documents DR Tulu’s RL training setup, runtime constraints, and prompt and tool references. The main run used 4,000 steps over 70 days, while prompt figures and a supported-tools table document implementation materials.

G.2. RL Training Details and Hyperparameters

  • Training used standard GRPO with token-level loss aggregation, sample packing, one-step asynchronous training, tool-output loss masking, and a 0.001 KL penalty.Rubric buffer management occurred after rollout generation and reward computation; one-step asynchronous training used rollouts from a policy one step behind the current policy.Source: §G.2 · p. 30
  • Citation rewards were turned off after 650 steps, then restored for steps 3,350–4,000 after citation performance dropped.The authors report that the reward had converged by step 650 and further use slowed training because of API calls; ablations also turned it off after 650 steps.Source: §G.2 · p. 30
  • Asynchronous tool calling overlapped tool requests with generation, while dr-agent-lib controlled concurrent API calls and cached repeated queries.A generation request sleeps after a tool call is sent, allowing the inference engine to work on other responses while awaiting the tool response.Source: §G.2 · p. 30
  • The final run used about 27,000 GPU hours for 4,000 steps (14.5 epochs) over 70 days; performance began saturating around 4,000 steps, with rollouts limited by API rate limits.The authors stopped the main run at that point, though they note further training might yield slight improvements; increasing compute did not improve RL training speed. Hyperparameters are provided in Table 9.Source: §G.2 · p. 30, Table 9 · p. 31

G.3. Prompt used for General Rubric Training

  • Figure 18 presents the system prompts used for general rubric training.Source: §G.3 · p. 30

H.1. Prompts for DR Tulu

H.2. Evaluation Details of Baseline Models … I.7.4. ROBUSTNESS TO THE BROWSER TOOL USED AT INFERENCE

This section details baseline and benchmark evaluation protocols, training ablations, and analyses of inference-time performance and evaluation variance. It also examines the effect of changing the inference-time browser on DR Tulu-8B’s benchmark performance.

H.2. Evaluation Details of Baseline Models

  • Evaluations use official code for Asta-ScholarQA-CS2 and DeepResearchBench; Asta scores 100 test questions with gemini-2.5-flash judging, while DeepResearchBench assesses articles answering 50 English and 50 Chinese questions across four dimensions.HealthBench evaluation randomly samples 1,000 cases, while specialized inference-pipeline models are excluded from short-form evaluation because they cannot follow answer-only instructions.Source: §H.3 · p. 32, §H.2 · p. 32

H.4. Details of Pathogenic Gene Variants Evaluation

  • The pathogenic-variant evaluation reformats expert-curated data into 47 questions and scores answer facts, requested evidence quality, source support, and synthesis across sources.The 24 variants were assessed by two working-group members, and GPT-4.1 judged the criteria without additional training.Source: §H.4 · pp. 35–36

I.1. Performance and Cost Breakdown

  • RLER improves DR Tulu over SFT by 11.0 rubric-coverage points, 7.8 answer-precision points, 7.9 comprehensiveness points, and 9.2 depth points.On SQAv2, citation precision and recall also rise by 25.2 and 20.0 points, respectively.Source: §I.1 · p. 36

I.2. Full RL Training Curves

  • Training showed three phases of sequence length and tool-call changes, while performance generally continued improving despite periods of search-tool errors.The authors hypothesize an initial unlearning of unsuccessful SFT behaviors followed by exploration, with later declines potentially reflecting output-token and tool-call limits.Source: §I.2 · p. 36, Figure 26 · p. 43

I.4. Ablation on the Citation Reward

  • Without citation reward, the step-1300 run averages 61.2 versus 60.6 with citation reward, indicating rubric reward drives most gains in this regime.The format reward saturates at 1.0 early; the authors also report that on-policy SFT slightly boosts SFT performance but weakens later RL performance on HealthBench and SQAv2 versus the regular SFT mixture.Source: §I.3 · p. 42, §I.4 · p. 43, Table 13 · p. 43

I.6. Effect of the tool-call budget at inference time.

  • Inference performance typically saturates around five allowed tool calls for both SFT and RL models, though RL sometimes benefits from budgets up to ten.Open-model costs are estimated from token counts, published model pricing, and average tool-use costs; proprietary costs use billed API rates, with search costs included for Gemini despite its daily waiver.Source: §I.5 · p. 43, §I.6 · p. 44, Figure 27 · p. 44

I.7.1. VARIANCES INTRODUCED BY TOOLS

  • Repeated Google searches are unstable: near-concurrent calls share about 7.67 of the top-10 snippets on average, with only 5.64 exact rank matches, and week-apart calls diverge more.The study compares 100 queries reissued about a week apart with near-concurrent calls.Source: §I.7.1 · p. 44, Table 14 · p. 44

I.7.2. VARIANCES INTRODUCED BY INFERENCE

  • Across 300 2Wiki cases, GPT-4.1’s two trajectory answers differ by 29.3%; Healthbench and ResearchQA show high absolute differences in LLM-judged scores, while evaluating the same trajectories twice yields relatively high consistency.Source: §I.7.2 · p. 46, Table 15 · p. 46, §I.7.3 · p. 46

I.7.4. ROBUSTNESS TO THE BROWSER TOOL USED AT INFERENCE

  • −0.5 average benchmark points: changing the inference-time browser causes at most a 1-point change on each benchmark, with no retraining required.DR Tulu-8B was trained with Crawl4AI and evaluated with Jina; it was also re-evaluated using Crawl4AI at inference.Source: Table 17 · p. 46, §I.7.4 · p. 46

I.8. Mismatch between RL Training and Downstream Evaluation

The section describes a mismatch between RL training reward and downstream performance: the highest-reward model underperforms, and the authors propose several possible causes.

I.8. Mismatch between RL Training and Downstream Evaluation

  • The No SFT model achieves higher RL training reward but performs worse downstream and uses more tool calls than Our SFT.Figure 29 compares training metrics for the two starting models; the downstream comparison is referenced in the text.Source: §I.8 · pp. 46–47, Figure 29 · p. 47
  • Possible causes include mismatched training and evaluation rubrics, different judge models, and model priors that may encourage reward hacking.The in-loop judge is GPT-4.1-mini, while downstream evaluations use different models, prompts, and harnesses; the authors defer deeper investigation.Source: §I.8 · p. 47
  • When models start from the same initial model, reward improvements usually correlate well with downstream scores.The authors contrast this pattern with the mismatch observed across starting models.Source: §I.8 · p. 47

J. Analysis on Searched Tools and Domain Distributions · J.1. Qualitative Examples

DR Tulu adapts its search behavior to task information needs, with retrieved domains aligning to each benchmark’s subject matter. The appendix also presents example trajectories for OpenScholar SFT data and DR Tulu on ResearchQA.

J. Analysis on Searched Tools and Domain Distributions

  • Scientific-paper search dominates DR Tulu’s tool use on SQAv2, matching the task’s literature-understanding focus.The analysis uses an intermediate RL checkpoint at step 1900.Source: J. Analysis on Searched Tools and… · p. 47
  • Figure 31’s 100-sample-per-task search results show top domains matching the evaluation domain across SQAv2, HealthBench, DeepResearchBench, and SimpleQA.Source: Figure 31 · p. 48
  • HealthBench searches emphasize authoritative biomedical and public-health sites, while DeepResearchBench mixes technical and policy sources.Examples include cdc.gov and ncbi.nlm.nih.gov for HealthBench, and oecd.org and github.com for DeepResearchBench.Source: J. Analysis on Searched Tools and… · p. 48
  • SimpleQA searches are dominated by general-reference and social or information platforms, whereas literature-centric tasks favor scholarly venues.Examples for SimpleQA include en.wikipedia.org, facebook.com, and youtube.com.Source: J. Analysis on Searched Tools and… · p. 48

J.1. Qualitative Examples

Related papers

Loading 2511.19399v3…