PaperlayerView exact revision on arXiv

Source-linked AI summary

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh

arXiv:2511.19399v3cs.CLcs.AIcs.LG

TL;DR

Open deep research agents often train on constrained short-form tasks, while realistic long-form research requires reliable, knowledge-grounded rewards. The paper introduces RLER, which evolves rubrics from search evidence and contrasting rollouts, and uses it to train DR Tulu-8B. Across four long-form benchmarks, DR Tulu-8B outperforms open agents and matches or exceeds proprietary systems while using far lower query cost.

  • Problem

    Existing open deep research agents largely target constrained short-form question answering, while open-ended long-form training requires reliable rewards for under-specified, knowledge-intensive responses.

  • Method

    RLER constructs and maintains instance-specific rubrics that co-evolve with the policy using externally retrieved knowledge and multiple sampled responses.

  • Results

    Across four long-form benchmarks, DR Tulu-8B averages 65.6, exceeding Tongyi Deep Research-30B by 15.6 points and matching or exceeding proprietary systems.

  • Takeaways & Limitations

    The released model, data, rubrics, code, infrastructure, and evaluation suite provide an end-to-end open training stack for long-form deep research.

  • Takeaways & Limitations

    Higher training reward did not necessarily correspond to higher downstream evaluation performance, reflecting mismatch between training and external evaluation setups.

Abstract

from arXiv · show

Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards, which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), where rubrics are constructed and maintained to co-evolve with the policy model during training. This allows the rubrics to incorporate newly explored information from search and contrasting model responses, enabling better fact checking and more discriminative on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first fully open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare, and general domains, DR Tulu substantially outperforms existing open deep research agents (by 15.6% over Tongyi DR on average) and matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average), while being significantly smaller and cheaper per query (1000x cheaper than OpenAI DR per query).

1. Introduction

Deep research agents must plan, search, and synthesize well-attributed long-form answers, but existing open approaches largely target constrained short-form tasks. The paper introduces DR Tulu-8B and RLER, achieving strong benchmark performance, tool-use adaptation, and substantially lower query cost.

  • Existing open deep research agents mainly use training-free prompts or reinforcement learning with verifiable rewards for constrained short-form question answering.These approaches do not directly address realistic open-ended, long-form research tasks.
  • DR Tulu-8B is trained end-to-end for open-ended, long-form deep research using Reinforcement Learning with Evolving Rubrics.The model is first finetuned on naturally occurring user data, then trained with RLER.
  • 4.8–41.8 percentage points: DR Tulu-8B outperforms strongest open 8–32B models across four long-form deep research benchmarks.The comparison includes Tongyi DR 30B and benchmarks spanning scientific, general-domain, and healthcare research.
  • USD 0.0019 per query: DR Tulu-8B is almost three orders of magnitude cheaper than OpenAI DR at USD 1.8 on SQAv2.The paper also releases data, code, models, an agent library, and an evaluation suite.
  • 6.4–16.0 points: RLER improves the finetuned model across four benchmarks while supporting task-dependent search-tool selection.The model uses paper search 90% of the time on SQAv2 and web search and browsing about 55% of the time on DeepResearchBench.

2. Preliminaries

The paper formulates deep research as language-model interaction with search-related tools and defines rubric-based rewards for evaluating long-form responses. Rubrics provide explicit, example-specific criteria scored by a judge model during reinforcement learning.

  • Deep research models use search-related tools that return textual resources which can be cited in final answers.The action space includes think, tool, answer, and cite, with interaction continuing until the model produces an answer.
  • A deep research trajectory appends model actions and tool observations to context until the answer action produces the final response.Think, answer, and cite outputs are appended directly, while tool actions execute calls and append returned observations.
  • Sample-wise rubrics specify natural-language evaluation criteria separately for each question and assign each rubric item a weight.The rubric for question x is represented as a weighted set of rubric items.
  • A judge LM scores each rubric item as 0, 0.5, or 1 according to how well the final response satisfies it.Reinforcement learning optimizes the expected rubric score over training questions.
  • Explicit rubric items reduce susceptibility to judge-model bias and support consistent, comparable evaluation across judge runs.Their concrete criteria make response assessment more objective than an unspecified overall judgment.

3. RLER: Reinforcement Learning with Evolving Rubrics

RLER addresses the difficulty of scaling reliable rewards for knowledge-intensive, open-ended research by grounding instance-specific rubrics in external knowledge and evolving them with policy rollouts. Its rubric generator uses search and response contrasts, while buffer management preserves discriminative criteria and auxiliary rewards encourage usable research behavior.

  • Motivation: Long-form research questions admit many plausible quality criteria, making fixed rubrics inadequate and closed-book generated rubrics vulnerable to missed evidence and reward hacking.The problem is intensified by the broad, evolving external knowledge required for deep research.
  • Core idea: RLER training samples policy rollouts, generates rubrics from them and the current rubric buffer, scores the rollouts, and updates the policy.The rubric buffer is then expanded and pruned using rollout-score variance.
  • Core idea: RLER uses instance-specific, externally grounded rubrics that co-evolve with the policy model during training.The framework is designed specifically for long-form deep research.
  • Search-augmented evolution: Privileged external search knowledge supports fact verification, while multiple independently sampled responses provide contrastive signals for relative quality assessment.These signals create a generation–verification gap between policy generation and rubric verification.
  • Rubric initialization: Initial rubric buffers are built per prompt by searching for documents and feeding those documents with the prompt to a rubric-generation LM.The resulting persistent rubrics remain available throughout RL training.
  • Online evolution: Rubric evolution concatenates each prompt, sampled responses with search contexts and answers, and the existing rubric pool before generating new rubrics.The new rubrics are added to the active scoring buffer.
  • Search-augmented evolution: RLER generates positive rubrics for newly discovered strengths or relevant knowledge and negative rubrics for recurring undesirable behaviors such as citation-driven copying.Negative rubrics can suppress verbatim copying of retrieved content to maximize citation precision.
  • Buffer and auxiliary rewards: The buffer removes zero-variance rubrics, ranks remaining rubrics by score standard deviation, and retains at most K_max criteria; auxiliary format, search, and citation rewards supplement rubric scores.The buffer strategy limits evaluation cost while preserving discriminative rubrics.

4. DR Tulu with Open Infrastructure and Training Recipe

DR Tulu-8B is trained from Qwen3-8B with an open agent infrastructure, supervised fine-tuning cold start, and RLER-based reinforcement learning over real tool-use rollouts. The recipe combines diverse long-form prompts, filtered teacher trajectories, and efficiency-oriented asynchronous training components.

  • DR Tulu-8B starts from Qwen3-8B and follows an SFT-then-RL recipe built on RLER.The training framework is paired with an open infrastructure stack for deep research.
  • dr-agent-lib integrates search and browsing tools through a unified MCP backend with concurrency, caching, locking, and composable prompts.These features support scalable, rate-limit-aware tool execution and configurable search workflows.
  • Supervised fine-tuning distills common search patterns from a teacher model to improve early rollout quality and accelerate later reinforcement learning.The paper evaluates this cold-start choice through ablations.
  • Long-form prompts come from SearchArena and OpenScholar, are filtered by an LM, and cover general-domain and scientific questions.Short-form prompts are additionally drawn from existing datasets.
  • GPT5 teacher trajectories include simulated reasoning, tool use, and final answers, with rejection filters for format compliance and short-form answer correctness.The filtering process yields 16K SFT trajectories.
  • RL training uses real tool calls and scores final answers against evolving rubrics with a customized GRPO variant.RL focuses exclusively on long-form questions drawn from multiple sources, including approximately 5K SearchArena/OpenScholar prompts and 4K RaR prompts.
  • The training implementation adds token-level loss, asynchronous training and tool calling, tool-output masking, and sample packing for efficiency.Tool requests are dispatched immediately when triggered during rollout generation rather than waiting for batch completion.

5. Experimental Results

Experiments evaluate DR Tulu-8B across four long-form benchmarks and an expert-curated clinical-genetics task, comparing open, fixed-pipeline, and proprietary systems. DR Tulu-8B achieves the strongest open-model performance, matches or exceeds proprietary systems, and offers a substantial cost advantage.

  • Experimental Settings: The evaluation uses four long-form benchmarks in healthcare, scientific literature synthesis, and general-domain deep research, with human-written or human-verified rubrics.The experiments compare multiple open deep research models, fixed-pipeline systems, and proprietary systems using official prompts and report-style outputs.
  • Main Results: Across four open-ended long-form benchmarks, DR Tulu-8B achieves an average score of 65.6, exceeding Tongyi Deep Research-30B by 15.6 points.
  • Main Results: DR Tulu-8B outperforms open fixed-pipeline systems on every benchmark despite using a smaller 8B backbone than WebThinker-32B.WebThinker-32B’s report mode improves performance by 21.3 points versus its default mode but still trails DR Tulu-8B.
  • Main Results: DR Tulu-8B matches or exceeds proprietary deep research systems across long-form benchmarks and achieves the strongest performance among all systems on SQAv2.It outperforms Claude Sonnet Search, Perplexity Sonar (high-reasoning), and Perplexity Deep Research, while remaining competitive with OpenAI Deep Research.
  • Main Results: $0.00008/query in tool-only costs and $0.0018/query including LM inference make DR Tulu-8B substantially cheaper than OpenAI Deep Research at $1.80/query.Ai2 ScholarQA using Claude Sonnet costs $1.30/query on SQAv2.
  • Application: Researching Pathogenic Gene Variants: GeneticDiseasesQA contains 47 expert-curated questions covering 24 pathogenic gene variants and evaluates final answers, evidence support, evidence quality, and evidence synthesis.The task requires citation-backed reports aggregating biological databases, research literature, and case reports.
  • Application: Researching Pathogenic Gene Variants: On GeneticDiseasesQA, DR Tulu-8B improves over Qwen3-8B across all metrics and outperforms Ai2 ScholarQA on Final Answer correctness.GPT-5 and Gemini-based systems score higher on Final Answer, while DR Tulu-8B remains competitive on the three evidence-focused metrics.

6. Analysis

The analysis finds that mixed SFT supervision supports both long- and short-form behavior, while search-grounded and evolving rubrics improve RL outcomes. Stronger SFT initialization, extended RL, and task-adaptive tool use further shape performance.

  • SFT data: Mixed long-form and short-form SFT data is important: removing long-form data degrades long-form benchmarks, while removing short-form data hurts short-form tasks.Long-form performance largely remains unchanged without short-form data, but 2Wiki performance noticeably declines.
  • Rubrics: Search-based static rubrics outperform general and closed-book rubrics during RL because they are grounded in retrieved, up-to-date information.The comparison uses 500 RL steps from an intermediate SFT checkpoint.
  • SFT data: Long-form benchmarks improve substantially with as little as 5% of the SFT data and largely saturate beyond 50%, while short-form tasks continue benefiting up to the full dataset.The strongest continued short-form gains occur on 2Wiki.
  • RL training: Starting RL from even 5% high-quality SFT data outperforms starting directly from Qwen3, while the full SFT mixture further improves performance and longer RL can recover initially flat evaluations.The analysis traces performance through 4000 RL steps.
  • Training behavior: Higher training reward does not necessarily correspond to higher downstream reward, and training remains robust to tools that consistently error.The robustness finding comes from extended training with a consistently failing tool.
  • Rubrics: Removing evolving rubrics causes up to a 2-point performance drop, with the gap widening as RLER captures newly explored knowledge during training.Both evolving and static search-based rubric approaches outperform random rewards.
  • Judge models: Qwen3-8B can serve as both judge and rubric generator: it gains +4.4 average points over SFT, 1.3 points behind the GPT-judge configuration.This shows RLER can improve without requiring a stronger proprietary judge.
  • Tool usage: Tool use adapts to task information needs: paper search dominates SQAv2, whereas web search is primary for HealthBench, DeepResearchBench, and SimpleQA.The pattern aligns scientific-literature questions with paper search and broader information needs with web search.

7. Related Works

Prior deep research agents often target short-form QA or rely on static workflows and proprietary components. Rubric-generation methods similarly remain costly, ungrounded, fixed, or weakly suited to knowledge-intensive long-form tasks.

  • Deep research agents: Recent deep research systems commonly focus on short-form QA, while long-form systems often use static workflows, proprietary components, limited tools, or unreleased code and data.These limitations motivate an end-to-end open training framework for long-form research.
  • Rubric design: Existing rubric methods include human-written, static, closed-book, and learned-critic approaches, but automating scalable rubric generation remains difficult for under-specified, knowledge-intensive questions.Fixed criteria may be inadequate when long-form questions admit many plausible notions of quality.

8. Conclusion

The paper presents DR Tulu-8B and RLER as an end-to-end framework for long-form deep research, releasing the model, data, rubrics, and infrastructure for reproducibility and future work.

  • Contribution: DR Tulu-8B and RLER provide an end-to-end training framework for long-form deep research tasks.The conclusion identifies adaptive verification, privileged information for judges, reward alignment, and specialized scientific workflows as future directions.
  • Contribution: The release includes the model, data, rubrics, and training infrastructure to support reproducibility and future deep research research.The paper explicitly frames these resources as support for future research on deep research agents.

Impact Statement

The impact statement presents DR Tulu as an open research artifact that may broaden access and reproducibility while requiring safeguards against plausible errors, selective citation, bias, and misinformation.

  • Potential benefits: Open-sourcing DR Tulu may enable broader access to long-form research capabilities, improve reproducibility, and support more rigorous evaluation.These are identified as potential positive impacts of the released model and infrastructure.
  • Risks and safeguards: Deployment requires uncertainty and source transparency plus domain-appropriate human oversight, especially in high-stakes settings.The statement warns that research assistants can generate plausible but incorrect claims, selective citations, or biased synthesis.

A. Discussion and Future Work

The discussion presents evolving rubrics as adaptive, knowledge-expanding verification criteria while identifying train-test mismatch and broader domain-tool integration as important boundaries and directions.

  • Evolving Rubrics: RLER updates rubric criteria by contrasting current policy rollouts, adapting training difficulty to the model’s evolving behavior.The approach uses evolving rubrics to make verification criteria better distinguish the model’s current outputs.
  • Verifier Scaling: RLER provides the verifier with contrastive responses, external searches, and process information, enriching the information available for verification.The paper frames this as a way to scale verifier compute through privileged information rather than reasoning tokens alone.
  • Knowledge Coverage: Search-based rubrics expand beyond the generator’s parametric knowledge, while evolving rubrics incorporate evidence discovered through policy rollouts.This enables evaluation criteria to evolve with the model’s distribution.
  • Challenges: Train-test mismatch limits the reliability of training reward as a proxy for downstream performance because training and external evaluation use different judges, rubrics, and setups.The authors note that reward hacking toward training-judge preferences can occur and emphasize customization of open models for downstream tasks.
  • Future Work: Experiments indicate that RLER generalizes to GeneticDiseasesQA without task-specific training, while future scientific applications may require structured tools and non-text modalities.The paper specifically mentions genomic sequences, molecular structures, and transcriptomics as examples.
  • Related Work: Prior open systems commonly focus on short-form outputs, single search tools, limited citations, or incomplete releases, while rubric methods often rely on static or closed-book criteria.The paper positions RLER and its release as responses to these limitations in open deep research.

F.2. RL Data Construction

The RL data pipeline constructs initial search-grounded rubrics and trajectories, filters training examples using rubric and citation quality, and documents the associated prompts and SFT settings.

  • Initial Rubrics: Initial rubrics are generated from retrieved results using paper search for OpenScholar queries and Google search plus browsing for SearchArena queries.The process retrieves the top 10 search results for SearchArena prompts.
  • Checkpoint Selection: The final RL run initializes from the original SFT checkpoint because it ultimately performs better than the checkpoint improved with on-policy SFT data.The comparison is reported with reference to Figure 5.
  • Trajectory Generation: The pipeline generates 2–4 trajectories per prompt from the DR Tulu SFT checkpoint using the evaluation-time inference pipeline.Trajectory generation uses randomly sampled prompts from the initial SFT dataset and dr-agent-lib.
  • Rejection Sampling: Rubric-based and citation-based rejection filters retain long-form trajectories scoring above 0.6 on both rubric coverage and citation precision.The offline filtering scheme mirrors the RL reward design during data generation.
  • Documentation: The appendix documents the trajectory-generation prompt, system prompts, and SFT hyperparameters used in the data-construction pipeline.The supplied passages identify these materials but do not provide their full contents.

G.2. RL Training Details and Hyperparameters

RL training uses GRPO with efficiency optimizations, asynchronous tool execution, and documented infrastructure and hyperparameters for long-horizon research trajectories.

  • Optimization: RL optimization uses standard GRPO with token-level loss aggregation, sample packing, and one-step asynchronous training.The one-step scheme trains on rollouts from a policy one step behind the current policy.
  • Tool Execution: Asynchronous tool calling overlaps inference and tool execution by placing waiting generation requests to sleep while other responses continue.The calls are mediated by the custom dr-agent-lib infrastructure.
  • Training Scale: The main run lasted 70 days and used roughly 27000 GPU hours for 4000 training steps, with performance beginning to saturate around 4000 steps.Increasing compute did not improve training speed because rollout API rate limits were a bottleneck.
  • Configuration: The section provides the GRPO hyperparameters, supported-tool list, and system-prompt references for the training setup.The supplied materials identify these resources without listing their full parameter values or prompt contents.

H.2. Evaluation Details of Baseline Models

The evaluation compares open, closed, fixed-pipeline, and retrieval-based systems across long- and short-form tasks, using standardized prompts, tools, and benchmark-specific judging procedures. DR Tulu uses iterative reasoning, search, browsing, citations, and RL-trained behavior, while evaluations also document training dynamics and tool-error robustness.

  • Baseline systems: Open baselines use official implementations and default settings, with WebThinker evaluated in both default and report modes.The compared open systems include WebExplorer, Tongyi Deep Research, ASearcher, and WebThinker.
  • Baseline systems: Closed-system baselines include GPT-5 + Search and Gemini3 Pro + Search, using search and URL-content tools provided by their APIs.The GPT-5 + Search and Gemini3 Pro + Search prompts are documented in Figure 21.
  • Evaluation protocol: Benchmark evaluation uses task-specific protocols, including rubric, precision, recall, and judge-based metrics across scientific, healthcare, and general-domain tasks.SQAv2 uses 100 questions, ResearchQA uses 776 official questions, HealthBench samples 1000 cases, and DeepResearchBench evaluates 100 questions.
  • DR Tulu procedure: DR Tulu’s system prompt supports iterative reasoning, repeated tool calls, final answer tags, and retrieved evidence for non-trivial claims.The answer format requires exact snippet IDs inside citation tags.
  • Inference setup: The unified inference pipeline uses Google search, web browsing, and paper search for long- and short-form tasks without task-specific customization.Search uses Serper, browsing uses Jina, and paper search uses the Semantic Scholar full-text API.
  • Training observations: Training analyses report consistent RLER gains across content and citation dimensions, while performance generally continued improving despite server-side search errors.The reported improvements include rubric coverage, answer precision, comprehensiveness, depth, citation precision, and citation recall.

I.3. On-Policy SFT and RL Results

The appendix examines on-policy SFT, qualitative outputs, reward ablations, tool-call budgets, evaluator variability, and the relationship between RL training rewards and downstream performance. These analyses show that RLER improves multiple response and citation measures, while some training choices and evaluation signals remain imperfectly aligned.

  • Qualitative examples: DR Tulu-8B’s example outputs include citation-backed genetic-disease reports with explicit uncertainty about mechanisms lacking direct functional evidence.The example discusses CACNA1A p.Val1392Met and identifies missing patch-clamp or Ca2+ imaging data.
  • On-policy SFT: On-policy SFT slightly improves initial SFT performance but weakens later RL performance on HealthBench and SQAv2.The comparison includes the on-policy-trained model, the original SFT set, an undertrained model, and no SFT.
  • Reward ablations: Citation-reward ablation produces comparable final performance, with the run without citation reward slightly ahead at 61.2 versus 60.6 average score.The authors conclude that rubric-based reward accounts for most gains in this regime, while noting that a full auxiliary-reward sweep was not performed.
  • Tool-call budget: Performance typically saturates near five inference-time tool calls, although RL sometimes benefits from budgets of up to ten calls.During RL training, the model averages 3–4 tool calls per sample.

J. Analysis on Searched Tools and Domain Distributions

The analysis shows that DR Tulu adapts retrieval behavior to task information needs and illustrates its research trajectories on ResearchQA and OpenScholar prompts. Domain-specific search patterns connect tool selection with the scientific, healthcare, general, and factoid settings represented in evaluation.

  • Tool adaptation: DR Tulu’s intermediate RL checkpoint adapts tool usage to task information needs, with paper search dominating on the literature-focused SQAv2 benchmark.The observed behavior is consistent with SQAv2’s emphasis on scientific-paper understanding.
  • Domain distributions: SimpleQA search results are dominated by general reference and social-information platforms, contrasting with the domain-focused retrieval used for long-form tasks.The distribution is based on the top domains returned by Google search for 100 samples from each task.
  • Domain distributions: HealthBench searches emphasize authoritative biomedical and public-health sites, while DeepResearchBench mixes technical and policy sources for broader exploratory research.Examples include CDC, PubMed Central, NCBI, Mayo Clinic, ResearchGate, OECD, and GitHub.
  • ResearchQA examples: The ResearchQA trajectory demonstrates planning around authoritative definitions, academic framing, reporting frameworks, and source browsing for a narrow-versus-broad CSR question.The trajectory explicitly separates reporting scope from stakeholder audience.
  • ResearchQA examples: The example CSR response characterizes narrow reporting as limited in issues and audience, often emphasizing regulatory compliance rather than broad stakeholder engagement.The response contrasts formal information-rights holders with a wider stakeholder community.
Loading 2511.19399v3…