PaperlayerView exact revision on arXiv

Source-linked AI summary

Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents

Yingjian Zhu, Zhenyi Wang, Jiaxin Guo, Kun Ding, Ying Wang, Shen Huang, Xunjie Zhu, Pengjun Xie, Shiming Xiang

arXiv:2609.34296v1cs.CLcs.AI

TL;DR

Open-ended rubric-based RL rewards final answers but often does not distinguish intermediate decisions, while ground-truth-based process credit does not directly transfer to tasks without canonical answers. Dr.Credit evaluates each tool turn's additional rubric support against its evidence history and combines process advantages with GRPO outcome advantages. Across four benchmarks it outperforms evaluated open deep-research baselines, and its 8B agent is competitive with evaluated frontier proprietary models.

  • Problem

    Rubric-based RL often rewards final answers without distinguishing intermediate decisions, while ground-truth-dependent process credit is limited for open-ended tasks without canonical answers.

    Source: §1 · p. 2

  • Method

    Dr.Credit assesses each tool turn's additional support for each rubric against its accepted-support history, then combines process advantages with GRPO outcome advantages.

    Source: §1 · p. 2

  • Results

    Across four benchmarks, Dr.Credit outperforms evaluated open deep-research baselines, and its 8B agent achieves average performance competitive with evaluated frontier proprietary models.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Analyses indicate more efficient evidence acquisition and higher-quality reports under limited research-turn budgets.

    Source: §1 · p. 2

    Process-credit scoring may introduce substantial computational overhead during training.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Rubric-based tasks are increasingly addressed through reinforcement learning (RL), with rubric scores used as training rewards. However, these rewards typically supervise final answers without distinguishing the contributions of intermediate decisions. Many existing credit assignment methods rely on ground-truth answers to define process rewards, limiting their applicability to open-ended tasks without canonical solutions. To address this limitation, the proposed rubric-grounded credit uses task requirements as a shared reference for final answer evaluation and process supervision. The information returned by tools is assessed for the additional support it provides toward satisfying each rubric relative to that rubric's history of accepted support. By referencing these histories, credit distinguishes new support from evidence already present in the trajectory while recognizing partial support for each rubric. Dr.Credit uses rubric-grounded credit to supervise intermediate tool turns in an RL framework for deep research agents. The resulting process advantages are combined with GRPO outcome advantages to guide research decisions while retaining supervision of final-report quality. Evaluations on four in-domain and out-of-domain benchmarks show that Dr.Credit outperforms the evaluated open deep research baselines on every primary metric and submetric. Meanwhile, with an 8B-parameter backbone, the trained agent achieves average performance competitive with the evaluated frontier proprietary models. Further analyses suggest more efficient evidence acquisition and higher-quality reports under limited research-turn budgets, motivating the extension of rubric-grounded process supervision to a broader range of rubric-based tasks.

1 INTRODUCTION

Rubric-based RL evaluates final answers, but intermediate-turn credit remains difficult for open-ended tasks without canonical answers. Dr.Credit uses per-rubric evidence histories to assess new tool support, and experiments report gains over evaluated open baselines.

1 INTRODUCTION

  • Rubrics specify expected content and qualities for open-ended responses, and prior deep-research agents have used rubric-based rewards to optimize report quality.The introduction situates Dr.Credit amid rubric-based RL for tasks such as deep research, where exact answer matching is difficult.Source: §1 · p. 1
  • Existing turn-level credit methods often rely on ground-truth answers, limiting their direct transfer to open-ended tasks without canonical answers.Sharing an outcome advantage can reinforce ineffective decisions or penalize useful steps after later synthesis errors.Source: §1 · p. 2
  • Rubric-grounded credit scores the additional support new information provides for each rubric relative to its support history, without requiring a canonical answer.The resulting process advantages combine with GRPO outcome advantages, retaining final-answer evaluation while supervising intermediate decisions.Source: §1 · p. 2
  • Dr.Credit applies rubric-grounded credit to intermediate tool turns: Visit evidence is assessed against rubric histories, and credits can flow to earlier searches or snippets.Searches can receive supplementary credit when snippets support rubrics despite no later visit to the page.Source: §1 · p. 2
  • Dr.Credit outperforms evaluated open deep research baselines across in-domain and out-of-domain benchmarks; its 8B agent is competitive with evaluated frontier proprietary models.Further analyses indicate more efficient evidence acquisition and higher-quality reports under limited research-turn budgets.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define research rollouts as successive reasoning and tool interactions followed by a rubric-evaluated answer. They also describe rubric rewards and GRPO's shared rollout-level outcome advantage.

2.1 PROBLEM FORMULATION

  • A deep-research rollout consists of successive ReAct turns in which the policy reasons and calls tools, the environment returns observations, and a final answer is evaluated against task rubrics.The observations remain in context for subsequent reasoning and tool use.Source: §2.1 · p. 3

2.2 RUBRIC-BASED REINFORCEMENT LEARNING

  • Rubric-based RL assigns each task rubric a nonnegative weight and uses an LLM judge's satisfaction scores to compute a weighted-average reward for the final answer.Each rubric score lies between 0 and 1, and the rubric weights have a positive sum.Source: §2.2 · p. 3
  • GRPO normalizes rollout-level outcome rewards across samples for a question, giving all policy-generated tokens in a rollout the same advantage rather than distinguishing turn contributions.This leaves intermediate turns without explicit outcome-based credit for their contributions toward rubric satisfaction.Source: §2.2 · p. 3

3 METHODOLOGY

Dr.Credit assesses each tool turn's additional rubric support against per-rubric histories, then combines process and outcome advantages for policy optimization. Its deep-research implementation assigns evidence credit to visits and searches.

3 METHODOLOGY

  • Dr.Credit combines rubric-grounded process advantages with GRPO outcome advantages to supervise deep-research tool turns and final-answer quality.The framework assesses tool-turn contributions toward task rubrics and maintains support histories.Source: §3 · p. 4

3.1 RUBRIC-GROUNDED PROCESS CREDIT

  • Evidence support comprises facts, premises, relations, and qualifications that help fulfill a rubric without specifying a canonical answer or action sequence.Process assessment identifies support in tool responses, while final-answer scoring measures rubric satisfaction.Source: §3.1 · p. 4
  • Each rubric has an independent history of accepted support points, initialized empty and updated within each rollout to distinguish new from previously established support.The assessor compares new information with the history before adding accepted support; an empty increment leaves the history unchanged.Source: §3.1 · p. 4
  • An assessor evaluates each turn's additional support per rubric against its history, and the turn's rubric-grounded credit sums those support contributions.Accepted support increments update the reference available to later turns.Source: §3.1 · p. 4

3.2 POLICY OPTIMIZATION WITH PROCESS CREDIT

  • Process credits and outcome rewards are normalized separately within rollout groups, and the process term supplements outcome advantage on each non-final turn.Process-credit normalization includes zero-credit non-final turns, while outcome statistics retain GRPO's rollout-reward normalization.Source: §3.2 · pp. 4–5
  • The turn-level advantage is assigned to policy-generated tokens in that turn; environment observations condition the policy but are masked out of the loss.The policy objective applies to policy-generated tokens across sampled rollouts.Source: §3.2 · p. 5

3.3 RUBRIC-GROUNDED CREDIT FOR DEEP RESEARCH

  • Visit turns receive credit for additional page support, while Search turns receive credit for discoveries and rubric support present in result snippets.Visit credits are attributed to searches through normalized URL matches; an independent judge assesses snippets from results without later visits, and the two Search credits are added.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

4 EXPERIMENTS

Across four deep-research benchmarks, Dr.Credit outperforms the evaluated open deep-research baselines on every metric and submetric, and is competitive with proprietary models. Ablations and trajectory analyses examine its credit design, evidence acquisition, report quality, and training efficiency.

4.2 OVERALL PERFORMANCE

  • With an 8B backbone, Dr.Credit averages 61.7 and improves over matched GRPO by 3.5 points, while outperforming open deep-research baselines across all four benchmarks and their submetrics.It is also competitive with frontier models, surpassing DeepSeek-V4-Pro, Opus 4.8, and GLM-5.3 in average score.Source: Table 1 · p. 6, §4.2 · p. 7

4.1 EXPERIMENTAL SETUP

  • Evaluation uses DeepRubric for training and in-domain validation, plus ResearchQA, DeepResearchBench, and ResearchRubrics for out-of-domain testing, with a shared set of search and visit tools for relevant baselines.The RL run starts from a Qwen3-8B checkpoint after two SFT epochs and trains for 200 steps on 7,149 query–rubric pairs.Source: §4.1 · pp. 6–7

4.3 ABLATION STUDY

  • Adding navigation attribution and independent scoring of unvisited search snippets yields the highest four-benchmark average, 61.7, although DeepRubric falls by 0.2 points; discounted propagation lowers scores across all benchmarks.These comparisons support crediting both source discovery and evidence available directly in snippets.Source: Table 2 · p. 7, §4.3 · pp. 7–8
  • Removing process credit lowers the three-benchmark average by 3.5 points, while process-only supervision lowers it by 2.3, supporting complementary roles for process and outcome signals.Process credit guides turn-level evidence acquisition, while outcome supervision aligns decisions with final-report quality.Source: Table 3 · p. 8, §4.3 · p. 8
  • Removing per-rubric evidence history lowers scores on all three benchmarks by an average of 3.0 points, consistent with histories helping distinguish new support from repeated retrieval.The authors note that a page can support a rubric without extending the evidence already available.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 IN-DEPTH ANALYSIS

  • Under the evidence proxy, Dr.Credit reaches 2.94 within six turns versus GRPO’s 2.83 within eight, and has higher mean proxy scores per actual turn at every measured prefix.At eight turns, the per-turn scores are 0.53 versus 0.36 (+47.2%); GRPO slightly surpasses Dr.Credit’s cumulative score at twelve and fourteen turns.Source: Figure 4 · p. 8, §4.4 · p. 8
  • Dr.Credit exceeds GRPO at all seven tested inference-time budgets; with eight turns it scores 67.69, above GRPO’s best observed score of 66.41 at twelve turns.The eight-turn allowance is one-third smaller than the budget at which GRPO reached its best observed score.Source: Figure 4 · p. 8, §4.4 · pp. 8–9

4.5 COMPUTATIONAL EFFICIENCY

  • Dr.Credit surpasses GRPO’s final validation score by step 120 and ends 3.2 points ahead; after step 34, it records 11.18 fewer training hours (14.9%) over 200 steps.The early cumulative-time disadvantage reflects added process-credit scoring; more efficient research behavior and asynchronous scoring are cited as contributors to later savings.Source: Figure 5 · p. 9, §4.5 · p. 9

5 RELATED WORK

Prior work develops rubric-based rewards for deep research and a range of methods for assigning credit across intermediate steps. These approaches include rubric refinement, stage-level supervision, and target-answer-based progress measures.

5 RELATED WORK

  • Deep research rubric work improves rubric coverage, grounds rubrics in retrieved evidence or human preferences, and uses rubric trees or intermediate-stage rewards.RubricEM assigns a shared advantage to policy tokens within each stage.Source: §5 · p. 9
  • Classical credit assignment propagates or redistributes delayed rewards, while LLM-agent methods estimate step-level values or compare actions from repeated states.The cited approaches include TD(λ), generalized advantage estimation, RUDDER (Arjona-Medina et al., 2019), GiGPO (Feng et al., 2025), HCAPO (Tan et al., 2026), and HiPER.Source: §5 · p. 9
  • Target-answer progress methods reward changes in answer probability, teacher-estimated answer potential, or belief in the target solution.These include IGPO (Wang et al., 2026), TIPS (Xie et al., 2026b), and ∆Belief-RL (Auzina et al., 2026).Source: §5 · p. 10

6 CONCLUSION

Across four benchmarks, Dr.Credit outperforms the evaluated open deep research baselines, and its 8B agent achieves average performance competitive with evaluated frontier proprietary models. The paper uses rubric-grounded process credit for evidence acquisition and research decisions, with analyses suggesting benefits under limited research-turn budgets.

6 CONCLUSION

  • Across four benchmarks, Dr.Credit outperforms evaluated open deep research baselines; its Qwen3-8B agent achieves average performance competitive with evaluated frontier proprietary models.Further analyses suggest more efficient evidence acquisition and higher-quality reports under the same research-turn budgets.Source: §6 · p. 10
  • Rubric-grounded credit extends task rubrics from final-answer evaluation to process supervision by scoring additional support relative to each rubric’s evidence history.The approach is applied in Dr.Credit, which combines process and GRPO outcome advantages to supervise research decisions and final-report quality.Source: §6 · p. 10
  • The analyses suggest that trained agents acquire useful evidence more efficiently and produce higher-quality reports under the same research-turn budgets.These findings support rubric-based supervision of evidence acquisition.Source: §6 · p. 10
  • The findings motivate extending rubric-grounded credit to other open-ended tasks trained with rubric-based reinforcement learning.Source: §6 · p. 10

A.1 DATA AND TRAINING

The appendix describes the data and training setup for SFT and RL, then specifies the shared tool environment and interaction limits. The evaluated setup uses fixed context and turn budgets, with one attempt and report per question.

A.1 DATA AND TRAINING

  • The RL data comprise 7,149 examples, with 128 of 7,277 eligible DeepRubric question–rubric pairs held out for validation and the sets checked for pairwise disjointness.The source collection contained 9,062 pairs; 1,785 questions represented in an intermediate SFT collection were excluded.Source: §A.1 · p. 13
  • GLM-5.2 generated rubric-blind research trajectories for SFT; 1,505 retained trajectories passed report, citation, tool-argument, and rubric-score checks.Each retained trajectory had 4–18 research turns followed by a report.Source: §A.1 · p. 13
  • Qwen3-8B received full-parameter SFT, then both RL methods started from the epoch-two checkpoint and ran for 200 steps with shared initialization, data, outcome reward, and budget.Dr.Credit adds process supervision; the SFT schedule spans three epochs, as summarized in Table 4.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 TOOLS AND INTERACTION LIMITS

  • Search accepts up to two queries and returns up to three results per query; Visit accepts up to two URLs, and each research turn makes exactly one tool call.Visit extracts evidence and a goal-conditioned summary within 2,048 tokens per page; Search responses are limited to 4,096 tokens.Source: §A.2 · p. 13

A.3 INFERENCE AND EVALUATION

Inference and evaluation constrain each question to one research attempt and one evaluated report, with limits on context, turns, and generated tokens. The appendix also specifies decoding settings and how Table 1 identifies reported results and computes averages.

A.3 INFERENCE AND EVALUATION

  • The policy context limit is 32,768 tokens, and RL and evaluation allow at most 20 assistant turns with up to 8,192 generated tokens per turn.Tool observations use context, and the final report must fit the remainder; if limits prevent a valid report, the model completes it from the original question and collected trajectory.Source: §A.3 · pp. 13–14
  • Each question receives one research attempt and one evaluated report; the separate turn-budget experiment is defined in Appendix B.1.Source: §A.3 · p. 14
  • Evaluation uses temperature 0.6, top-p 0.95, and top-k 20, except for stated provider-specific settings; Table 1 marks prior reported results and averages available scores across four benchmarks.DeepRubric and ResearchRubrics results are marked with ∗ and †, respectively; the Average is unweighted and reported only when all scores are available.Source: §A.3 · p. 14

A.4 BENCHMARK METRICS

The appendix defines benchmark scoring and outcome rewards, then analyzes evidence acquisition, history-conditioned credit, and parameter sensitivity. Process advantages remain close to the reference across displayed parameter settings, while evidence history changes some Visit credit assessments.

A.4 BENCHMARK METRICS

  • ResearchQA rubric coverage is scored at five levels, while DeepResearchBench and ResearchRubrics use their respective rubric-based evaluation procedures.DeepRubric-Val measures weighted rubric satisfaction, including factual and logical breakdowns; benchmark scores are aggregated on a 100-point scale without auxiliary training rewards.Source: §A.4 · p. 14

A.5 OUTCOME REWARD

  • The shared outcome reward combines weighted rubric satisfaction with format, search, citation, and report-length rewards.Rubric judgments are normalized from 0–4 before aggregation; citation reward combines reference resolution and semantic support.Source: §A.5 · p. 14

B.1 EVIDENCE ACQUISITION AND TURN BUDGETS

  • The evidence-acquisition analysis compares accumulated evidence contributions and reports generated at seven research-turn budgets, excluding the final-answer turn.Prefix credit excludes navigation credit, and snippet eligibility is determined retrospectively by whether a result remains unvisited in the complete trajectory.Source: §B.1 · pp. 14–15

B.2 EFFECT OF EVIDENCE HISTORY

  • For transitions among Visit–rubric pairs with nonempty histories, the reported rate reaching level 0 is 20.9% for GRPO; Figure 8 includes 916 Dr.Credit and 2,342 GRPO pairs.Source: Figure 8 · p. 16, §B.2 · p. 16

B.3 SENSITIVITY TO CREDIT PARAMETERS

  • W1 ≤0.077, RMSE ≤0.192, and at least 96.6% of fused advantage signs are preserved across displayed credit-parameter settings.These results support the adopted Visit scores, cap, and snippet weight as reasonable defaults.Source: Figure 9 · p. 17, §B.3 · p. 17

B.4 TEMPORAL DISCOUNTING ABLATION

The ablation normalizes process credits over non-final turns and propagates them backward within each rollout. Training diagnostics also track rubric scores, research turns, and report lengths.

B.4 TEMPORAL DISCOUNTING ABLATION

  • Process credits are normalized across all non-final turns, including zero-credit turns, before advantages are accumulated backward within each rollout.The discounted variants use Visit and navigation credit, with or without independent snippet scoring; γ = 0 recovers immediate credit, while γ = 0.95 is also considered.Source: §B.4 · p. 17
  • The temporal-discounting ablation varies temporal propagation while keeping PPO clipping and the policy-token mask unchanged.The final report retains outcome-only supervision, without renormalization or additional advantage clipping.Source: §B.4 · p. 17

C.1 TRAINING DIAGNOSTICS

  • Figure 10 tracks rubric scores, research turns, and validation report lengths during training; reports average approximately 4,576 and 4,900 tokens over the final five checkpoints.Report lengths increased for both methods during training.Source: §C.1 · p. 17

C.2 TRAINING TIME AND SCORING WORKLOAD

Over the same 200-step window and accelerator configuration, Dr.Credit records lower training time than GRPO. Its interaction-turn count and process-evaluation tasks also decrease over training.

C.2 TRAINING TIME AND SCORING WORKLOAD

  • Scoring latency is included in recorded training time because completed trajectories are scored while others continue generating.The reported latency should not be added again to total training time.Source: §C.2 · pp. 17–18, Table 7 · p. 18
  • 63.96 hours for Dr.Credit versus 75.14 hours for GRPO represents a 14.9% reduction over 200 steps.The comparison fixes training steps rather than time to a matched quality threshold; the total includes work beyond the listed stages.Source: §C.2 · p. 17, Table 6 · p. 18
  • From steps 1–40 to 161–200, Dr.Credit’s interaction turns fall from 13.2 to 7.8 per trajectory, while process-evaluation tasks drop 38.0%.Tasks decrease from 10,043.93 to 6,231.20 per step.Source: §C.2 · p. 18

D PROMPT TEMPLATES

The appendix collects the prompts used for research behavior, data generation, outcome assessment, and process-credit assessment. They specify tool-use and evidence rules as well as how rubric support is judged.

D PROMPT TEMPLATES

  • The appendix templates cover data generation, tool interaction, and supervision, with runtime placeholders serialized as JSON.The shared research instructions use Search and Visit, cite successful visits, and put the report in one answer block.Source: D PROMPT TEMPLATES · p. 18, Figure 11 · p. 19

D.1 RESEARCH AND DATA GENERATION

  • The research-agent prompt requires one Search or Visit call per tool turn and citations grounded in successful Visit evidence.The separate SFT-generation prompt specifies reasoning, tool use, evidence-grounded citations, and report structure; its instructions are replaced by the agent prompt at export.Source: §D.1 · p. 19, Figure 12 · p. 20
  • The Visit extraction template returns original webpage evidence and a summary conditioned on page content and the research goal.The research-agent and SFT-generation prompts receive the question without rubrics.Source: §D.1 · p. 18, Figure 13 · p. 20

D.2 OUTCOME ASSESSMENT

  • Final-report rubric assessment judges an answer against one criterion, which the template uses to mean one rubric.The rubric judgment is normalized from a 0–4 score before aggregation into the weighted reward.Source: §D.2 · p. 20, Figure 14 · p. 21
  • Citation verification checks whether supplied sources support each material proposition in the exact claim, assigning full credit only when all are supported.For claims citing multiple pages, the same assessment considers both individual sources and their joint support.Source: §D.2 · pp. 20–21, Figure 15 · p. 22

D.3 PROCESS CREDIT ASSESSMENT

  • The Visit judge assesses one complete visit’s marginal contribution to one rubric relative to prior confirmed support in the rollout.Its input separates trusted rubric evidence and prior support points from current-visit pages, reasoning, and tool calls.Source: §D.3 · pp. 22–23, Figure 16 · p. 23
  • Search-snippet assessment gives sparse rubric matches only for explicit, grounded clues in literal snippet text, without using Visit histories.The instructions reject topical relevance and require an exact supporting quote for each match.Source: §D.3 · pp. 22, 24, Figure 18 · p. 24
Loading 2609.34296v1…