Source-linked AI summary
Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards
Jiajie Zhang, Xin Lv, Ling Feng, Lei Hou, Juanzi Li
TL;DR
Binary outcome rewards do not adequately assess the comprehensiveness and factuality of deep-search reasoning, leaving agents vulnerable to shortcuts and hallucinations. The paper introduces CaRR, which decomposes complex questions into citation-supported single-hop rubrics, and C-GRPO, which combines rubric and outcome rewards. C-GRPO consistently outperforms outcome-based RL baselines across deep-search benchmarks and generalizes well to open-ended research tasks.
Problem
Binary outcome rewards fail to capture reasoning comprehensiveness and factuality, permitting shortcut exploitation and hallucinations in deep search agents.
Method
CaRR decomposes complex questions into verifiable single-hop rubrics, while C-GRPO combines context-aware rubric rewards with traditional outcome rewards.
Results
C-GRPO consistently outperforms GRPO across four deep search benchmarks, with average gains over GRPO of 5.1/8.0 for 4B models and 2.6/6.0 for 30B models at 64k/128k context.
Takeaways & Limitations
C-GRPO discourages shortcut exploitation, promotes comprehensive citation-supported reasoning, and generalizes well to open-ended deep research tasks.
Takeaways & Limitations
CaRR’s rubric generation relies on synthetic multi-hop question structure and may not directly adapt to open-ended questions whose requirements are implicit.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents. However, existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process, and often lead to undesirable behaviors such as shortcut exploitation and hallucinations. To address these limitations, we propose \textbf{Citation-aware Rubric Rewards (CaRR)}, a fine-grained reward framework for deep search agents that emphasizes reasoning comprehensiveness, factual grounding, and evidence connectivity. CaRR decomposes complex questions into verifiable single-hop rubrics and requires agents to satisfy these rubrics by explicitly identifying hidden entities, supporting them with correct citations, and constructing complete evidence chains that link to the predicted answer. We further introduce \textbf{Citation-aware Group Relative Policy Optimization (C-GRPO)}, which combines CaRR and outcome rewards for training robust deep search agents. Experiments show that C-GRPO consistently outperforms standard outcome-based RL baselines across multiple deep search benchmarks. Our analysis also validates that C-GRPO effectively discourages shortcut exploitation, promotes comprehensive, evidence-grounded reasoning, and exhibits strong generalization to open-ended deep research tasks. Our code and data are available at https://github.com/THUDM/CaRR.
1 Introduction
Deep search agents use RL for long-horizon web information seeking, but binary outcome rewards overlook reasoning quality and permit shortcuts or hallucinations. The paper proposes citation-aware rubric rewards and C-GRPO to promote comprehensive, factually grounded evidence chains, with experiments showing consistent gains over outcome-only RL.
- Motivation: LLM-based deep search agents use external web browsing and RL to solve complex, knowledge-intensive problems over long information-seeking horizons.Existing RL work commonly trains on synthetic multi-hop questions with short-form answers and binary outcome rewards.
- Problem: Binary outcome rewards cannot capture reasoning comprehensiveness or factuality, allowing shortcut solutions and fortunate hallucinations.These flawed trajectories can produce less robust agents and suboptimal performance.
- CaRR: CaRR decomposes multi-hop questions into single-hop factual rubrics that reward revealing hidden entities and supporting claims with correct citations.The framework targets reasoning comprehensiveness, factual grounding, and evidence connectivity.
- C-GRPO: C-GRPO combines context-aware rubric rewards with outcome rewards, preserving the goal of correct answers while encouraging comprehensive, evidence-grounded reasoning.It assigns additional weighted rubric rewards to trajectories whose outcome reward is 1.
- Evaluation: Across four deep search benchmarks and 4B and 30B models, C-GRPO consistently outperforms pure outcome-reward GRPO and improves performance with extended context budgets.The analysis reports less shortcut exploitation, more citation-supported solutions, stronger factuality, and generalization to open-ended deep research tasks.
2 Methodology
The methodology builds citation-aware rubric rewards by decomposing synthetic questions into verifiable hidden-entity checkpoints, then combines them with outcome rewards in C-GRPO. Rewards assess identification, citation grounding, and evidence connectivity while preserving correct-answer optimization.
- 2.1 Preliminary: Deep search agents iteratively generate thoughts, browsing actions, observations, and a cited final response using search, open, and find tools.The trajectory follows the ReAct paradigm until the agent produces an explanation with citations and the final answer.
- 2.1 Preliminary: Synthetic training questions use multi-hop entity-centric structures with short-form answers, enabling automatic outcome verification.Information obfuscation increases search complexity while the short final answer supports correctness checking.
- 2.2.1 Rubric Initialization: CaRR decomposes each synthetic question into atomic factual rubrics involving hidden entities, which become fixed checkpoints for evaluating trajectory comprehensiveness and factuality.Rubrics are initialized before training and remain unchanged throughout reinforcement learning.
- 2.2.2 Reward Computation: The reward pipeline identifies hidden entities in the final response, instantiates fully identified rubrics, and evaluates whether their claims are supported by cited web content.Cited URLs are extracted from the final answer and matched to deduplicated snippets, opened pages, and keyword results from the trajectory.
- 2.2.2 Reward Computation: Supported rubrics are retained only when connected through a bipartite entity-rubric graph to the predicted answer, preventing unrelated factual matches from hacking the reward.Breadth-first search from the predicted answer determines reachable rubrics, and the final reward measures the proportion fully identified, citation-supported, and connected.
- 2.3 C-GRPO: C-GRPO adds a normalized, weighted rubric reward to outcome-positive trajectories, while assigning zero reward to format-error or overlength rollouts before multi-turn GRPO optimization.The mixing coefficient α balances outcome and rubric rewards, and token-level advantages are computed from the mixed group rewards.
3 Experiments
Experiments evaluate C-GRPO across model scales, benchmarks, context budgets, and analyses of training behavior, evidence coverage, and reward design. Results consistently favor C-GRPO over outcome-reward baselines, while ablations identify hidden-entity checks, connectivity checks, and moderate rubric weighting as important design choices.
- Experiment Setup: Experiments use 4B and 30B models, DeepDive training data, Serper-based search, and four challenging deep-search benchmarks.The benchmarks are BrowseComp, BrowseComp-ZH, xbench-DeepSearch, and text-only GAIA.
- Main Results: C-GRPO significantly outperforms GRPO and E-GRPO across all benchmarks at both 4B and 30B scales.Compared with GRPO, average improvements are 5.1/8.0 for 4B and 2.6/6.0 for 30B under 64k/128k context budgets.
- Main Results: C-GRPO improves SFT models at both 64k and 128k context lengths, whereas pure-outcome GRPO may compromise longer-context test-time scaling.Figure 3 compares accuracy improvements at 64k and scaling performance under context and tool-call budgets.
- Training Dynamics: C-GRPO training increases tool-call steps after an initial decline, while GRPO continues decreasing, indicating different search behaviors during optimization.The paper links C-GRPO’s increasing tool use to gathering evidence for rubric satisfaction and verification, while GRPO favors shortcut solutions.
- Comprehensiveness and Generalization: C-GRPO cites more webpages and satisfies more rubrics than SFT and GRPO, and surpasses other RL algorithms across DeepResearch Bench dimensions.The reported dimensions include comprehensiveness, insight, instruction following, and readability; the 30B C-GRPO model also outperforms several proprietary-data agents.
- Ablation Studies: Performance peaks at α=0.3, while removing hidden-entity identification or evidence-connectivity checks causes performance declines.Adding rubric rewards to all rollouts can incorrectly advantage incorrect trajectories, especially when correct rollouts are scarce or overlength rollouts are common.
4 Related Works
Prior work largely applies RL to synthetic multi-hop deep-search tasks or adapts RL algorithms to multi-turn agents, but typically relies on outcome-only rewards. Rubric-based alignment offers a related strategy for providing verifiable supervision in complex instruction-following and long-form generation.
- Deep-search RL work mainly addresses complex QA data synthesis, training infrastructure, or RL algorithms suited to multi-turn agentic settings.
- Most existing deep-search approaches rely solely on outcome rewards, with limited attention to their limitations.
- E-GRPO uses entity match rate as fine-grained rewards for incorrect rollouts, distinguishing near-miss samples from complete failures.
- Rubric-reward research equips training instances with verifiable rubric lists for complex instruction following and long-form generation where traditional reward models provide unreliable supervision.
5 Conclusion
The paper introduces CaRR for fine-grained deep-search rewards and C-GRPO for combining these rubric rewards with outcome rewards. Extensive experiments report improvements over GRPO on deep-search benchmarks and open-ended research tasks.
- CaRR evaluates reasoning comprehensiveness, factual grounding, and evidence connectivity with fine-grained rewards for deep-search agents.
- C-GRPO combines CaRR with outcome rewards to train robust deep-search agents.
- C-GRPO achieves significant improvement over GRPO on deep-search benchmarks and open-ended research tasks.
6 Limitations
The rubric-generation procedure relies on synthetic multi-hop question structure, creating a direct-adaptation limitation for open-ended questions whose requirements are not explicitly stated. The paper nevertheless reports benefits on open-ended deep research tasks.
- CaRR’s rubric generation relies on the compositional structure of synthetic multi-hop questions.
- This structure may not directly adapt to open-ended QA training when question requirements are not explicitly stated.
- Synthetic short-form question answering is presented as an effective proxy for open-ended deep research because both require long-horizon information seeking.
- The paper reports that context-aware rubric rewards improve open-ended deep-research performance through better reasoning comprehensiveness and factual grounding.
7 Ethical Considerations
The paper states that its models and datasets are publicly published with permissible licenses and points readers to figures describing tool descriptions and trajectory format.
- All models and datasets used in the work are publicly published with permissible licenses.
- Figure 5 presents the tool descriptions, while Figure 6 presents the trajectory format.
B Details of Referred Deep Search Agents
The paper compares its deep-search agents with a broad set of systems reported in prior work and official reports. It also includes DeepResearch Bench systems in its comparison scope.
- The comparison includes OpenAI o3, DeepSeek-v3.1, Tongyi-DeepResearch, GLM-4.5, GLM-4.6, Aseacher, WebSailor, WebExplorer, and DeepDive.
- For DeepResearch Bench, the paper reports scores for OpenAI-DeepDeesearch, Kimi-Researcher, and Tongyi-DeepRe...The supplied passage truncates the final system name.
C Human Verification for LLM Judge
A manual review evaluates whether the judge LLM reliably identifies hidden entities and applies citation-based rubric judgments in CaRR. Against human assessments, the judge achieves high accuracy on both tasks.
- 97.7% accuracy was achieved for hidden-entity identification across 128 hidden entities.The review covered 10 DeepDive-30B-SFT trajectories.
- 95.1% accuracy was achieved for citation-based rubric evaluation across 164 rubrics.Human assessments served as the gold standard.
- The manual review indicates strong reliability for the CaRR judge LLM.
D Case Studies
The case studies contrast GRPO with C-GRPO on matched queries from training and evaluation sets. C-GRPO continues gathering evidence to verify constraints and supports response statements with citations.
- Case comparisons: GRPO tends to infer answers from the last several hops in a trajectory, illustrating shortcut behavior.
- Case comparisons: C-GRPO continues gathering evidence until every query constraint can be confirmed.
- Case comparisons: C-GRPO ensures that each statement in its response is supported by a corresponding citation.
Trajectory Format
The paper depicts deep-search trajectories as alternating reasoning and tool-use steps, alongside case examples and prompts for outcome and citation-aware rubric evaluation. The examples distinguish shortcut failures from rigorous verification.
- Trajectory format: The trajectory format includes assistant reasoning followed by tool calls with named tools and arguments.
- DeepDive cases: The case studies include a shortcut solution by DeepDive-30B-GRPO and a fully solved case by DeepDive-30B-C-GRPO on DeepDive.
- BrowseComp cases: On BrowseComp, the examples include a failed DeepDive-30B-GRPO case and a fully solved DeepDive-30B-C-GRPO case.
- BrowseComp examples: The displayed examples include a question about identifying a multilingual spy through linked biographical clues.
- CaRR prompts: The paper separately presents prompts for outcome rewards, rubric initialization, entity identification, and citation-based rubric judgment.
- CaRR prompts: The prompt framework asks for an answer entity, decomposed constraints, intermediate entities, and citation-based rubric judgments.