PaperlayerView exact revision on arXiv

Source-linked AI summary

Repurposing Synthetic Data for Fine-grained Search Agent Supervision

Yida Zhao, Kuan Li, Xixi Wu, Liwen Zhang, Dingchu Zhang, Baixuan Li, Maojia Song, Zhuo Chen, Chenxi Wang, Xinyu Wang, Kewei Tu, Pengjun Xie, Jingren Zhou, Yong Jiang

arXiv:2510.24694v2cs.CLcs.AI

TL;DR

Outcome-based GRPO discards entity information and treats near-misses like complete failures, despite their partially correct reasoning. E-GRPO uses entity-match-based rewards, and across QA and deep-research benchmarks it consistently outperforms GRPO while using fewer tool calls.

  • Problem

    Outcome-based GRPO discards intermediate entity information and treats incorrect near-misses like complete failures, while process-level alternatives can be costly or computationally intractable for web search.

    Source: §1 · pp. 1–2

  • Method

    E-GRPO uses a dense entity-aware reward, assigning incorrect samples partial credit proportional to their entity match rate.

    Source: §1 · p. 2

  • Results

    Across QA and deep-research benchmarks, E-GRPO consistently and significantly outperforms GRPO and learns more efficient policies with fewer tool calls.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Entity match rate is a strong proxy for factual correctness and can serve as a fine-grained reward signal.

    Source: §1 · p. 2

Abstract

from arXiv · show

LLM-based search agents are increasingly trained on entity-centric synthetic data to solve complex, knowledge-intensive tasks. However, prevailing training methods like Group Relative Policy Optimization (GRPO) discard this rich entity information, relying instead on sparse, outcome-based rewards. This critical limitation renders them unable to distinguish informative "near-miss" samples-those with substantially correct reasoning but a flawed final answer-from complete failures, thus discarding valuable learning signals. We address this by leveraging the very entities discarded during training. Our empirical analysis reveals a strong positive correlation between the number of ground-truth entities identified during an agent's reasoning process and final answer accuracy. Building on this insight, we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect samples proportional to their entity match rate, enabling the model to effectively learn from these "near-misses". Experiments on diverse question-answering (QA) and deep research benchmarks show that E-GRPO consistently and significantly outperforms the GRPO baseline. Furthermore, our analysis reveals that E-GRPO not only achieves superior accuracy but also induces more efficient reasoning policies that require fewer tool calls, demonstrating a more effective and sample-efficient approach to aligning search agents.

1 INTRODUCTION

GRPO discards entity information from synthetic training data and treats incorrect responses alike, overlooking informative near-misses. E-GRPO uses entity matches as a denser reward signal, and its evaluations report improved accuracy and fewer tool calls.

1 INTRODUCTION

  • Synthetic questions are created by transforming seed questions through operations such as fact injection or deliberate obfuscation, leaving key entities as the answer's factual backbone.The paper identifies these entities as information that outcome-based training methods discard.Source: §1 · p. 1
  • GRPO uses final-answer outcomes and treats negative samples uniformly, so it cannot distinguish a near-miss from a complete failure.This discards useful learning signals in partially correct reasoning.Source: §1 · pp. 1–2
  • Process-level reward models and complex sampling are poorly suited to web search because annotation is costly and long trajectories make sampling computationally intractable.Search trajectories may involve dozens of tool calls and reasoning steps.Source: §1 · p. 2
  • Entity match rate strongly correlates with agent performance and serves as a proxy for factual correctness.The paper proposes repurposing ground-truth entities embedded in synthetic data as a fine-grained reward signal.Source: §1 · pp. 2–3
  • E-GRPO gives incorrect samples partial reward proportional to their entity match rate, at negligible computational cost.This differentiates near-misses with many correct entities from complete failures.Source: §1 · p. 2
  • Across 11 benchmarks, E-GRPO significantly and consistently surpasses GRPO and requires fewer tool calls.The evaluations span diverse models and environments.Source: §1 · p. 3

2 PRELIMINARY

This section introduces search-agent rollouts and reviews entity-centric methods for synthesizing complex question-answer pairs.

2.1 SEARCH AGENTS

  • ReAct rollouts alternate thought, action, and observation until a final answer; search retrieves top-10 results, while Visit retrieves pages and extracts information relevant to browsing goals.The action space includes search queries, webpage browsing, and final-answer generation.Source: §2.1 · p. 3

2.2 ENTITY-CENTRIC DATA SYNTHESIS

  • Entity-centric synthesis raises question difficulty through ASearcher's entity injection and fuzzing, or SailorFog-QA's random-walk knowledge graphs, sampled subgraphs, and attribute obfuscation.Both methods build complex questions around entities.Source: §2.2 · pp. 3–4

3 METHODOLOGY

Section 3 examines entity matching as a signal of search-agent reasoning quality, then incorporates that signal into E-GRPO’s reward formulation. The method gives incorrect rollouts partial rewards based on their normalized entity match rate.

3.1 ANALYZING ENTITY MATCHING IN AGENTIC REASONING

  • The entity match rate measures the fraction of ground-truth entities mentioned in a rollout’s thoughts, with a full-phrase exact match counting as a mention.The normalized rate scales each rollout’s raw rate against the maximum in its question group, placing rollouts on a common 0-to-1 scale.Source: §3.1 · p. 4
  • 1939 vs. 487 questions favored higher entity match rates in correct rollouts; correct samples peaked at 1.0, while incorrect samples included near-misses with many matched entities.The analysis identifies entity match rate as a granular signal that distinguishes near-misses from complete failures.Source: §3.1 · pp. 4–5

3.2 ENTITY-AWARE GROUP RELATIVE POLICY OPTIMIZATION

  • E-GRPO assigns correct rollouts reward 1, incorrect rollouts reward α times their normalized entity match rate, and error rollouts reward 0.The parameter α balances answer accuracy and entity matching.Source: §3.2 · p. 5
  • This reward distinguishes incorrect rollouts by entity match rate and supplies a training signal even when every rollout in a group is wrong.A near-miss identifying a correct entity receives a partial reward, unlike a complete failure.Source: §3.2 · p. 5
  • The refined reward produces group-relative advantages used to optimize the GRPO objective.The training setup removes KL regularization, increases the upper clipping bound, assigns zero reward to format errors, and excludes overlength rollouts from the loss while retaining them for advantage normalization.Source: §3.2 · p. 6

4 EXPERIMENTS

Across QA and deep-research evaluations, E-GRPO outperforms GRPO; the analyses report higher training accuracy, fewer tool calls, and improved performance with a moderate entity-matching weight.

4.1 EXPERIMENT SETUP

  • Experiments span 11 benchmarks, local and web environments, and 7B and 30B models, with GRPO as the primary algorithmic baseline.The evaluation reports Pass@1 and Pass@3; training uses 1k-sample RL datasets and retains generated ground-truth entities.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • Local-7B-E-GRPO achieves a 64.2 average score on Local QA, 2.8 points above GRPO and 4.0 above the initial SFT model.Its superiority is consistent across most individual benchmarks.Source: Table 1 · p. 7, §4.2 · p. 8
  • Across four deep-research benchmarks, E-GRPO significantly outperforms GRPO at 7B and 30B; Web-30B-E-GRPO scores 12.9 on BrowseComp and 26.4 on BrowseComp-ZH.It also achieves the best performance among open-source agents on both benchmarks and surpasses Claude-4-Sonnet on BrowseComp.Source: Table 2 · p. 8, §4.2 · p. 8
  • On QA benchmarks evaluated in the Web environment, Local-7B-E-GRPO achieves the highest peer average score, 67.8.It outperforms its GRPO counterpart and larger open-source baselines in this unfamiliar evaluation environment.Source: §4.2 · p. 8
  • On GAIA Pass@3, E-GRPO gains 6.8 points to reach 51.5, while GRPO offers minimal gains over the SFT baseline, including a score of 44.7.The paper attributes the Pass@3 advantage to exploring promising incomplete paths and producing a more diverse set of solutions.Source: §4.2 · p. 8

4.3 ANALYSIS

  • For Web-environment 30B training, E-GRPO shows higher training accuracy and consistently uses fewer tool calls per rollout than GRPO.Figure 3 also plots entity matching against training accuracy.Source: Figure 3 · p. 9, §4.3 · p. 9
  • For both GRPO and E-GRPO, entity match rate and accuracy rise together; E-GRPO achieves higher entity matching and final-answer accuracy in the plotted comparison.The authors describe entity match rate as an effective proxy for final-answer accuracy.Source: §4.3 · p. 9
  • Across four benchmarks, performance peaks at α = 0.3; increasing α to 0.5 reduces performance on most benchmarks.α balances the outcome-based reward and entity-matching bonus; α = 0.0 reduces E-GRPO to GRPO.Source: §4.3 · p. 9

5 RELATED WORK

Prior search-agent training uses synthetic question-answer pairs while discarding the entities generated with them, and GRPO-like methods rely on sparse outcome-based rewards. This work frames those entities as a computationally efficient source of fine-grained supervision.

Search Agents

  • Search agents build on ReAct-style reasoning and actions to retrieve and synthesize information from noisy web sources.The cited work positions this paper as improving training methodology for robust policies in complex web environments.Source: §5 · p. 10

Synthetic Data Generation for Search Agents

  • Synthetic-data methods generate ground-truth entities as the factual backbone of answers, but prior post-training work used only question-answer pairs; this work repurposes the entities as reward supervision.The authors describe this as bridging data generation and RL alignment.Source: §5 · p. 10

Reinforcement Learning for Search Agents

  • GRPO-like search-agent methods rely on sparse outcome-based rewards, while PRMs and tree-based sampling are described as costly or intractable for open-ended web search.E-GRPO is presented as fine-grained and computationally efficient without additional annotation, model training, or complex sampling.Source: §5 · p. 10

6 CONCLUSION

E-GRPO uses discarded ground-truth entities as a factuality signal, and the paper reports stronger accuracy and fewer tool calls than GRPO across QA and deep-research benchmarks. The appendix explains how entity sets are constructed and matched, and gives the rollout format and training settings.

6 CONCLUSION

  • Across QA and deep-research benchmarks, E-GRPO consistently and significantly outperforms GRPO while achieving higher accuracy with fewer tool calls.The conclusion characterizes this as a more effective and sample-efficient approach to aligning search agents.Source: §6 · p. 10

A FORMAT

  • A complete rollout alternates thoughts, tool calls, and tool responses before ending with an answer; responses outside the prescribed format count as format errors.The provided template shows these elements inside the specified tags.Source: A FORMAT · p. 15

B.1 ENTITY CONSTRUCTION

  • For ASearcher, ground-truth entities are the selected and modified entities; for SailorFog-QA, they are the nodes in the sampled entity subgraph.The appendix states that the question-generation processes center questions on those entities.Source: §B.1 · p. 15

B.2 ENTITY MATCHING

  • Exact string matching suits the definite, short entity strings and is computationally trivial, while avoiding LLM-judge reward hacking observed in preliminary experiments.The authors report that an LLM judge could award partial credit to verbose, superficially relevant thoughts that omitted the correct entities.Source: §B.2 · pp. 15–16
  • Thought-only matching separates most incorrect trajectories into low match rates, whereas trajectory-wide matching yields more incorrect rollouts with high match rates.The authors attribute these false positives to entities appearing in tool observations without being extracted into the agent’s reasoning.Source: §B.2 · p. 16, Figure 5 · p. 16

C HYPERPARAMETERS

  • RL uses a group size of 8, learning rate 2e-6, and about 5 training epochs; E-GRPO’s default entity-matching weight α is 0.3.The appendix also specifies context length 32k, tool-call budget 40, batch sizes 64, and inference temperature 0.6 with top p 0.95.Source: C HYPERPARAMETERS · p. 16

D CASE STUDY

The case study contrasts E-GRPO’s more direct successful reasoning path with GRPO’s longer unsuccessful trajectory. In the polar-explorer example, E-GRPO reaches Karl Weyprecht, while GRPO’s displayed trajectory ends with Julius von Payer.

D CASE STUDY

  • E-GRPO identifies the key entities and reaches the correct answer with fewer tool calls, while GRPO uses more calls but misses a critical entity and answers incorrectly.The authors describe the GRPO path as longer and less fruitful.Source: D CASE STUDY · p. 17
  • The example’s ground-truth entities are Tegetthoff, International Polar Year, and the Royal Geographical Society’s Founder’s Medal; its stated answer is Karl Weyprecht.The case question combines expedition, vessel, scientific-initiative, and award clues.Source: D CASE STUDY · p. 19
  • GRPO’s displayed trajectory ultimately answers Julius von Payer, whereas the E-GRPO case trajectory answers Karl Weyprecht.The GRPO trajectory’s final round says it concluded Julius von Payer; the E-GRPO trajectory’s final answer names Karl Weyprecht.Source: D CASE STUDY · pp. 19, 22

E FAILURE ANALYSIS

The failure analysis identifies distracted querying, information overload from concurrent queries, and broken reasoning coherence as distinct agent error modes. Its case studies illustrate how these failures disrupt entity-focused reasoning, while the authors report that E-GRPO mitigates broken coherence.

E FAILURE ANALYSIS

  • Broken reasoning coherence occurs when the agent fails to act on its stated goal after a tool call, even after extracting a relevant fact.The example agent visits a page and extracts “a single frame,” then reasons about an unrelated subject.Source: E FAILURE ANALYSIS · p. 23
  • Distracted querying occurs when an agent shifts from its core objective to secondary information and may ultimately exceed the context length.In the example, later searches drop the temporal terms needed to find the requested year after an initial query included “early 1970s.”Source: E FAILURE ANALYSIS · p. 23
  • Concurrent similar queries can bury a useful result in dense returned text, leaving the agent unable to isolate the key signal and follow a flawed reasoning path.In the example, only the third query contains the critical term “Chair.”Source: E FAILURE ANALYSIS · pp. 23–24
  • The authors’ case study indicates E-GRPO mitigates broken coherence by incentivizing agents to carry critical entities into their thoughts.They say this implicitly enforces topical continuity and prevents reasoning from drifting into irrelevance.Source: E FAILURE ANALYSIS · pp. 23–24

F DECAYING ENTITY MATCHING WEIGHTS

The authors compare Pass@1 across entity-matching weights and find no clear, consistent advantage for decaying α over fixed 0.3. They retain fixed α as a practical setting while noting that longer decay schedules may be worth exploring.

F DECAYING ENTITY MATCHING WEIGHTS

G EXTENDED TRAINING DYNAMICS

With extended Web-environment training, E-GRPO retains a performance lead over GRPO while using fewer steps on average.

G EXTENDED TRAINING DYNAMICS

H ROBUST MATCHING ANALYSIS

Robust Matching addresses the rigidity of exact string matching and better captures the relationship between entity matches and factual correctness. In an additional experiment, it yields slight performance gains on three of four deep research benchmarks.

H ROBUST MATCHING ANALYSIS

  • Exact string matching is computationally efficient and mitigates reward hacking, but struggles with spelling variations, synonyms, and abbreviations.Source: H ROBUST MATCHING ANALYSIS · p. 24
  • Robust Matching prompts an advanced LLM for 5–10 plausible variations per ground-truth entity, then exact-matches those candidates against the agent’s thought process.An entity matches when any generated variation is found.Source: H ROBUST MATCHING ANALYSIS · p. 25
  • Robust Matching better captures near-misses than exact string matching, with more incorrect samples spread above a zero match rate.For correct samples, its density peak at match rate 1.0 is significantly higher; for incorrect samples, the peak at 0.0 is reduced.Source: H ROBUST MATCHING ANALYSIS · p. 25, Figure 7 · p. 26
  • Robust Matching achieves slight performance gains on three of the four deep research benchmarks.The additional RL experiment keeps other settings identical to Web-30B-E-GRPO; Table 4 reports Pass@1 performance by matching strategy.Source: H ROBUST MATCHING ANALYSIS · pp. 25–26, Table 4 · p. 25

I EXPLANATION OF THE ANALYSIS FIGURE

For most questions, correct responses match more entities than incorrect ones, establishing a strong correlation between entity match rate and final-answer correctness. The paper reports using Gemini-2.5-Pro for language polishing only.

I EXPLANATION OF THE ANALYSIS FIGURE

  • For most questions, correct responses match more entities than incorrect ones, establishing a strong correlation with final-answer correctness.The analysis compares mean matched entities among eight sampled responses, separated into correct and incorrect responses for each question.Source: I EXPLANATION OF THE ANALYSIS FIGURE · p. 26

J USE OF LARGE LANGUAGE MODELS (LLMS)

Related papers

Loading 2510.24694v2…