PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Training Reward Models for Cross-Source Evidence Grounding in Search Agents

Haochen Zhao, Minsheng Hao, Zheng Li, Yifan Li, Zikang Liu, Bowen Qin, Yuhao Cheng, Chuang Fan, Haosheng Zou, Jeff Chen, Sujian Li

arXiv:2610.17827v1cs.AI

TL;DR

Search-agent evaluation often emphasizes final answers or citation support, leaving source independence and conflict handling unassessed together. CSEG applies a six-layer grounding hierarchy to complete trajectories, and its trained reward model improves a search policy’s independently judged grounding.

  • Problem

    Outcome-focused evaluation and citation support do not jointly assess whether evidence sources are independent or whether conflicting evidence is handled appropriately.

    Source: §1 · pp. 1–2

  • Method

    CSEG evaluates complete search trajectories with a six-layer decision tree, and CSEG-Bench supplies verified examples for training reward models on the hierarchy.

    Source: §1 · p. 2

  • Results

    83.9% exact accuracy is reached by the 4B CSEG-RM, while Qwen3-4B SFT+GRPO reward raises independently judged mean label from 2.834 to 4.193.

    Source: §1 · p. 2, §5.3 · p. 9

  • Takeaways & Limitations

    The results show that hierarchical grounding judgments can be learned and used as a reward to improve a search agent’s independently judged evidence use.

    Source: §1 · p. 2, §7 · p. 9

    CSEG covers single-turn trajectories and grounding relative to the retrieved pool, not world-level truth, so it complements rather than replaces outcome verification.

    Source: H Limitations · p. 25

Abstract

from arXiv · show

Existing approaches to training and evaluating search agents focus heavily on final-answer quality, largely neglecting how generated answers map back to retrieved evidence. Recent methods track citations to enforce grounding, but citation support alone does not establish source independence or whether an answer appropriately handles conflicting evidence. We argue that credibility in open-web retrieval depends on principled cross-source evidence grounding that accounts for source independence and systematically attributes and adjudicates conflicting evidence, while hierarchical judgments of these grounding decisions provide a unified signal for training and evaluation. To this end, we propose an evaluation framework, Cross-Source Evidence Grounding (CSEG), that employs a fine-grained, six-layer decision tree over the complete search trajectory. We introduce CSEG-Bench and trainable reward models to provide targeted supervision following this framework. Our approach helps eliminating the severe data skew of natural trajectories by providing targeted coverage over complex, deep-layer decision paths, such as cross-source conflict resolution and nuanced attribution, that standard rollouts fail to capture. Specifically, (1) we define the hierarchy at label, node, and path levels; (2) we construct 1,160 verified trajectories over nine canonical paths by rewriting answers on real retrieved document pools, with independent frontier judges reproducing the entire gold path and human experts deciding inclusion; (3) we train CSEG-RM, which reaches 83.9% exact accuracy on our benchmark while lying on the performance-token efficiency Pareto frontier. Experiments that use this reward alone to supervise a frozen-retrieval answer policy raise the independently judged mean label from 2.83 to 4.19, and show that learning the hierarchy generalizes across layers at a fraction of frontier-judge cost.

1 Introduction

Search agents can produce correct answers while relying on dependent sources or mishandling conflicting evidence, gaps that outcome-only rewards and citation support do not jointly assess. CSEG evaluates grounding across complete trajectories, and its reward model improves independently judged grounding when used to supervise a search policy.

1 Introduction

  • Outcome-only rewards can assign the same score to trajectories with different evidence quality, while citation-support checks do not jointly assess source independence and conflict handling.Figure 1 illustrates four trajectories answering 1945 that all receive outcome reward 1 despite different evidence.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • In 256 production trajectories whose answers passed a closed-QA verifier, 33.2% cited multiple URLs tracing to the same evidence root.The paper motivates judgments that are trajectory-conditional, root-aware, and conflict-sensitive.Source: §1 · p. 2
  • CSEG evaluates source independence, conflict handling, and adjudication across complete search trajectories, addressing dimensions that existing evaluation frameworks do not jointly assess.The framework returns an ordered label and the decision path that produced it.Source: §1 · p. 2
  • CSEG-Bench contains 1,160 verified trajectories, and the trained 4B CSEG-RM reaches 83.9% exact accuracy while improving a search agent’s independently judged mean label from 2.83 to 4.19.The contribution includes a hierarchical measure, expert-adjudicated path validation, and reward models trained on separate data.Source: §1 · p. 2, Figure 2 · p. 3

2 CSEG Evaluation Framework

The CSEG framework treats grounding as a relation among an answer, its citations, and the retrieved pool, then evaluates it through a six-layer hierarchy and label-, node-, and path-level metrics.

2 CSEG Evaluation Framework

  • CSEG defines cross-source grounding and reports evaluation metrics at label, node, and path levels.Its ordered labels encode the depth of the first failed decision.Source: §2 · p. 3, Table 2 · p. 4

2.1 Concept Formulation

  • Grounding is a relation among an answer, its citations, and the retrieved pool, so identical answer text can be well grounded under one pool and unsupported under another.The judge consumes the question, retrieved pool, and answer; its grounding assessment complements rather than replaces outcome verification.Source: §2.1 · p. 3

2.2 The six-layer hierarchy

  • The six-layer hierarchy checks citation existence and relevance, multiple cited sources, independent evidence roots, conflict detection, evidence combination, and conflict adjudication.Evaluation stops at the first failed layer; both all-pass branches receive label 7, although the consensus branch checks five layers and the conflict branch checks six.Source: §2.2 · pp. 3–4, Table 2 · p. 4

2.3 Evaluation metrics at label, node, and path levels

  • CSEG reports label-level accuracy, macro recall, MAE, and QWK, node-level classifier metrics, and path-level Levenshtein distance to the gold path.Node metrics use examples whose gold path reaches the node, and count premature stopping against recall; mean completion tokens proxy judging cost.Source: §2.3 · p. 4

3 CSEG-Bench

CSEG-Bench is built from real outcome-passing trajectories, with evidence scenarios and rewritten answers used to cover canonical paths before full-path validation selects the released examples.

3 CSEG-Bench

  • CSEG-Bench construction combines outcome-passing real trajectories, evidence scenario construction, answer rewriting, and full-path validation; unmodified trajectories separately estimate natural failure prevalence.The same construction procedure on disjoint base questions supplies reward-model training data.Source: §3 · p. 4

3.1 Outcome-passing real trajectories

  • The base records come from logged Tencent Yuanbao search-assistant rollouts and already pass both the service rubric and a closed-QA verifier.Deduplication reduces 12,332 records to 1,045 unique questions, partitioned into disjoint benchmark, reward-model-training, and answer-policy bases.Source: §3.1 · pp. 4–5

3.2 Evidence scenario construction and answer rewriting

  • Because natural trajectories provide sparse deep-layer supervision, construction groups supporting documents by evidence root, injects missing evidence properties when needed, freezes the pool, and rewrites the final answer to target a canonical path.Controls include host-pool profile matching, random insertion with whole-pool renumbering, and a constant injection budget across construction types.Source: §3.2 · p. 5

3.3 Full-path validation

  • Full-path validation releases 1,160 trajectories after frontier judges reproduce complete paths and human experts decide inclusion.Of 1,856 constructed examples, 1,718 land on their target path under the initial judge; the released set includes 692 two-judge-agreement and 468 single-judge examples.Source: §3.3 · p. 5

4 CSEG-RM: Trainable Grounding Rewards

CSEG-RM turns the grounding hierarchy into layer-wise training examples and learns to score trajectories. The learned reward is then tested on answer-policy optimization with retrieval held fixed.

4 CSEG-RM: Trainable Grounding Rewards

  • The model is invoked layer by layer on the question, retrieved sources, and final answer; deterministic routing composes each judgment into a terminal label and path.Trajectories are flattened into one example per visited layer; an all-pass consensus path contributes five examples and an all-pass conflict path six.Source: §4.1 · p. 5

4.1 Training data

  • Training data comprises 4,607 trajectories flattened into 17,477 layer examples, with disjoint bases and a split by base question.The teacher’s complete outputs—including reasoning and layer-specific fields—are distilled, although only the verdict is graded.Source: §4.1 · p. 6

4.2 Training and evaluation

  • SFT, GRPO, and SFT+GRPO train Qwen3-4B and Qwen3-8B; GRPO balances examples across layer-verdict cells to supply rare layers with within-group variance.SFT retains 14,876 of 17,477 teacher traces; GRPO optimizes a verifiable reward for 400 steps, and SFT+GRPO starts from the SFT checkpoint.Source: §4.2 · p. 6
  • Because the three routes share data, node prompts, and router, their comparison isolates the training route.Evaluation uses label-, node-, and path-level metrics.Source: §4.2 · p. 6

4.3 Supervising a Search Agent’s Answer Policy

  • In a deliberately narrow setting, a CSEG reward model scores a search agent’s final answer while its question, tool call, and retrieved-document prefix stays frozen, isolating evidence-use effects.The policy generates only the final assistant answer; the objective includes neither factuality reward nor a reference answer.Source: §4.3 · p. 6
  • Before-and-after trajectories are rescored by an independent frontier judge, with a separate ceiling check for pools whose support collapses to one evidence root.For such pools, no truthful answer can exceed label 3; scores above that ceiling may reflect judge error or learned syndication-as-corroboration.Source: §4.3 · p. 6

5 Experiments

The experiments diagnose grounding failures in natural trajectories, examine factors associated with benchmark performance, and test learned reward models. They find that training improves benchmark judgments and that CSEG-RM supervision raises independently judged grounding on a frozen-retrieval policy.

5.1 RQ1: why do we need synthetic data?

  • In 256 production trajectories that passed the closed-QA verifier, 112 (43.8%) terminate by L3, while only 85 (33.2%) traverse the full hierarchy.Natural rollouts also yield only 72 L5-X examples, versus roughly 976 per layer in the constructed arm.Source: §5.1 · p. 7

5.2 RQ2: what governs CSEG ability on CSEG-Bench?

  • At fixed effort, larger models are usually stronger within a family, but parameter count does not order performance across families.Thinking effort also interacts with model family: some models improve, while others regress or saturate as effort increases.Source: §5.2 · p. 7

5.3 RQ3: can compact models learn CSEG well enough to serve as rewards?

  • Conditional macro F1 at L5-X rises from 0.410 to 0.982, while L5-C remains the weakest layer.L5-C is the layer closest to a conventional attribution judgment.Source: §5.3 · p. 7
  • On CSEG-Bench, SFT+GRPO reaches exact accuracy 0.839 for Qwen3-4B and 0.833 for Qwen3-8B at roughly 2.1k completion tokens, and all three training routes improve both sizes.The trained models use 1.7× and 2.3× fewer completion tokens than their respective high-effort initializations; only one of 22 zero-shot configurations exceeds the best trained model.Source: §5.3 · p. 7
  • Qwen3-4B SFT+GRPO remains on the accuracy–token Pareto frontier until GLM-5.3-Flash, which uses 80× more parameters and 6.4× more tokens for 2.8 additional accuracy points.Qwen3-8B GRPO also enters the frontier; the practical result is a reward model inexpensive enough to run once per rollout during training.Source: Figure 3 · p. 8, §5.3 · pp. 8–9
  • Constructed data improves exact accuracy over natural-rollout training by 12.6 points at 4B and 20.2 points at 8B under the matched training protocol.The seed questions, row counts, splits, and GRPO recipe are held fixed.Source: §5.3 · p. 8
  • 2.834 to 4.193: CSEG-RM supervision raises the search agent’s independently judged mean label, while labels 5–7 rise from 19.1% to 40.2%.The evaluation uses 512 frozen-retrieval questions and an independent frontier judge.Source: Table 5 · p. 9, §5.3 · p. 9

6 Related Work

Related work assesses answer correctness, citation support, conflict handling, or source authority, but does not jointly measure evidence-root independence and cross-source adjudication. CSEG evaluates these dimensions over the full search trajectory and supplies layer-level labels for reward-model supervision.

6 Related Work

  • Outcome-based rewards score answer correctness, while citation-aware methods check claim support; neither alone establishes source independence or appropriate conflict handling across a retrieved pool.GAIA and BrowseComp assess information seeking through verifiable answers but do not assess duplicated evidence or overlooked conflicts.Source: §6 · p. 9
  • CSEG jointly judges evidence-root independence, conflict identification and attribution, and adjudication by authority and recency over the question, retrieved pool, answer, and citation links.Its layer-level labels also supervise grounding reward models.Source: §6 · p. 9

7 Conclusion

The conclusion frames CSEG as an auditable hierarchy for measuring cross-source grounding and as a learned signal for improving search-agent evidence use.

7 Conclusion

  • CSEG separates outcome success from evidence use with a six-layer judgment at label, node, and path levels, and its 4B and 8B graders lie on the Pareto frontier.One grader raises a search agent’s independently judged mean label from 2.83 to 4.19.Source: §7 · p. 9

Ethics Statement

The related-work discussion distinguishes outcome scoring, evidence-support evaluation, and conflict handling from CSEG’s joint trajectory-level grounding judgments. The production audit shows substantial false corroboration and sparse deep-layer coverage, while the ethics statement describes consent and safeguards for released data.

Ethics Statement

  • The source trajectories come from Tencent Yuanbao users who consented to research use; release safeguards address identifiers, sensitive content, query exposure, retention, access, and page licenses.Injected documents remain inside the controlled benchmark environment and are not presented as genuine web documents.Source: Ethics Statement · p. 10

A Related Work in Full

B The Natural Distribution as Evidence

  • Among 256 outcome-passing production trajectories, 43.8% failed within the first three layers, L3 accounted for 33.2%, and only two trajectories failed at L6.The conflict branch appeared in just 21 trajectories; the audit motivates controlled coverage of deep paths but is not a precise estimate of rare deep-layer failures.Source: B The Natural Distribution as Evidence · p. 17, Table 6 · p. 17

C The Evaluand and the Metrics

This section defines evaluation over complete search trajectories and specifies label-, node-, and path-level measures. It also details controlled benchmark construction and design choices intended to limit shortcuts and preserve interpretable supervision.

C The Evaluand and the Metrics

  • The evaluation unit is a complete trajectory of question, retrieved pool, and answer; the judge returns a terminal label from 1 to 7 and its decision path.Figure 4 illustrates the inputs, layer decisions, rationales, and resulting path and metrics.Source: C The Evaluand and the Metrics · p. 18, Figure 4 · p. 18
  • Label metrics include exact accuracy, macro recall, MAE, and primary QWK; conditional node scores penalize premature stopping, while path metrics measure edit distance and exact-path match.Node scores apply only when the gold path reaches the node; path edit distance distinguishes wrong verdicts from early or late stopping.Source: C The Evaluand and the Metrics · p. 18

D Construction and Validation Detail

  • Construction groups documents by evidence root, injects missing pool properties when needed, freezes the resulting pool, and rewrites answers to reach target paths.The evaluation arm has 1,856 examples and full validation; the 4,607-example training arm uses a disjoint base pool and a stricter, limited path check.Source: D Construction and Validation Detail · p. 20

E Why Surface Form Is a Label Shortcut

  • 100.0% precision and 90.5% recall identified injected documents using layout rules before host-pool alignment; afterward, the detector made no positive predictions.Alignment preserves authority and publication-time signals while matching other surface features to each document pool.Source: E Why Surface Form Is a Label Shortcut · p. 21
  • At a 15,872-token budget, random insertion and renumbering reduced records losing every injected document from 33.8% to 4.9%; at 20,480 tokens, losses fell from 8.8% to 1.4%.Truncation occurs at complete-document boundaries.Source: E Why Surface Form Is a Label Shortcut · p. 21
  • 72.9% versus a 54.3% majority baseline, an oracle identifying planted documents predicted the coarse branch when injection counts varied by type.Holding the planted-document budget constant removes that branch information from the count.Source: E Why Surface Form Is a Label Shortcut · p. 21

F Design Decisions Behind the Hierarchy

  • L1 checks citation existence and relevance, L3 counts distinct evidence roots, and L6 judges evidence substance rather than emphatic tone; full teacher outputs are retained after strict parsing.These choices separate topical but insufficient support from later semantic checks, and keep layer-specific grounds auditable.Source: F Design Decisions Behind the Hierarchy · p. 22

G.1 Resources and split discipline

  • The answer-policy evaluation reuses the 512 seed questions used to build reward-model training data, making the reward model more in-domain on evaluation questions than on the agent’s training questions.The authors state this understates rather than flatters generalization.Source: §G.1 · p. 23

G.2 Reading the three evaluation levels against each other

The three evaluation levels can rank judges differently: terminal accuracy alone misses differences in label severity and path fidelity. The section also reports weak semantic-layer performance and a secondary answer-policy comparison.

G.2 Reading the three evaluation levels against each other

  • 75.4% versus 73.5% exact accuracy favored Hy3 over the 4B GRPO reward model, but QWK (0.653 versus 0.705) and path edit distance (0.937 versus 0.847) favored the reward model at 3.7× lower cost.The paper reports the same accuracy-versus-hierarchy inversion for the distilled 8B model against GPT-OSS-120B.Source: §G.2 · p. 23
  • 0.466 conditional macro F1 at L5-C and 0.410 at L5-X were the weakest post-distillation layers; reinforcement raised L5-X to 0.937 and 0.982.The post-distillation L5-X positive-class F1 was 0.039, indicating the model passed nearly everything.Source: §G.2 · p. 23, Table 11 · p. 24

G.3 Conditional per-layer F1 and the data-source ablation

  • Table 12 compares training-data sources using 7,320 training and 911 development layer examples, with settings held byte-identical between arms.The table caption identifies the comparison setup but supplies no outcome values in these passages.Source: Table 12 · p. 24

G.4 Answer-policy detail

  • On the same 512 prefixes, a weaker GRPO-only grader produced about 10 percentage points fewer trajectories with labels 5–7 than the official SFT+GRPO-400 reward.Table 5 gives the official reward’s before-and-after label distribution; Figure 5 is the secondary comparison.Source: §G.4 · p. 23, Figure 5 · p. 24

H Limitations

The authors bound the claims to single-turn, frozen-retrieval answer rewrites and pool-relative grounding, and describe limits in data scope, selection, and downstream evaluation.

H Limitations

  • CSEG covers single-turn trajectories and rewrites only the final answer without re-retrieving; its grounding scores are relative to the retrieved pool, not world-level truth.Authority labels are operational, so CSEG should accompany rather than replace an outcome verifier.Source: H Limitations · p. 25
  • The data are mainly Chinese and come from one online service; the 43.8% gap should not be extrapolated without replication.With only two L6 failures in 256 trajectories, the production audit supports qualitative motivation for synthetic deep-path coverage, not precise rare-failure estimates.Source: H Limitations · p. 25
  • Human experts decided inclusion using frontier-judge evidence, conditioning both the released composition and those judges’ scores; downstream evaluation uses one initial policy and only re-scores grounding.The answer-policy experiment does not establish a task-performance gain.Source: H Limitations · p. 25

I Implementation Detail

The implementation trains CSEG reward models and a frozen-retrieval answer policy, then evaluates them with the same six-layer grounding pipeline. The node prompts route trajectories by their first failed layer, with distinct branches for consensus and conflict handling.

I Implementation Detail

  • 14,876 teacher traces survive filtering from 17,477 layer prompts; GRPO then uses 7,320 re-scored examples and is capped at 400 steps.SFT uses a byte-identical training and inference prefix; GRPO checkpoints and validates every 25 steps.Source: I Implementation Detail · p. 26
  • The answer policy trains on 2,624 tool-return prefixes and does not reopen retrieval; strong evaluation uses a frontier judge with web access disabled.Evaluation samples once per checkpoint on fixed prefixes and uses the benchmark’s six-layer pipeline.Source: I Implementation Detail · p. 27
  • 0.8388 exact accuracy is reached by the reported Qwen3-4B SFT+GRPO-400 CSEG-RM checkpoint.The passage distinguishes this benchmark result from the independent-judge comparison in Table 5.Source: I Implementation Detail · p. 27
  • 12/256 Qwen3-4B trajectories cite; thinking does not fix citation emission, so the data-source ablation uses Qwen3-8B with eight samples per seed.Since citation presence is the first gate, roughly 95% of the 4B trajectories can receive only label 1.Source: I Implementation Detail · pp. 27–28

J CSEG-Bench Node Prompts

  • The first failed layer sets labels 1–6, and only a pass through conflict attribution proceeds to adjudication; the L5-X pass can then end at label 7.L4 routes consistent evidence to L5-C, identified conflict to L5-X, and missed conflict to a stop at label 4.Source: J CSEG-Bench Node Prompts · p. 29, Figure 11 · p. 33, Figure 13 · p. 35
  • L1–L3 test citation presence, multiple sources, and source independence in sequence; L5-C consensus pass ends at label 7, while failure stops at label 5.Each failed early gate stops evaluation with its corresponding label.Source: Figure 8 · p. 30, Figure 9 · p. 31, Figure 10 · p. 32

Related papers

Loading 2610.17827v1…