PaperlayerView exact revision on arXiv

Source-linked AI summary

SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents

Shengze Huang, Zihao Tang, Jinluan Yang, Xin Yu, Zhenghao Lin, Xiao Liu, Zengxuan Wen, Yeyun Gong, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Fei Wu, Kun Kuang

arXiv:2610.13157v1cs.CL

TL;DR

Search-agent RL can reward answers without distinguishing grounded external evidence from internal shortcuts, contributing to evidence fabrication. SOAR separates those sources into observable streams and aligns rewards with declared provenance; across seven benchmarks it improves performance and reduces fabrication.

  • Problem

    Opaque search-agent trajectories hide whether answers rely on retrieved evidence or internal knowledge, so answer-level rewards cannot distinguish grounded reasoning from shortcuts.

    Source: §1 · p. 1

  • Method

    SOAR separates external evidence and internal knowledge into observable streams and uses Source-Consistent Reward to align each reasoning step with its declared evidence provenance.

    Source: §1 · p. 1, §3.2.1 · p. 3

  • Results

    Across seven benchmarks, SOAR achieves average relative improvements of 7.3% (3B) and 5.4% (7B) and a 74.81% relative reduction in evidence fabrication.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SOAR fosters a grounded information-acquisition policy while improving performance across seven benchmarks.

    Source: §1 · p. 2, §5 · p. 8

    Primary training and evaluation use Wikipedia as a static, high-quality source, leaving performance under malicious misinformation or highly unstructured, conflicting web sources for future study.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) for search agents often suffers from evidence fabrication, a failure mode where agents bypass external information ($E_{ext}$) and fabricate an incorrect answer. We argue that this fabrication arises from an over-reliance on unreliable internal knowledge ($E_{int}$) as shortcuts due to the inherent opacity of reasoning trajectories. To address this, we propose SOAR (SOurce-Aware Reinforcement learning), a paradigm that transforms monolithic reasoning into source-explicit trajectories. SOAR (i) disentangles $E_{ext}$ and $E_{int}$ into parallel, observable streams via a Source-Aware Workflow, and (ii) incorporates Source-Consistent Reward to guide the policy evolution. By making information-source attribution explicit, SOAR enables precise credit assignment between different knowledge streams. Evaluations show that SOAR outperforms baselines on 3B and 7B backbones by 7.3% and 5.4%, while reducing evidence fabrication by 74.8%. These confirm that SOAR significantly improves performance on knowledge-intensive benchmarks while effectively mitigating evidence fabrication, fostering more faithful and grounded reasoning with enhanced robustness.

1 Introduction

SOAR frames evidence fabrication as a source-attribution and credit-assignment problem in opaque search-agent reasoning. It makes internal and external information explicit in separate streams and aligns rewards with declared evidence sources.

1 Introduction

  • Evidence fabrication arises when opaque trajectories hide whether reasoning draws on external evidence or internal shortcuts, leaving rewards unable to distinguish grounded from ungrounded answers.The introduction describes this as a credit-assignment flaw that can reinforce unfaithful reasoning.Source: §1 · p. 1
  • SOAR's Source-Aware Workflow separates external information and internal knowledge into observable streams; Figure 1 depicts targeted retrieval when internal information is deemed unreliable.This makes reasoning evidence traceable across trajectories.Source: §1 · p. 1, Figure 1 · p. 2
  • Source-Consistent Reward aligns the model's self-declared evidence source with its reasoning process, supporting precise credit assignment across evidence sources.The framework is designed to prevent reliance on internal shortcuts.Source: §1 · pp. 1–2
  • 7.3% (3B) and 5.4% (7B) average relative improvements across seven benchmarks, alongside a 74.81% relative reduction in evidence fabrication.The comparison is against competitive RL baselines.Source: §1 · p. 2

2 Preliminary

The preliminary section defines search agents and their information sources, then identifies evidence fabrication as reasoning that departs from the available context, especially when retrieval is insufficient.

2.1 Search Agents

  • Search agents interleave reasoning, search actions, and retrieved observations before answering, while answer-level rewards leave evidence sources hidden and create a credit-assignment flaw.The setup describes an LLM policy interacting with external search tools on knowledge-intensive tasks.Source: §2.1 · p. 2

2.2 Internal vs. External Information

  • Internal knowledge is stored in model parameters and accessible without tools, whereas external information is task-specific and retrieved through search tools.Source: §2.2 · p. 2

2.3 Challenge

  • Evidence fabrication is reasoning that departs from the existing information context, particularly when that context lacks evidence needed to resolve the task.The section connects this pattern to elevated fabrication rates in conventional RL-based search agents.Source: §2.3 · p. 3

3 Methodology

SOAR makes external evidence and internal knowledge explicit in reasoning trajectories, then uses source-aligned rewards and comparative credit assignment to guide policy learning. Its workflow addresses the ambiguity that the authors identify as enabling shortcut exploitation.

3.1 Motivation

  • Opaque trajectories can conflate rewards for shortcut-driven and retrieval-grounded paths while obscuring whether errors arise from knowledge gaps or evidence synthesis.The authors argue that this ambiguity encourages shortcut exploitation and impedes robust grounding.Source: §3.1 · p. 3

3.2.1 Overview

  • SOAR conditions policy learning on explicitly decoupled external-evidence and elicited-internal-knowledge streams, with Source-Consistent Reward aligning reasoning steps to declared provenance.The framework is built on GRPO and treats the reward as a structural constraint on optimization.Source: §3.2.1 · p. 3

3.2.2 Source-Aware Workflow

  • The Source-Aware Workflow organizes internal and external information as attributable sources, making reasoning paths distinguishable by which stream grounds them.The framework treats internal parameters as a query-contingent knowledge source organized with the same structural rigor as an external corpus.Source: §3.2.2 · pp. 3–4
  • For each atomic query, the workflow retrieves external evidence while eliciting internal knowledge in a context stripped of the existing reasoning trajectory.This isolation is intended to make internal recall reflect parametric state rather than the current reasoning context.Source: §3.2.2 · pp. 3–4
  • Contrasting trajectories with different source selections lets Group Advantage assign credit to information-selection behavior and favor pathways with higher advantages for a query type.The policy can therefore navigate the internal-versus-external information trade-off by context.Source: Figure 2 · p. 4, §3.2.2 · p. 4

3.2.3 Source-Consistance Reward

  • The total reward combines answer correctness, format compliance, and source-consistency matching, treating format and provenance as constraints on terminal accuracy optimization.The source match uses a verifier to assess whether an answer is consistent with its specified information stream.Source: §3.2.3 · pp. 4–5
  • Answer correctness provides the primary task-completion signal through a binary reward for an exact match between the parsed answer and ground truth.The reward follows Jin et al. (2025a).Source: §3.2.3 · pp. 4–5

4 EXPERIMENTS

SOAR is evaluated across QA benchmarks, model behaviors, component ablations, and corrupted-training conditions. It outperforms baselines, reduces evidence fabrication, and maintains search behavior and performance under noise.

4.1 Experimental Setting

  • Evaluation covers four single-hop and three multi-hop QA benchmarks, using Exact Match as the primary metric and combining NQ and HotpotQA training splits.The study uses Qwen-2.5-3B/7B-Base, top-3 retrieval from a 2018 Wikipedia corpus, and GRPO; fabrication judgments use GPT-5.6-Sol over four runs.Source: §4.1 · p. 5

4.2 Main Results

  • SOAR scores 0.414 average on 3B, surpassing Search-R1 by +4.2%, and scores 0.448 on 7B versus AutoRefine (0.425) and IKEA (0.402).Table 1 reports Exact Match accuracy across QA benchmarks and backbones.Source: §4.2 · p. 5, Table 1 · p. 6
  • On 3B, SOAR reduces Evidence Fabrication Rate to 4.05% (a 74.81% relative reduction).SOAR also maintains advantages on ASQA and WebWalkerQA; the authors attribute accuracy gains to effective selection and use of internal and external knowledge.Source: §4.2 · pp. 5–6

4.3 Model Behaviors

  • During training, search frequency rises and stabilizes at approximately 1.6 calls per query, and active search generalizes across evaluation benchmarks.The authors describe the Source-Aware Workflow and Source-Consistent Reward as exposing and penalizing internal shortcuts, encouraging proactive retrieval.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4 Ablation Study

  • The full SOAR configuration leads across multi-hop benchmarks; removing both components drops performance from 0.333 to 0.294.Removing the Source-Consistent Reward alone degrades performance, and removing the Source-Aware Workflow as well causes further degradation.Source: §4.4 · p. 7, Table 3 · p. 7
  • Removing the reward raises fabrication from 4.05% to 6.75% overall and from 1.34% to 2.75% for correct answers; SOAR retains comparable token efficiency.The reward aligns declared evidence sources with reasoning content; token efficiency is maintained through parallel internal elicitation and external retrieval across GPUs.Source: §4.4 · pp. 7–8, Table 4 · p. 8

4.5 Systematic Robustness

  • With 5% of training samples corrupted, SOAR’s performance decreases by 0.019, compared with average drops of 0.038 for Search-R1 and IKEA.The corruption replaces correct retrieved evidence with misleading information for long-tail queries answerable from internal knowledge.Source: §4.5 · p. 8, Table 5 · p. 8
  • Under corrupted training, SOAR’s fabrication rate changes by +0.33%, while Search-R1’s increases by +2.15%.The authors report that SOAR maintains its search-proactive policy by filtering noise and prioritizing the global grounding signal.Source: §4.5 · p. 8, Table 6 · p. 8

5 Conclusion

SOAR makes search-agent reasoning source-explicit to address evidence fabrication and credit-assignment flaws. Across seven benchmarks, it improves performance while promoting grounded information acquisition at controlled token cost relative to baselines.

5 Conclusion

  • SOAR makes reasoning evidence traceable by separating internal and external knowledge into observable streams and aligning source claims with the evidence used.The workflow and source-consistent reward support credit assignment across trajectories using different evidence sources.Source: §5 · p. 8
  • The paper identifies monolithic reasoning and its credit-assignment flaw as drivers of over-reliance on internal shortcuts over external evidence.Source: §5 · p. 8
  • Across seven benchmarks, SOAR improves performance while fostering grounded information acquisition at controlled token cost relative to baselines.Source: §5 · p. 8

6 Limitations

SOAR’s primary training and evaluation use Wikipedia, a static, high-quality knowledge source that does not capture the full volatility of open-web environments. Preliminary online-search validation is reported, while more severe retrieval settings remain for future study.

6 Limitations

  • Training and evaluation rely primarily on static, high-quality Wikipedia, which does not fully represent the volatility of open-web environments.The paper says this setting enables clean diagnosis of evidence fabrication.Source: §6 · p. 9
  • Preliminary ASQA and WebWalkerQA validation indicates that SOAR maintains its advantage under online-search settings.Source: §6 · p. 9
  • Performance under severe retrieval constraints, including malicious misinformation or highly unstructured, conflicting sources, remains an open research direction.Source: §6 · p. 9

7 Ethics and Potential Risks

The paper frames SOAR as a way to improve truthfulness by making knowledge-source attribution explicit in search-agent reasoning. It also identifies limits from reliance on retrieved information and discusses gaps in prior search-agent and faithful-learning approaches.

7 Ethics and Potential Risks

  • Retrieved evidence may contain inaccuracies, biases, or omissions that model outputs could inherit or amplify.The experiments use public academic benchmarks with no known personally identifiable information, and future work will explore stronger verification mechanisms.Source: §7 · p. 9

A.1 RL for Search Agents

  • Prior search-agent approaches emphasize static source choices or search frequency, leaving internal-knowledge-dominated trajectories able to earn high rewards while ignoring external evidence (Huang et al., 2025b; Lin et al., 2025; Liu et al., 2026).Source: §A.1 · p. 12
  • SOAR makes source attribution explicit during rollouts so RL can optimize distinct knowledge-dependent paths rather than search efficiency alone.This targets the credit-assignment flaw between internal and external sources.Source: §A.1 · p. 12

A.2 Faithful Learning in RL

B Implementation of Verification

The implementation uses exact-match verification for source consistency during RL training. The authors report that it captures most source-consistency signals with lower token cost and latency than an LLM verifier.

B Implementation of Verification

  • The implementation uses a rule-based exact-match verifier for computational efficiency during RL training.The verifier is used for Verify(y, ECy) and the Source-Consistency Match Reward.Source: B Implementation of Verification · p. 12
  • 0.4187 vs. 0.4160 average performance: LLM-based verification yields only marginal improvement over exact-match verification.The paper adopts exact-match verification throughout its experiments to avoid the substantially higher token cost and inference latency of the LLM-based verifier.Source: B Implementation of Verification · p. 12

C Experimental Details

  • The appendix describes additional implementations, dataset configurations, and environment setups for reproducibility, with AI-assisted revisions reviewed and verified by the authors.Source: C Experimental Details · p. 12

C.1 Dataset Configurations and Statistics

SOAR’s reward combines answer correctness, format compliance, and source consistency, with stated weights for the latter two components. On the Qwen2.5-3B-Instruct backbone, it leads on the reported multi-hop tasks and average score.

C.1 Dataset Configurations and Statistics

  • Dataset splits and training and evaluation sample counts are summarized in Table 8.The supplied passage does not provide the individual dataset counts.Source: §C.1 · pp. 12–13, Table 8 · p. 13

C.2 Hyperparameters and Infrastructure

  • Training uses GRPO to fine-tune the Qwen2.5 model family for three epochs, with five rollout outputs per prompt and up to five search turns.The setup uses a single node with 8× H200 GPUs, a global batch size of 256 prompts, and top-3 retrieval.Source: §C.2 · p. 13

D Reward Configuration

  • Following Search-R1-v0.3, the final reward sums answer-correctness, format-compliance, and Source-Consistency Match Reward components, applies normalization, and uses α = 0.2 and γ = 0.1 in all experiments.Source: D Reward Configuration · p. 13

E Performance on Instruct-tuned Backbone

  • SOAR achieves leading results on 2Wiki (0.411) and Musique (0.144), with a leading average score of 0.397 on Qwen2.5-3B-Instruct.The passage presents this evaluation as a check of generalizability to an instruct-tuned backbone.Source: E Performance on Instruct-tuned Backbone · p. 13

F Performance on Challenge benchmark

SOAR outperforms competitive baselines on ASQA and WebWalkerQA, two benchmarks involving multi-document synthesis and longer reasoning chains. The experiments use offline retrieval for ASQA and live search for WebWalkerQA.

F Performance on Challenge benchmark

  • ASQA uses E5base-v2 to retrieve three documents from a 2018 Wikipedia dump, while WebWalkerQA uses a live search API returning three results per query.The retrieval budget is top three in both settings.Source: F Performance on Challenge benchmark · p. 13
  • SOAR consistently outperforms competitive baselines on both challenging long-form QA benchmarks, with the advantage transferring to complex reasoning and online search settings.The passage characterizes the results as generalizing to real-world online search scenarios.Source: F Performance on Challenge benchmark · p. 13

G.1 Token Cost Analysis

SOAR improves over Search-R1 across four-seed evaluations, with gains much larger than run-to-run variation, and adds only a small per-turn token cost. Human verification also supports the reliability of its automatic evidence-fabrication evaluation; case studies illustrate continued retrieval when evidence is incomplete.

G.1 Token Cost Analysis

  • 431 output tokens per turn for SOAR versus 408 for Search-R1 is a slight increase attributed to explicit internal-knowledge elicitation.The authors report no additional peak VRAM usage and describe parallel elicitation and retrieval as overhead mitigations.Source: §G.1 · pp. 13–14, Table 11 · p. 14

G.3 Human Verification of Fabrication Evaluation

  • 98.5% agreement (197 of 200 decisions) between the automatic judge and human annotation supports the reliability of the reported fabrication rates.The randomly sampled decisions were manually checked using the same criteria as the judge prompt.Source: §G.3 · p. 14

G.2 Statistical Significance Analysis

  • +4.74 points on average score (0.4187 vs. 0.3713) separates SOAR from Search-R1, exceeding the observed run-to-run variance by more than an order of magnitude.Across four seeds, average standard deviation is ≤0.0021, and SOAR outperforms Search-R1 on every benchmark across all seeds.Source: §G.2 · p. 14, Table 12 · p. 15

H.1 Agent RL Training Prompt; H.2 Internal Knowledge Extraction Prompt

  • The training prompt guides interaction with the external environment, while the extraction prompt structures internal knowledge as k independent declarative facts for source-consistency evaluation..Source: §H.1 · p. 14, §H.2 · p. 15

I.1 Spurious Correctness from evidence fabrication

  • Search-R1 can answer correctly despite missing key retrieved evidence, filling the gap with internal knowledge and potentially receiving a misleading positive reward.The paper describes this as a hidden retrieval failure that may reinforce suboptimal retrieval patterns during RL optimization.Source: §I.1 · p. 15, Table 15 · p. 17

I.2 Source-Aware Reasoning Improves Deep Evidence Retrieval

  • When retrieved evidence is incomplete, SOAR initiates a targeted second search and combines externally confirmed dates to resolve a multi-hop comparison.In the example, Search-R1 hallucinates a release date, while SOAR retrieves dates of 1992 and 2002 and reaches the correct conclusion.Source: §I.2 · p. 16, Table 16 · p. 17
Loading 2610.13157v1…