PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward

Senkang Hu, Yong Dai, Yuzhi Zhao, Yihang Tao, Yu Guo, Zhengru Fang, Sam Tak Wu Kwong, Yuguang Fang

arXiv:2602.00845v3cs.AI

TL;DR

Agentic retrieval is difficult to optimize with dense, principled reward signals. InfoReasoner defines information gain as uncertainty reduction over belief states and estimates a synthetic semantic reward from model outputs. Across seven QA benchmarks, it reports state-of-the-art performance at 3B and 7B scales, including up to 5.4% average improvement over strong retrieval-augmented baselines.

  • Problem

    Optimizing retrieval in agentic reasoning lacks dense, principled rewards, while traditional information-gain metrics require unavailable ground-truth annotations or oracle belief states.

    Source: §1 · p. 1

  • Method

    InfoReasoner defines information gain as belief-state uncertainty reduction and estimates an output-aware semantic reward by clustering generated answers, without manual retrieval annotations.

    Source: §1 · p. 2

  • Results

    Across seven QA benchmarks, InfoReasoner achieves state-of-the-art performance at 3B and 7B scales, with up to 5.4% average improvement over strong retrieval-augmented baselines.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors present InfoReasoner as a theoretically grounded and scalable path toward optimizing agentic reasoning with retrieval.

    Source: §5 · p. 9

    The semantic-entropy reward can remain positive when the model confidently collapses onto an incorrect semantic class.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Agentic reasoning enables large reasoning models (LRMs) to dynamically acquire external knowledge, but yet optimizing the retrieval process remains challenging due to the lack of dense, principled reward signals. In this paper, we introduce InfoReasoner, a unified framework that incentivizes effective information seeking via a synthetic semantic information gain reward. Theoretically, we redefine information gain as uncertainty reduction over the model's belief states, establishing guarantees, including non-negativity, telescoping additivity, and channel monotonicity. Practically, to enable scalable optimization without manual retrieval annotations, we propose an output-aware intrinsic estimator that computes information gain directly from the model's output distributions using semantic clustering via bidirectional textual entailment. This intrinsic reward guides the policy to maximize epistemic progress, enabling efficient training via Group Relative Policy Optimization (GRPO). Experiments across seven question-answering benchmarks demonstrate that InfoReasoner consistently outperforms strong retrieval-augmented baselines, achieving up to 5.4% average accuracy improvement. Our work provides a theoretically grounded and scalable path toward agentic reasoning with retrieval. The code is available at https://github.com/dl-m9/InfoReasoner

1. Introduction

InfoReasoner frames retrieval as uncertainty reduction over answer beliefs and uses model outputs to construct a synthetic information-gain reward without manual retrieval annotations. The authors report improved reasoning accuracy across seven question-answering benchmarks using this reward to train retrieval policies.

1. Introduction

  • Existing retrieval optimization relies on human-annotated demonstrations, sparse task-specific rewards, or heuristic process supervision rather than a formal framework for retrieval value.The paper also identifies the lack of dense ground-truth annotations or oracle belief states needed by traditional information-gain metrics.Source: §1 · p. 1
  • InfoReasoner defines information gain as uncertainty reduction over belief states and establishes non-negativity, telescoping additivity, and information-channel monotonicity guarantees.The framework uses these properties to connect stepwise retrieval rewards with the goal of reducing uncertainty about the final answer.Source: §1 · p. 2
  • Its output-aware estimator samples answers under different evidence conditions, clusters semantically equivalent answers using bidirectional textual entailment, and computes a gold-class gain reward.The reward supplies intermediate retrieval supervision without manual annotations, while final-answer exact match anchors training to task correctness.Source: §1 · p. 2
  • Across seven question-answering benchmarks, InfoReasoner consistently improves reasoning accuracy over strong retrieval-augmented baselines, with up to 5.4% average improvement.The policy is optimized using Group Relative Policy Optimization (GRPO).Source: §1 · p. 2

2. Rethinking Information Gain as Uncertainty Reduction over Belief States

The section formalizes retrieval as information gain: the reduction in an agent’s uncertainty about a latent answer after observations. It establishes properties that connect local retrieval rewards to global uncertainty reduction and motivate a practical semantic reward.

2. Rethinking Information Gain as Uncertainty Reduction over Belief States

  • The reasoning process is modeled as a POMDP, with latent states, actions such as retrieval, and observations such as retrieved documents or tool outputs.The belief state is a posterior distribution over the latent task variable given the observation history; the practical approximation uses semantic classes of sampled answers.Source: §2 · p. 2, §2.1 · p. 2

2.2. Uncertainty Functional

  • The uncertainty functional maps belief states to non-negative uncertainty values, with certainty assigned zero uncertainty and uncertainty not increasing in expectation after new information.Shannon entropy is given as a typical example.Source: §2.2 · p. 3

2.3. Information Gain as Uncertainty Reduction

  • Under consistent Bayesian belief updates, expected information gain is non-negative, although misleading evidence can produce negative realized gain that penalizes confusing retrieval.The theoretical guarantee assumes the expected non-increase axiom for the uncertainty functional.Source: §2.3 · p. 4
  • Stepwise information gains telescope to the trajectory’s global reduction in uncertainty, supporting local retrieval credit assignment under ideal belief updates.The practical reward uses a supervised gold-class surrogate to align the signal with final-answer correctness.Source: §2.3 · p. 4
  • An observation channel that Blackwell-dominates another receives equal or higher expected information gain, ordering actions by informativeness.The paper gives a more reliable knowledge base or precise tool as examples of more informative sources.Source: §2.3 · p. 4

2.4. Interpretation

  • The framework treats information gain as reduction in a generalized uncertainty functional and motivates a tractable semantic reward that preserves the retrieval contrast while anchoring it to correctness.The practical estimator instantiates belief states as distributions over semantic equivalence classes of sampled answers.Source: §2.4 · p. 4

3. Method

InfoReasoner operationalizes uncertainty reduction as information gain over belief states represented by semantic answer classes. It uses this reward, alongside answer correctness, to optimize retrieval policies.

3. Method

  • The method approximates the belief state with a probability distribution over semantic equivalence classes and uses semantic entropy as its uncertainty measure.Information gain is the entropy reduction after retrieved evidence is incorporated.Source: §3 · p. 4

3.1. Semantic Clustering via Bidirectional Textual Entailment

  • Answer sequences are clustered as equivalent when they mutually entail one another in the question context, with connected components forming semantic classes.This targets semantic equivalence that token overlap or embedding similarity can miss.Source: §3.1 · pp. 4–5

3.2. Synthesizing Information Gain Reward via Semantic Uncertainty Reduction

  • The full-distribution estimator measures aggregate uncertainty reduction over semantic classes, but may reward a confident shift toward an incorrect class.The paper also defines a gold-class reward that tracks whether retrieval increases probability mass on the correct semantic class.Source: §3.2 · p. 5
  • The gold-class reward compares the correct class's probability with and without retrieved evidence, rewarding increases after retrieval.It provides a direct signal for retrieval that supports the correct answer.Source: §3.2 · p. 5

3.3. Policy Optimization with Information Gain Reward

  • GRPO optimizes retrieval using a weighted sum of exact-match correctness and cumulative information gain, balancing accurate answers with informative retrieval.GRPO estimates advantages from grouped sampled outputs, reducing training instability in sparse-reward settings.Source: §3.3 · pp. 5–6

4. Experiments

Experiments assess InfoReasoner across QA, tool-integrated, and long-horizon settings, then examine coefficient choice, training dynamics, reward behavior, and estimator efficiency.

4.1. Experimental Setup

  • Training uses NQ and HotpotQA; evaluation spans seven short-answer QA datasets plus MATH500 tool reasoning and WebDetective deep search against task-specific baselines.The seven QA benchmarks include NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, and Bamboogle.Source: §4.1 · p. 6

4.3. Ablation Study on the Information Gain Coefficient

  • 34.6% average accuracy at λ = 0.6 is highest; λ = 0.0 drops to 29.9%, while λ = 0.8 and 1.0 reach 30.4% and 32.0%.The λ = 0.0 condition degenerates to Search-R1; the reported optimum balances information gain with the output reward.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Training Dynamics Analysis

  • InfoReasoner grows more slowly in early EM than Search-R1, then accelerates and surpasses it in final EM after exploration of diverse retrieval strategies.The authors attribute early slower growth to rewarding useful retrievals even when the final answer is incorrect.Source: Figure 2 · p. 7, §4.4 · pp. 7–8
  • The IG component supplies a dense signal alongside sparse EM supervision, while entropy loss rises during exploration before decreasing and stabilizing at a lower level.The authors interpret the later entropy pattern as convergence to a more certain and confident policy.Source: Figure 2 · p. 7, §4.4 · p. 8
  • InfoReasoner produces response lengths approximately 30% shorter than Search-R1, and the authors report more efficient outputs without sacrificing accuracy.They attribute this concision to prioritizing key information from retrieved documents.Source: Figure 2 · p. 7, §4.4 · p. 8

4.2. Main Comparative Results

  • InfoReasoner-3B reaches 34.6% average accuracy across seven QA benchmarks, versus 30.1% for Search-R1-3B-Instruct; InfoReasoner-7B reaches 39.1%, best among evaluated 7B baselines.On MATH500 with Python tool execution, InfoReasoner-7B reaches 85.8% versus SimpleTIR-7B at 84.6%; on WebDetective, InfoReasoner-3B reaches 30.0% Pass@1 versus Search-R1-3B-Base at 26.5%.Source: §4.2 · p. 7, Table 4 · p. 8, Table 3 · p. 8

4.5. Study on the Information Gain Reward

  • Info B has higher median information gain than Info A, while jointly providing both documents yields more gain than their naive sum.The authors describe the joint effect as synergistic and relevant to gathering complementary evidence across reasoning hops.Source: §4.5 · pp. 8–9

4.6. Sensitivity Analysis of Group Size on Reward Estimation

  • M = 12 reaches MAE below 0.02 with 4× less computational overhead than the N = 64 oracle, while larger groups yield diminishing returns.The estimate error decays rapidly as group size increases; the authors adopt M = 12 to balance reward quality and training efficiency.Source: §4.6 · p. 9, Figure 4 · p. 9

5. Conclusion

InfoReasoner combines a belief-state information-gain reward with semantic output-based estimation to optimize retrieval without manual intermediate annotations. Across seven QA benchmarks, it achieves state-of-the-art performance at 3B and 7B scales.

5. Conclusion

  • InfoReasoner computes a synthetic semantic information-gain reward from semantic equivalence classes using bidirectional textual entailment, without manual intermediate retrieval annotations.The framework redefines information gain as uncertainty reduction over belief states and establishes formal guarantees.Source: §5 · p. 9
  • Across seven QA benchmarks, InfoReasoner achieves state-of-the-art performance at both 3B and 7B scales.The information-gain component enables beneficial exploration during training, leading to more robust retrieval policies.Source: §5 · p. 9

Impact Statement

The paper situates its retrieval-reasoning framework among agentic search, RAG, and RL work, then discusses its potential impact and theoretical assumptions. It also states an idealized belief-update assumption and the conditions for nonnegative expected information gain.

Impact Statement

  • The framework is presented as potentially supporting trustworthy agentic search, more resource-efficient retrieval, and semantic metrics beyond QA.The impact discussion connects uncertainty reduction to active verification, suggests searching when confidence is lacking to minimize API calls, and identifies scientific research, planning, and code generation as possible applications.Source: Impact Statement · p. 9

A.1. Agentic Reasoning

  • Agentic reasoning work combines LRMs with retrieval and tools, enabling dynamic information gathering during reasoning.Examples include Search-o1, Search-R1, and RAG-Gym, which use agentic RAG, multiple search queries, or policy optimization.Source: §A.1 · p. 13

A.2. Retrieval-Augmented Generation

  • Prior RAG work improves retrieval, integration, or generation, whereas this method proposes a synthetic semantic information-gain reward to guide RAG and reasoning end to end.The cited approaches include query decomposition, complexity-aware planning, evidence extraction, knowledge aggregation, and context-aware generation.Source: §A.2 · p. 14

A.3. Reinforcement Learning for LLM Reasoning

  • InfoReasoner differs from IGPO and EMPO by using a retrieval-specific gold-semantic-class reward to give dense credit to intermediate retrieval actions.IGPO rewards answer-probability changes at the surface-form level, while EMPO minimizes semantic entropy for general reasoning.Source: §A.3 · p. 14

B.1. Derivation of LLM Belief Update

  • The belief-update derivation simplifies the observation likelihood by assuming the LLM's interpretation of new evidence dominates its prior bias.The authors describe this conditional-independence assumption as an idealization; theoretical uncertainty-reduction bounds then apply to ideal grounding scenarios.Source: §B.1 · p. 15

B.2. Proof of Proposition 2.6 (Non-negativity under Ideal Updates)

  • Under ideal Bayesian updates, expected information gain is nonnegative and equals zero only when observations leave the posterior equal to the prior.With Shannon entropy, expected information gain coincides with conditional mutual information; realized information gain may be negative and penalize misleading retrievals.Source: §B.2 · p. 15

B.3. Proof of Proposition 2.7 (Telescoping Additivity)

Stepwise information gains telescope to the total uncertainty reduction over a complete reasoning trajectory. This makes maximizing dense local gains equivalent to minimizing final uncertainty about the answer.

B.3. Proof of Proposition 2.7 (Telescoping Additivity)

  • Summed stepwise information gains equal the initial uncertainty minus the final uncertainty over a trajectory.The intermediate uncertainty terms cancel, so maximizing immediate dense rewards optimizes the global objective of minimizing final answer uncertainty.Source: §B.3 · pp. 15–16

B.4. Proof of Proposition 2.8 (Monotonicity)

The monotonicity result favors information channels that Blackwell-dominate alternatives. The reward-computation procedure estimates semantic uncertainty before and after retrieval and uses the change in gold-class probability as information gain.

B.4. Proof of Proposition 2.8 (Monotonicity)

  • When one action's observation channel Blackwell-dominates another, the framework assigns it at least as much expected information gain.The paper says this property favors higher-quality sources, such as more reliable knowledge bases or more precise reasoning tools.Source: §B.4 · p. 16

C.1. Information Gain Reward Computation

  • The estimator samples candidate responses, clusters semantically equivalent answers, and measures the reduction in semantic entropy after evidence is added.It uses the change in probability assigned to the gold semantic class as the retrieval reward; the reported group size M = 12 balances estimation variance and computational efficiency.Source: §C.1 · pp. 16–17

C.2. InfoReasoner Training Algorithm

InfoReasoner trains retrieval and reasoning through parallel search-and-reason trajectories, combining outcome feedback with intermediate information-gain rewards. GRPO then favors trajectories whose composite rewards exceed the group average.

C.2. InfoReasoner Training Algorithm

  • For each question, the algorithm samples G parallel trajectories that alternate search-and-reason turns until an answer or the turn limit.Search turns retrieve top-k documents and receive an intermediate information-gain reward.Source: §C.2 · pp. 17–18
  • Each trajectory’s composite reward combines its Exact Match outcome with cumulative intrinsic information-gain rewards, balanced by λ.The intrinsic rewards are computed for recorded search steps after the reasoning process ends.Source: §C.2 · p. 18
  • GRPO updates the policy by comparing each trajectory’s reward with the group mean, favoring retrieval strategies that are more efficient and informative than average.Source: §C.2 · p. 18

D.1. Training Details

The training details specify the datasets, models, retrieval backends, and prompts used for InfoReasoner. The additional efficiency analysis reports training overhead and inference-time search-turn savings.

D.1. Training Details

  • Training used merged NQ and HotpotQA data, Qwen2.5-3B and Qwen2.5-7B backbones for main QA experiments, and specified GPU and retrieval setups.The method used full-parameter fine-tuning on 8 NVIDIA A800 GPUs; retrieval used E5 with a Wikipedia corpus for QA and BGE-M3 for real-time web tasks.Source: §D.1 · pp. 18–19, Table 5 · p. 19

D.2. Prompt Details

E. Additional Experimental Results

E.1. Training and Inference Efficiency

  • A 1.9× training slowdown over 300 steps adds 18.82 wall-clock hours, or 150.53 GPU-hours, from semantic information-gain estimation.This overhead arises from multiple samples and NLI checks and is incurred during RL training.Source: §E.1 · p. 20
  • On HotpotQA, InfoReasoner surpasses Search-R1’s peak performance after 2 turns, reducing retrieval calls by 66.7%; on 2Wiki, peak-matching takes 50.0% fewer turns.At inference, the learned policy requires no additional sampling or NLI checks.Source: Table 6 · p. 19, §E.1 · p. 20

E.2. Ablation Study on Search Turns

The search-turn analysis finds that InfoReasoner outperforms Search-R1 across the reported settings and that performance gains diminish after the early turns. Case studies illustrate how information-gain rewards respond to misleading evidence and useful retrieval even when final reasoning fails.

E.2. Ablation Study on Search Turns

  • From T = 1 to T = 2, InfoReasoner gains +30.4 points on HotpotQA (11.0% to 41.4%) and +28.6 points on 2WikiMultihopQA (1.6% to 30.2%).Search-R1 gains +22.6 points on HotpotQA (9.8% to 32.4%) and +17.8 points on 2WikiMultihopQA (1.6% to 19.4%).Source: §E.2 · p. 20, Figure 8 · p. 21
  • Performance gains diminish after T = 2: InfoReasoner plateaus at 42.6% on HotpotQA by T = 4 and at 39.5% on 2WikiMultihopQA by T = 4.Search-R1 plateaus at 34.6% on HotpotQA by T = 6 and 30.1% on 2WikiMultihopQA by T = 4.Source: §E.2 · pp. 20–21, Figure 8 · p. 21
  • At T = 2, InfoReasoner leads Search-R1 by +9.0 EM on HotpotQA (41.4% vs. 32.4%) and +10.8 EM on 2WikiMultihopQA (30.2% vs. 19.4%).Source: §E.2 · p. 21, Figure 8 · p. 21

E.3. More Case Studies

  • With irrelevant song information retrieved instead of the target magazine, InfoReasoner assigns a negative information-gain reward of −0.915.The case study characterizes the misleading context as increasing epistemic uncertainty.Source: §E.3 · p. 21, Figure 8 · p. 21
  • When retrieval identifies Samuel L. Jackson as the hitman, InfoReasoner assigns positive IG of 0.808 despite a later reasoning error, whereas EM gives the trajectory zero.The example distinguishes credit for useful retrieval from credit for the final answer.Source: §E.3 · p. 21

F. Limitations and Future Work

The framework’s main stated limitation is the training cost of estimating semantic information gain with sampled outputs and pairwise NLI. The passage confines this overhead to training and describes cost-reduction approaches as future work.

Limitations

  • Estimating semantic entropy requires multiple candidate sequences, such as M = 12, and pairwise NLI, adding computational burden compared with outcome-based RL.Source: F. Limitations and Future Work · p. 21

Future Work

  • The added overhead is confined to training; inference uses the learned policy without auxiliary sampling or NLI computation.Future work proposes hidden-state uncertainty estimates or calibrated embedding-based clustering to mitigate training cost and improve scalability to larger models and broader domains.Source: F. Limitations and Future Work · p. 22

Implications

  • The paper presents uncertainty reduction as a basis for agents that verify knowledge, with reliability particularly relevant to medical and legal information seeking.Source: F. Limitations and Future Work · p. 22
  • Case studies report positive information gain for successful evidence discovery despite reasoning failure and negative gain for misleading retrieval.The reported rewards are 0.808 for useful evidence and −0.915 for irrelevant context.Source: Figure 9 · p. 22, Figure 10 · p. 23
Loading 2602.00845v3…