PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents

Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, Zhenzhe Ying

arXiv:2510.14967v2cs.CLcs.AIcs.LG

TL;DR

Multi-turn search agents receive sparse final-answer supervision that causes advantage collapse, weak intermediate credit assignment, and inefficient learning. IGPO instead rewards turn-level increases in ground-truth answer probability alongside outcome rewards, and experiments show consistently higher accuracy and sample efficiency than strong baselines, particularly for smaller models.

  • Problem

    Outcome-only rewards provide supervision only at the final answer, causing advantage collapse, weak fine-grained credit assignment, and poor sample efficiency in multi-turn training.

  • Method

    IGPO assigns each interaction turn an intrinsic reward equal to the marginal increase in the policy’s probability of producing the ground-truth answer, combined with outcome supervision.

  • Results

    IGPO consistently outperforms strong baselines across in-domain and out-of-domain benchmarks, improving accuracy and sample efficiency, especially for smaller models.

  • Takeaways & Limitations

    Dense, ground-truth-aware turn-level supervision provides stable training signals for multi-turn search agents while preserving alignment with the final objective.

  • Takeaways & Limitations

    IGPO relies on ground-truth answers, limiting its applicability in open-ended settings.

Abstract

from arXiv · show

Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely on outcome-based rewards that are only provided exclusively upon generating the final answer. This reward sparsity becomes particularly problematic in multi-turn settings, where long trajectories exacerbate three critical issues: (i) advantage collapse, where all rollouts receive identical rewards and provide no useful learning signals; (ii) lack of fine-grained credit assignment, where the correctness of intermediate turns is obscured, especially in long-horizon tasks; and (iii) poor sample efficiency, where each rollout yields only a single outcome signal, leading to low data utilization. In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's probability of producing the correct answer. Unlike prior process-level reward approaches that depend on external reward models or costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense reward signals. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines in multi-turn scenarios, achieving higher accuracy and improved data efficiency. Our code is available at https://github.com/GuoqingWang1/IGPO.

1 INTRODUCTION

Multi-turn search agents need denser supervision than final-answer rewards provide. IGPO supplies turn-level information-gain rewards and consistently improves accuracy and sample efficiency across benchmarks.

  • Motivation: Outcome-only rewards can collapse to zero group-relative advantages when rollouts receive identical answers, eliminating useful gradient signals.This issue is especially substantial for smaller models handling complex queries.
  • Motivation: Final-answer supervision obscures the correctness of intermediate turns and wastes dense trajectory information, reducing credit-assignment quality and sample efficiency.
  • IGPO: IGPO defines each turn’s reward as the marginal increase in the policy’s probability of producing the correct answer, while retaining outcome supervision.
  • Results: IGPO consistently outperforms strong baselines across in-domain and out-of-domain benchmarks, improving answer accuracy and sample efficiency, especially for smaller models.

2 PRELIMINARIES

The agentic RL setup models tool-using rollouts as sequences of reasoning, tool calls, tool responses, and a final answer. GRPO optimizes these rollouts with group-normalized outcome rewards while masking environment responses.

  • Task formulation: Each rollout solves a question through iterative tool interaction, with earlier turns containing reasoning, tool calls, and responses, followed by a rationale-then-final-answer turn.
  • Policy optimization: GRPO normalizes returns within sampled rollout groups and optimizes a clipped surrogate objective with KL regularization.
  • Training pipeline: Optimization updates only decision tokens, including reasoning, tool calls, and answers, while masking tool-response tokens.
  • Rewards: Outcome rewards combine answer correctness, measured by word-level F1, with a negative format penalty for malformed outputs.

3 INFORMATION GAIN-BASED POLICY OPTIMIZATION

IGPO turns multi-turn search into incremental ground-truth acquisition by rewarding changes in answer probability. It combines these intrinsic rewards with outcome supervision and discounted returns for dense, future-aware policy optimization.

  • 3.1 MOTIVATION: IGPO addresses advantage collapse and coarse credit assignment by measuring each turn’s increase in confidence that the final answer is correct.
  • 3.2 INFORMATION GAIN-BASED TURN-LEVEL REWARD: The information-gain reward is the difference between consecutive states’ log probabilities for the ground-truth answer.For turn t, it is defined as log πθ(a | q, o_i,≤t) − log πθ(a | q, o_i,≤t−1).
  • 3.2 INFORMATION GAIN-BASED TURN-LEVEL REWARD: Information-gain rewards provide ground-truth-aware, dense supervision that reduces reliance on external reward models and Monte Carlo estimation.
  • 3.2 INFORMATION GAIN-BASED TURN-LEVEL REWARD: A vectorized implementation appends ground-truth copies and uses a custom attention mask to compute all turn probabilities in one forward pass.The implementation achieves an asymptotic speedup of approximately T/3, such as 3× for T = 10.
  • 3.3 POLICY OPTIMIZATION WITH TURN-LEVEL DISCOUNTED RETURN: IGPO computes discounted returns from turn rewards so each turn reflects future effects, then assigns the return to all decision tokens in that turn.The discount factor satisfies γ ∈ (0, 1].

4 EXPERIMENTS

Experiments across seven in-domain and out-of-domain datasets show that IGPO outperforms prompt-based, outcome-reward, step-reward, and standard RL baselines while improving training efficiency and stability.

  • Overall Performance: Training-based methods substantially outperform prompt-based approaches across the evaluated datasets.The result supports explicit policy optimization over prompting alone for agentic search.
  • Overall Performance: 60.2 average score, +6.3 over DeepResearcher: IGPO achieves the best overall performance across in-domain and out-of-domain datasets.The comparison covers seven datasets and attributes the gains to intrinsic, ground-truth-aware credit at every turn combined with outcome rewards.
  • Overall Performance: IGPO achieves the highest overall score among PPO, Reinforce++, RLOO, GRPO, and GSPO, with denser turn-level supervision than outcome-only methods.The authors link this supervision to stronger generalization and more efficient training.
  • Ablation Study: Combining information-gain and outcome rewards outperforms using either information gain or outcome reward alone.The two components provide complementary intermediate-step guidance and final-task alignment.
  • Ablation Study: +16.6 points on the 3B model versus GRPO, compared with +8.3 points on the 7B model, showing larger improvements for the smaller model.The reported scores are 32.3 →48.9 for 3B and 51.9 →60.2 for 7B.
  • In-Depth Analysis: IGPO learns faster and more stably, achieves greater entropy reduction than GRPO, and improves performance with fewer training tokens.The training-curve and entropy analyses attribute these dynamics to denser, more informative gradients.

5 RELATED WORK

Research on RL-based search agents has progressed from retrieval-oriented rewards to multi-turn process supervision, but sparse outcome rewards remain a central challenge. Recent approaches address intermediate credit assignment through techniques such as MCTS and retrieval-memory supervision, while incurring stability or efficiency concerns.

  • IGPO incurs 0.0227s per step, increasing information-gain reward computation by <0.4% and end-to-end cost by <0.02%.
  • Search-agent RL evolved from PPO with retrieval metrics toward multi-turn systems such as Search-R1, DeepResearcher, and ReSearch.
  • Outcome-only rewards remain sparse in multi-turn settings, often causing unstable optimization and inefficient sample utilization.
  • Process-level methods address intermediate actions using techniques including Monte Carlo Tree Search and retrieved-document memory vectors.

6 CONCLUSION, LIMITATIONS AND FUTURE WORK

IGPO provides intrinsic, ground-truth-aware supervision at every turn while preserving alignment with the final objective. Across in-domain and out-of-domain benchmarks, it improves accuracy and sample efficiency, especially for smaller models, but requires ground-truth answers and therefore has limited applicability in open-ended settings.

  • IGPO consistently outperforms strong baselines across in-domain and out-of-domain benchmarks, with higher accuracy and better sample efficiency.
  • IGPO supplies intrinsic ground-truth-aware supervision at every turn while preserving alignment with the final objective.
  • The method is particularly effective for smaller models, where sparse rewards are most problematic.
  • IGPO remains robust to reward hacking and introduces negligible computational costs.
  • Ground-truth answer availability limits IGPO’s applicability in open-ended settings.

ETHICS STATEMENT

The paper reports adherence to ethical research practices, including dataset-use compliance, privacy protection, and avoidance of human or animal subjects.

  • No human or animal subjects were involved, and the listed datasets were used according to their respective guidelines.
  • The study included no personally identifiable information and reports avoiding potential privacy or security risks.
  • The authors state that the research was conducted with transparency and integrity.

REPRODUCIBILITY STATEMENT

The paper emphasizes reproducibility through public code and datasets, detailed experimental documentation, and an anonymous repository for replication and verification.

  • Code and datasets are publicly available in an anonymous repository to facilitate replication and verification.
  • The appendix documents training procedures, model configurations, and hardware specifications for reproducing the experiments.
  • The authors state that these reproducibility measures are intended to support other researchers’ replication and further contributions.

A THEORETICAL ANALYSIS

Theoretical analysis connects IGPO’s process reward to information loss, cumulative snowball error, and final-answer error. Under the stated assumption, maximizing process reward minimizes an upper bound on accumulated reasoning errors and lowers the theoretical minimum final error rate.

  • Process reward is defined as dense per-turn information gain, measured by the increase in correct-answer log-probability from the previous state.The total process reward is the cumulative sum across turns, and the training objective maximizes it.
  • The final-answer error probability is lower-bounded by cumulative snowball error accumulated during the reasoning process.Cumulative snowball error is defined as the sum of per-turn information losses before the final answer.
  • The analysis treats final-answer generation as the last step of a multi-step process conditioned on information accumulated over earlier turns.The proof sketch applies an external theorem stating that step error is lower-bounded by average accumulated snowball error.
  • Under Assumption A.3, expected process reward is linked to per-turn information loss through a bounded, monotonically non-increasing convex function.Theorem A.4 uses this assumption to upper-bound cumulative snowball error.
  • Maximizing process reward minimizes an upper bound on cumulative snowball error, which lowers the theoretical minimum final error rate.The analysis links IGPO’s objective to reduced error accumulation and improved task-success probability.

C MORE IMPLEMENTATION DETAILS

The implementation section describes IGPO’s training setup, reward-normalization choices, computational profile, and comparisons with other process-reward methods. IGPO uses on-policy incremental information-gain rewards without annotation or Monte Carlo estimation.

  • Training Setup: Training experiments use 8 NVIDIA A100-80G GPUs, with detailed hyperparameters reported in Table 4.
  • Information-Gain Bases and Normalization: LogProb with Separate normalization outperforms the other evaluated information-gain and normalization settings.Table 5 compares probability-based joint/separate normalization with log-probability-based separate normalization.
  • Comparison with Other Process-Reward Methods: ReasoningRAG relies on MCTS-based step labeling and off-policy DPO, while StepSearch uses resource-intensive golden annotations.The cited methods are described as facing value-estimation, annotation-cost, or annotator-bias concerns.
  • Comparison with Other Process-Reward Methods: IGPO requires no annotation or Monte Carlo estimation and uses ground-truth-aware incremental rewards within the on-policy GRPO setting.The comparison positions IGPO against MCTS labeling, golden annotations, and Monte Carlo-based value estimation.
  • Training Efficiency: Sampling dominates training-step time, while recomputing log-probabilities takes less than 5% and return or advantage computation takes less than 1%.The timing comparison reports IGPO’s profile as matching GRPO’s.

D.4 SPURIOUS CORRELATIONS ANALYSIS

The spurious-correlation analysis tests whether reasoning traces remain informative after final answers are removed. IGPO traces outperform GRPO traces on all seven datasets for both model sizes, alongside larger out-of-domain gains.

  • The analysis uses a teacher LLM to infer answers from intermediate reasoning traces, measuring trace informativeness after final-answer removal.
  • IGPO reasoning traces outperform GRPO traces on teacher-inferred answer accuracy across all seven datasets for both 3B and 7B models.The evaluation retains only intermediate traces from samples that both agents originally answered correctly, then removes final answers before teacher inference.
  • OOD average gains over GRPO reach 13.7% for 7B models and 55.2% for 3B models, versus 12.6% and 42.8% in-domain.The comparison covers NQ, TQ, HotpotQA, and 2Wiki in-domain, and Musique, Bamboogle, and PopQA out-of-domain.

E FAILURE ANALYSIS

IGPO’s degradation is minor overall but concentrates on ambiguous questions whose designated ground truth excludes other valid answers. Its additional computation is reported as negligible relative to GRPO.

  • Approximately 3.6% of test samples show lower performance for IGPO than GRPO across datasets.
  • Because IGPO’s turn-level reward depends on ground-truth probabilities, noisy or defective ground truths can suppress valid reasoning and tool use.
  • Ambiguous questions with multiple valid answers are identified as the primary cause of IGPO’s performance degradation.
  • On “Who is the author of Childhood?”, IGPO penalizes a factually correct non-ground-truth answer, producing Info Gain = -0.81.
  • IGPO’s extra recompute-log-prob computation is reported as about one-thousandth of the original FLOPs and about one ten-thousandth of the additional per-step time.
Loading 2510.14967v2…