PaperlayerView exact revision on arXiv

Source-linked AI summary

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, Bang Liu

arXiv:2603.00656v2cs.AI

TL;DR

Underspecified user requests require agents to gather information while making correct downstream decisions, but sparse, trajectory-level rewards hinder fine-grained learning. InfoPO rewards turns whose feedback changes subsequent action probabilities relative to a masked counterfactual, fuses this signal with task outcomes, and consistently improves performance and stability across interactive benchmarks. The paper also reports robustness to user-simulator shifts and generalization to environment-interactive tasks.

  • Problem

    Underspecified multi-turn requests and sparse, delayed trajectory-level rewards make it difficult to assign credit to information-gathering decisions.

  • Method

    InfoPO measures how feedback changes the policy’s next-action distribution against a masked-feedback counterfactual and combines information gain with outcome advantages using variance-gated fusion.

  • Results

    InfoPO consistently improves performance, sample efficiency, and training stability over prompting and reinforcement-learning baselines across three interactive benchmarks.

  • Takeaways & Limitations

    The method provides a task-agnostic, scalable learning signal for long-horizon interaction where group-relative updates can stall under sparse or non-discriminative rewards.

  • Takeaways & Limitations

    Experiments are conducted only in simulated benchmark environments, and deployment requires safeguards and user oversight because sensitive-information inference and intent misunderstanding may raise privacy and safety concerns.

Abstract

from arXiv · show

Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent's subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment-interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent-user collaboration. Code is available at https://github.com/kfq20/InfoPO.

1. Introduction

InfoPO addresses sparse, delayed credit in underspecified multi-turn tasks by rewarding information-bearing turns and adaptively aligning that signal with task outcomes. It is evaluated across interactive benchmarks and reported to improve task performance and training stability.

  • Motivation: User-centric agents must elicit missing constraints while advancing underspecified requests toward completion.The paper frames effective interaction as balancing intent clarification with task progress.
  • Motivation: Sparse, delayed rewards and trajectory-level aggregation make intermediate decisions difficult to credit, especially when GRPO groups have little reward variation.This can limit fine-grained supervision and produce near-zero advantage signals.
  • Method: InfoPO compares factual feedback with a masked-feedback counterfactual to reward turns whose feedback changes the policy’s next-action distribution.The resulting signal provides dense, turn-level information-gain credit.
  • Method: An adaptive variance-gated fusion increases information-gain weight when outcome rewards are non-discriminative and shifts toward task objectives when outcomes discriminate.This preserves a usable learning signal while maintaining task-oriented direction.
  • Theory: The information-gain reward corresponds in expectation to conditional mutual information, while its cumulative form represents directed information flow from feedback to decisions.The paper further states that minimum cumulative information gain is necessary for task success.
  • Evaluation: Across intent clarification, collaborative coding, and tool-augmented decision making, InfoPO improves task performance and learning stability over prompting and reinforcement-learning baselines.The study includes UserGym, ColBench, and τ 2-Bench, with ablations and diagnostics of interaction behavior.

2. Related Works

Related work spans user-centric intent and personalization, agentic reinforcement learning, and intrinsic or process-based reward shaping. These directions motivate finer-grained credit assignment for long interactive rollouts, which InfoPO targets without task-specific heuristics.

  • User-centric agents: User-centric agents infer latent intent, preferences, and user state through multi-turn interaction beyond simple task completion.Related work studies intention elicitation, clarification policies, transparency, controllability, and personalization benchmarks.
  • Open challenge: Prior advances leave fine-grained credit assignment as an open challenge in long interactive rollouts.InfoPO derives a turn-level learning signal without requiring task-specific heuristics.
  • Agentic reinforcement learning: Agentic reinforcement learning improves LLM decision-making through hierarchical training, collaborative training, grounded tool use, information seeking, and rollout-stability analysis.These studies cover search, clarification, tool actions, and long-horizon interaction dynamics.
  • Reward Shaping in RL: Reward shaping adds intrinsic signals such as novelty, prediction error, controllability, preference supervision, or process-level step scores when task rewards are sparse.Step-wise feedback has improved verification and self-correction in complex reasoning tasks.

3. Preliminaries

The paper models user-centric interaction as a Dec-POMDP in which an agent observes feedback about latent intent and updates its policy over actions. Optimization maximizes expected external return and uses GRPO-style group comparisons for token-level advantages.

  • 3.1. Multi-Turn Interaction as a Dec-POMDP: The user-centric task is modeled as a decentralized partially-observable Markov decision process in which the agent infers latent intent through repeated interaction.The environment transitions through latent states and emits observations based on those states.
  • 3.1. Multi-Turn Interaction as a Dec-POMDP: At turn t, the policy generates action tokens conditioned on the interaction history and current observation.The history includes the initial query and prior actions and observations.
  • 3.1. Multi-Turn Interaction as a Dec-POMDP: Observations reduce uncertainty about the task goal and shift the action distribution for the next turn.This feedback-to-action shift is the basis for InfoPO’s information-driven signal.
  • 3.2. Optimization with Group-Relative Policy Gradient: The optimization objective maximizes expected cumulative external return over trajectories sampled from the policy.For LLM agents, parameters are updated at the token level using an advantage signal.
  • 3.2. Optimization with Group-Relative Policy Gradient: GRPO estimates variance-reduced advantages by comparing trajectories within a rollout group without an explicit critic.InfoPO is built upon this group-relative policy optimization framework.

4. Methods

InfoPO assigns turn-level information credit by comparing factual and masked-feedback contexts, then combines it with outcome advantages through group-relative variance gating. Its analysis connects the signal to mutual information and establishes information progress as necessary for task success.

  • 4.1. Turn-level Counterfactual Information Gain: InfoPO computes turn-level information gain by comparing the policy’s next-action likelihood under factual and masked-feedback contexts.The reward measures how much received feedback changes the subsequent action distribution.
  • 4.1. Turn-level Counterfactual Information Gain: The same realized next-action tokens are evaluated in both contexts, isolating feedback effects and enabling efficient parallelized forward passes.Teacher forcing avoids stochastic generation differences and multiple autoregressive counterfactual rollouts.
  • 4.2. Unified Group-Relative Advantage Construction: InfoPO computes normalized outcome and information-gain advantages within rollout groups before broadcasting turn-level information rewards to corresponding action tokens.The outcome advantage is based on trajectory-level external rewards, while the information-gain advantage is mapped from each token to its interaction turn.
  • 4.2. Unified Group-Relative Advantage Construction: The variance gate increases information-gain influence when within-group outcomes are nearly indistinguishable and shifts weight toward task success as outcomes become discriminative.The temperature parameter controls the gate, while β controls the peak influence of information progress.
  • 4.2. Unified Group-Relative Advantage Construction: The unified advantage updates πθ against πref with a KL-divergence penalty to control distribution shift.The final objective uses the fused token-level advantage in the policy optimization procedure.
  • 4.3. Theory: Info-Gain as a Necessary Resource: The turn-level information-gain reward equals conditional mutual information between feedback and subsequent actions, while cumulative information gain has a necessary lower bound for task success.The theoretical results formalize information flow from feedback to decisions and link successful hidden-intent recovery to accumulated information progress.

5. Experiments

Across three interactive benchmarks and environment-generalization tests, InfoPO improves task performance, learning stability, and interaction quality by assigning fine-grained information-based credit and adaptively balancing it with task outcomes.

  • RQ1: Overall Performance: InfoPO improves performance across UserGym, ColBench, and τ 2-Bench, including gains on underspecified-goal and code-centric tasks.On UserGym it improves over the strongest baseline in 7 of 8 sub-environments; ColBench Pass reaches 0.534 vs. 0.457 and Success reaches 0.426 vs. 0.352.
  • RQ1: Overall Performance: 31.3%–38.4% of early UserGym/ColBench rollout groups and up to 76.3% in τ 2-Bench have zero outcome variance, limiting standard group-relative advantages.InfoPO uses dense info-gain rewards to bootstrap learning when outcome-based gradients are near zero.
  • RQ1: Overall Performance: InfoPO reaches higher reward levels earlier with reduced oscillations, while learned agents clarify intent early and use structured, proactive interaction strategies.The results connect dense turn-level credit with earlier optimization and observable behavioral maturation.
  • RQ2: InfoPO Mechanism Analysis: InfoPO adapts interaction depth to task structure, using explore-then-consolidate behavior on shorter tasks and pruning-first behavior on high-horizon τ 2-Bench.In UserGym and ColBench, early turns increase while responses shorten; in τ 2-Bench, both turns and response lengths decrease from training onset.
  • RQ2: InfoPO Mechanism Analysis: Turn-level rewards increasingly concentrate on early turns, producing a learned clarify-then-act pattern in which feedback shifts subsequent action distributions.The counterfactual objective rewards informative questions rather than simply longer interaction.
  • RQ2: InfoPO Mechanism Analysis: Variance-gated fusion keeps information seeking task-relevant: targeted clarifications comprise 58.8% of high-rinfo turns versus 12.4% for generic open questions, with weak length correlation of 0.157.Removing the gate increases generic questioning, interaction length, and response length; removing standardization also increases length sensitivity and degrades performance.
  • RQ3: Generalization: InfoPO generalizes beyond user-centric benchmarks, maintaining an upward success trend on Sokoban and WebShop where standard GRPO baselines collapse into repetitive templates.The results frame multi-turn interaction as active uncertainty reduction across environment-interactive tasks.

6. Conclusion

InfoPO provides turn-level credit by measuring how observations change next-action distributions and fuses this signal with outcome advantages. It improves performance, sample efficiency, and training stability across benchmarks while generalizing to shifted simulators and non-user environments.

  • InfoPO measures how each observation counterfactually changes the policy’s next-action distribution to provide dense, turn-level credit.The intrinsic signal is fused with outcome advantages through variance gating.
  • Across three interactive benchmarks, InfoPO consistently improves performance, sample efficiency, and training stability over prompting and reinforcement-learning baselines.
  • InfoPO generalizes to user-simulator shifts and non-user environment-interaction tasks.

Impact Statement

InfoPO is presented as an information-driven method for interactive agents, with a training procedure and information-theoretic analysis spanning benchmark interaction and latent-intent inference. Its stated impact includes improved reliability, alongside privacy and safety concerns and a simulated-environment scope boundary.

  • Impact and limitations: The impact statement says clarification before action may improve reliability, but warns that sensitive inference and misunderstood intent raise privacy and safety concerns.
  • Impact and limitations: Experiments are conducted only in simulated benchmark environments, limiting the demonstrated scope of deployment evidence.
  • Method: InfoPO’s training procedure samples environment or simulator trajectories, computes external and counterfactual information-gain rewards, and combines standardized advantages through variance gating.The factual and masked-feedback contexts support the intrinsic comparison, while the gate emphasizes information gain when external rewards are non-discriminative.
  • Information-theoretic analysis: The information-gain signal is interpreted as conditional mutual information per turn and directed information cumulatively.
  • Information-theoretic analysis: For hidden-intent tasks, the analysis connects successful intent inference to information flow from feedback to actions under causal-policy assumptions.
  • Evaluation setting: The evaluated benchmarks require agents to balance information gathering with environment-grounded execution in multi-turn settings.UserGym uses an Action–Search–Answer interface and starts from underspecified requests or latent goals.

A.4. Additional diagnostics used in ablations

The diagnostics define performance, regression, collapse, interaction length, and verbosity-related metrics, then describe implementation details for counterfactual information-gain computation. Compute cost comes from paired teacher-forced evaluations, but batching and shorter prefixes reduce practical overhead.

  • Evaluation diagnostics: Final performance is the mean benchmark score at the final checkpoint, while best-to-final drop measures late-training regression.Larger best-to-final values indicate stronger regression after a good checkpoint.
  • Evaluation diagnostics: Collapse probability marks seeds whose final score falls below half their own best checkpoint.
  • Evaluation diagnostics: Interaction diagnostics measure average turns, average response length, utilization of the per-turn generation cap, and length–reward correlation.
  • Intrinsic-reward diagnostics: InfoPO computes each valid turn’s reward as the difference between factual and masked-feedback next-action log-likelihoods under teacher forcing.This adds policy forward evaluations without requiring extra environment interactions.
  • Compute cost: Each KL mini-batch requires two teacher-forced forward calls, so small BKL values increase mini-batch and forward-invocation counts.
  • Compute cost: Batched teacher-forced evaluation and shorter effective prefixes keep observed wall-clock overhead below a naive 2× rollout-cost estimate.The implementation recommends making BKL as large as GPU memory permits.

B.3. Sensitivity Analysis

The sensitivity analyses test whether InfoPO depends on mask design and information-gain weighting. Mask variants produce consistent divergence behavior, while performance remains stable for moderate β but degrades when information gain is overweighted.

  • Masking strategies: Four masking strategies evaluate whether the placeholder representing absent feedback changes InfoPO’s intrinsic divergence signal.The strategies include a default string, alternative string, random tokens, and a fixed mask token.
  • Masking strategies: Mean KL divergence is nearly identical across strategies: Default (0.207), Alternative String (0.193), Random Tokens (0.220), and Fixed Mask Token (0.205).
  • Masking strategies: The maximum relative difference from the default is 6.44% for Alternative String, below the stated 10% significance threshold.
  • Masking strategies: All masking comparisons are non-significant with all p > 0.24, supporting robustness to placeholder implementation.
  • Information-gain weight: InfoPO maintains high, stable performance for β from 0.1 to 0.5 but degrades when β reaches 2.0.The analysis frames excessive information-gain weight as weakening alignment with the task objective.

B.4. More Results

Additional analyses extend diagnostics across benchmarks and stratify TravelGym by subset, while tracking interaction length, turns, information gain, and reward-weight sensitivity.

  • TravelGym analysis: TravelGym is broken down across eight subsets that vary in constraint density and latent preference structure.The breakdown complements aggregate UserGym results by showing where improvements concentrate and which harder variants remain challenging.
  • Additional diagnostics: Figure 9 adds UserGym and τ 2-Bench training trajectories, information-gain curves, and sensitivity analysis for β from 0.1 to 2.0.The diagnostics cover response length, interaction turns, absolute information-gain reward, and information-gain ratio.

B.5. Trajectory-Quality Diagnostics

Trajectory-quality diagnostics indicate that InfoPO’s information-gain reward concentrates on useful downstream behavior rather than arbitrary likelihood changes or verbosity.

  • Next-action sensitivity: 0.275 vs. 0.098: good next actions show higher observation sensitivity than meaningless actions on τ 2-Bench.For tool calls, the corresponding averages are 0.553 and 0.110, respectively.
  • Current-action quality: 0.380 vs. 0.017: good current actions receive much larger mean information-gain rewards than meaningless actions.The medians are 0.247 and 0.012, with a highly significant difference of p = 1.4 × 10^-108.
  • Transition quality: Good current actions are more likely to produce productive continuations, whereas meaningless actions more often lead to meaningless behavior.This transition pattern supports crediting actions whose feedback changes subsequent task-relevant decisions.
  • Support assumption: The diagnostic interpretation assumes that behavior-policy and user/environment interactions include some non-degenerate, task-relevant transitions.When such transitions exist, the counterfactual signal can distinguish them from meaningless actions despite sparse final rewards.
  • Reward-hacking diagnostics: 58.8% vs. 12.4%: high-information-gain turns are more often targeted clarifications than generic open questions.Targeted clarifications also receive higher average information-gain reward, 0.3589 vs. 0.2029.
  • Variance-gated fusion: Removing the variance gate increases generic open questions, response length, and average interaction turns.The ablation raises turns from 10.4 to 13.5 and response length from 105.3 to 140.7, while the length-matched gap becomes negative.

C. Limitations

InfoPO has computational, modality, and simulator-fidelity limitations that constrain its current evaluation and training setup.

  • Limitations: InfoPO requires an extra model evaluation per turn for counterfactual masking, marginally increasing training time over standard GRPO.Current evaluations are text-centric and remain subject to the logical fidelity of simulated users.
  • Limitations: The current evaluation does not extend to multimodal or vision-language tasks.The paper also notes dependence on the logical fidelity of simulated users during training.

D. Case Studies

Case studies illustrate InfoPO’s interaction patterns across travel planning, collaborative coding, and tool-augmented troubleshooting, alongside simulator-instruction variants for each environment.

  • UserGym: In UserGym, the agent resolves an underspecified request through targeted clarification, tool retrieval, user confirmation, and final booking.The example illustrates goal-directed disambiguation and tool-grounded completion.
  • ColBench: In ColBench, the agent gathers information iteratively before producing code matching the clarified nested-list structure.The case demonstrates collaborative programming with progressive feedback.
  • τ 2-Bench: In τ 2-Bench, sequential checks and configuration fixes diagnose a mobile-data issue before a speed test validates recovery.Each query or action reduces uncertainty and enables the next corrective step.
  • Simulator instructions: Optimized simulator instructions emphasize balanced information disclosure and progress awareness to support denser learning signals.These optimized prompts support sensitivity analysis, while primary comparisons use the environments’ original prompts.
  • Simulator instructions: τ 2-Bench simulator variants address premature termination and improper tool execution, with tool-augmented guidance reserved for Telecom troubleshooting.Airline and Retail use dialogue-based guidance, whereas Telecom uses diagnostic tools such as status checks and speed tests.
Loading 2603.00656v2…