PaperlayerView exact revision on arXiv

Source-linked AI summary

T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning

Haixin Wang, Hejie Cui, Chenwei Zhang, Xin Liu, Shuowei Jin, Shijie Geng, Xinyang Zhang, Nasser Zalmout, Zhenyu Shi, Yizhou Sun

arXiv:2605.02178v1cs.AI

TL;DR

Multi-turn RL training remains unstable because agents can spend interactions on low-information reasoning and repetitive turns. T2PO controls exploration at token and turn levels, improving stability, efficiency, and task performance across multi-turn benchmarks.

  • Problem

    Multi-turn RL lacks a stable, scalable training paradigm amid difficult credit assignment and inefficient exploration that can destabilize policy updates.

  • Method

    T2PO monitors intrinsic uncertainty to intervene when token-level information gain saturates and resamples turn-level trajectories with negligible exploration progress.

  • Results

    Across challenging multi-turn agentic benchmarks, T2PO consistently improves training stability, exploration efficiency, and task performance while mitigating training collapse.

  • Takeaways & Limitations

    Fine-grained regulation using intrinsic signals can jointly improve exploration efficiency and training effectiveness without additional reward shaping.

Abstract

from arXiv · show

Recent progress in multi-turn reinforcement learning (RL) has significantly improved reasoning LLMs' performances on complex interactive tasks. Despite advances in stabilization techniques such as fine-grained credit assignment and trajectory filtering, instability remains pervasive and often leads to training collapse. We argue that this instability stems from inefficient exploration in multi-turn settings, where policies continue to generate low-information actions that neither reduce uncertainty nor advance task progress. To address this issue, we propose Token- and Turn-level Policy Optimization (T$^2$PO), an uncertainty-aware framework that explicitly controls exploration at fine-grained levels. At the token level, T$^2$PO monitors uncertainty dynamics and triggers a thinking intervention once the marginal uncertainty change falls below a threshold. At the turn level, T$^2$PO identifies interactions with negligible exploration progress and dynamically resamples such turns to avoid wasted rollouts. We evaluate T$^2$PO in diverse environments, including WebShop, ALFWorld, and Search QA, demonstrating substantial gains in training stability and performance improvements with better exploration efficiency. Code is available at: https://github.com/WillDreamer/T2PO.

1. Introduction

The introduction attributes multi-turn RL training collapse to insufficient exploration, or “hesitation,” which produces uninformative behavior, noisy credit assignment, and unstable updates. T2PO addresses this through uncertainty-guided token interventions and turn-level resampling, improving exploration efficiency and training outcomes.

  • Motivation: Multi-turn RL lacks a stable and scalable training paradigm, while efficiency-oriented solutions can introduce off-policy drift and stale-policy effects that amplify instability and cause training collapse.The introduction frames effectiveness and efficiency as intertwined challenges in long-horizon, sparse-reward interactions.
  • Motivation: The authors identify insufficient exploration as the root cause of instability and call its systematic violation of the exploration–exploitation trade-off “hesitation.”At the token level, agents overthink after information gain saturates while sampling noise accumulates.
  • Motivation: Hesitant agents may leave the successful action space early, repeat unproductive turns, and introduce credit-assignment noise, unstable gradients, and high-variance policy updates.These behaviors leave little chance of recovery within a limited interaction budget.
  • Method: T2PO monitors uncertainty dynamics and triggers token-level thinking intervention when marginal uncertainty change falls below a threshold.Its self-calibrated uncertainty signal fuses entropy and confidence during rollouts.
  • Method and Results: At the turn level, T2PO dynamically resamples interactions with negligible exploration progress to avoid wasted rollouts, improving exploration efficiency without additional reward shaping.Experiments, ablations, and analyses on challenging multi-turn agentic benchmarks support its effectiveness and superiority.

2. Related Works

Related work has developed frameworks for agentic RL, heuristic trajectory selection for stabilizing multi-turn training, and uncertainty-based internal rewards for sparse-reward settings. These approaches motivate explicit regulation of reasoning dynamics and exploration within trajectories.

  • RAGEN established a unified framework for training and benchmarking agentic RL systems.
  • SimpleTIR filters rollouts with void turns, while rStar2-Agent oversamples rollout groups and retains high-quality trajectories to improve multi-turn training stability.Both methods rely on external heuristic filtering rather than explicitly regulating reasoning dynamics within trajectories.
  • Recent agentic RL methods derive denser internal rewards from uncertainty, typically measured by policy entropy, but entropy can encourage either confidence or exploration.SEED-GRPO incorporates high-entropy exploration into advantage estimation, whereas other methods minimize entropy to encourage confident predictions.

3. Preliminaries

The paper models multi-turn agentic RL as an LLM agent interacting with an external environment, producing tokenized reasoning-and-action outputs under a policy and receiving rewards across a trajectory.

  • Agentic RL framework: Each task starts from a user prompt q and unfolds over turns k = {1, 2, . . . , K}, with environment states s_k ∈ S.The environment returns the next state after each action unless the maximum turn count K is reached.
  • Agentic RL framework: At each turn, the agent generates an action a_k ∈ V^n from the tokenizer vocabulary, combining thinking tokens a_k^c and action tokens a_k^o within special tags.The action is a sequence of tokens with maximum response length T.
  • Agentic RL framework: The policy π_θ(a_k|s_k, q) defines output probabilities, while rewards r_k ∈ R and subsequent states form the completed trajectory τ.The trajectory is τ = {(s_1, a_1, r_1), (s_2, a_2, r_2), . . . , (s_K, a_K, r_K)} after turn K.

4. Method

T²PO controls exploration at token and turn levels using a self-calibrated uncertainty signal. It truncates redundant reasoning when predictive distributions stabilize and resamples turns unlikely to advance the agent toward success.

  • Self-calibrated uncertainty signal: Entropy and confidence have complementary blind regions, so T²PO combines them into a non-degenerate uncertainty signal M_t that captures residual probability-mass differences.M_t preserves top-1-driven stratification while adding curvature within each stratum, distinguishing distributions with identical maximum probability but different residual allocations.
  • Token-level thinking intervention: TTI monitors temporal changes in M_t and identifies the first sustained low-variation point after a minimum prefix, where further reasoning contributes little new information.A trailing window of size N and tolerance ε prevent premature or spurious triggers, while task-relevant exploratory tokens remain preserved.
  • Token-level thinking intervention: At the detected stopping point, TTI forces the reasoning terminator </think> and injects the deterministic queue [</think>, \n, <action>] to separate reasoning from execution.The forced logits operation deterministically ends reasoning, and the queue tokens are emitted without sampling.
  • Turn-level dynamic sampling: At the turn level, TDS discards and resamples the current turn whenever Γ_k < η, repeating until a satisfactory trajectory is found or the resampling budget is exhausted.The control signal is recomputed only after regeneration completes, targeting turns whose actions are unlikely to advance the agent efficiently.

5. Experiment

T2PO is evaluated on WebShop, ALFWorld, and Search QA, where it improves task performance, exploration efficiency, and training stability. Ablations show that RFT cold-start, token-level intervention, and turn-level regeneration each contribute to these gains.

  • Evaluation Setup: T2PO is evaluated on WebShop, ALFWorld, and Search QA, spanning online shopping, embodied multi-step interaction, and single- and multi-hop question answering.WebShop contains over 1.1M products and 12k user instructions, while ALFWorld includes 3,827 task instances across six categories.
  • Main Results: RL substantially improves WebShop and ALFWorld performance, with GRPO clearly outperforming PPO among single-turn baselines.Direct prompting and instruction tuning provide limited reliable task completion compared with RL-based training.
  • Main Results: T2PO achieves top performance across single-hop QA and more than doubles prior best performance on MuSiQue, while also showing strong results on 2Wiki and Bamboogle.The results indicate improved evidence retrieval, grounding, multi-step reasoning, and out-of-domain generalization.
  • Ablation Study: Without RFT cold-start, task score and success rate noticeably degrade because malformed or low-quality actions are no longer filtered during early policy optimization.The ablation identifies structured rejection fine-tuning as important for stabilizing training and preventing downstream rollout error propagation.
  • Ablation Study: Removing TTI causes redundant low-information reasoning and lower success rates, while removing TDS causes repeated cross-turn traces and deteriorates both task score and success rate.TTI adaptively terminates reasoning when predictive stability is reached; TDS regenerates turns to preserve trajectory-level diversity.
  • Stability and Efficiency: T2PO improves steadily without collapse across three environment seeds, while successful trajectories use fewer tokens and achieve consistently higher exploration efficiency than the SOTA baseline.The token-consumption distribution is substantially lower for T2PO, and its exploration-efficiency curve remains higher for successful trajectories.

6. Conclusion

T2PO regulates reasoning at both token and turn levels with intrinsic signals, suppressing low-information actions and mitigating training collapse without additional reward shaping. Experiments show consistent improvements in training stability, exploration efficiency, and task performance.

  • Conclusion: T2PO regulates reasoning at both token and turn levels using intrinsic signals.This design targets exploration control at two granularities.
  • Conclusion: T2PO suppresses low-information actions and mitigates training collapse without additional reward shaping.Its stabilization mechanism relies on intrinsic signals rather than externally added reward terms.
  • Conclusion: Experiments consistently improve training stability, exploration efficiency, and task performance with T2PO.The reported gains span all three evaluation dimensions.

Impact Statement … A.2. Hyper-parameter Setting

The paper frames inefficient exploration as a fundamental source of instability in multi-turn reasoning-agent RL and proposes token- and turn-level uncertainty control for more stable training. It evaluates task completion across WebShop, ALFWorld, and Search QA using task-specific metrics under a standardized training design.

  • Impact Statement: The work attributes multi-turn RL training collapse to inefficient exploration and introduces token- and turn-level uncertainty control as a general stabilization framework.The authors expect this approach to support scalable and reproducible training of interactive LLM agents.
  • A.1.1. WEBSHOP: WebShop evaluates task completion using six complementary metrics, including Task Score, defined as 10 × avg. reward, and Success Rate, the proportion of episodes with terminal reward r = 1.Success Rate can equal 1 even when the selected product does not exactly match the annotated target because multiple products may satisfy an instruction.
  • A.1.1. WEBSHOP: WebShop instructions combine target attributes, option field–value pairs, and a price constraint, enabling lightweight, scalable data collection while preserving realistic user intent.The terminal reward is received when the agent chooses a product at the end of an episode.
  • A.1.1. WEBSHOP: The WebShop reward includes a TextMatch-based type reward that penalizes category mismatches even when predicted and target products share attributes or options.The passage illustrates this distinction with butter and plant-based meat.
  • A.1.2. ALFWORLD: ALFWorld follows the standard evaluation protocol across 3,827 household-activity tasks spanning six categories, with Success Rate as the primary metric.Overall performance aggregates results across all six task categories unless otherwise specified.
  • A.1.3. SEARCH QA: Search QA uses Exact Match (EM), comparing the extracted answer between <answer> and </answer> tokens with the ground-truth answer by exact string matching.EM is both the outcome-based RL reward and the final testing metric.
  • A.1.3. SEARCH QA: Search QA reports EM across seven benchmark datasets covering general and multi-hop question answering tasks.The outcome-only metric reflects query formulation, external-knowledge retrieval, and evidence integration during multi-step reasoning.
  • A.2. Hyper-parameter Setting: The experiments use a unified hyperparameter design in which model family, optimization strategy, and rollout framework are shared, while task-specific parameters reflect environment characteristics.This standardization is intended to reduce confounding from inconsistent training setups so performance differences can be attributed to algorithmic behavior.

B. RL Training Techniques … D. More Experimental Results

The paper combines rejective initialization, format control, trajectory decomposition, and turn-relative advantages to stabilize multi-turn RL, while relating these techniques to prior methods for controlling reasoning length and invalid turns.

  • B.1. Rejective Fine-tuning: Rejective fine-tuning retains only high-scoring multi-turn trajectories for one-epoch supervised fine-tuning, producing a reliable initial policy that reduces malformed rollouts and improves subsequent training stability.RFT also strengthens instruction following, output formatting, and action-output success, although excessive RFT can degrade performance.
  • B.2. Format Penalty: A format-constrained projection validates exactly structured <think> and <action> outputs, extracts executable actions when possible, and penalizes strict-format failures during RL.Relaxed parsing recovers the first action field but marks the output format-invalid; otherwise a fallback placeholder is stored.
  • B.2. Format Penalty: The format penalty supplies a direct signal against malformed thinking–action generations without requiring additional external supervision.The penalty is applied when the strict validator fails.
  • B.3. Trajectory Decomposition: Trajectory decomposition enables scalable, fine-grained multi-turn optimization and dynamic interventions, but pipelined rollouts create off-policy staleness that grows with rollout batch size and interaction horizon.Larger update micro-batches and greater prompt-group parallelism reduce the staleness ratio, whereas staleness can increase importance-weight variance and destabilize optimization.
  • B.4. Policy Update Details: GiGPO computes turn-relative advantage by grouping occurrences of identical states, discounting returns, and normalizing them so advantages compare actions taken under the same state.Positive values indicate above-average decisions, while negative values indicate sub-optimal choices under identical state conditions.
  • C. More Related Work: The related-work discussion covers lengthy reward, short-CoT cold start, hard thinking budgets, and void-turn filtering as established approaches to reasoning control.These methods address reasoning length, structured search initialization, deliberation-token limits, and invalid multi-turn outputs, respectively.
  • C.1. Lengthy Reward: Lengthy reward encourages shorter correct responses, penalizes longer correct responses and long incorrect responses, and adds this control to task reward through a weighting coefficient.When all sampled responses have identical lengths, the length reward is zero.
  • C.2. Short-CoT Cold Start / C.3. Hard Thinking Budget / C.4. Void-Turn Filtering: Short-CoT cold start uses concise reasoning demonstrations to initialize instruction following, valid formatting, backtracking-related search behavior, and a reduced effective action space, while hard thinking budgets cap internal deliberation tokens.Void-turn filtering removes responses lacking a valid final answer or complete executable structure from policy-loss computation; Figure 6 compares output lengths and truncation proportions for GiGPO and T2PO.

D.1. Ablation Study on other Policy Optimization Algorithm

T$^2$PO is plug-and-play with alternative policy update schemes: replacing the base optimization with GSPO on WebShop, TTI+TDS raises success rate from 85.18 to 91.79, a 7.76% relative improvement.

  • Ablation Study on other Policy Optimization Algorithm: 91.79 success rate on WebShop after applying TTI+TDS with GSPO, up from 85.18, corresponding to a relative improvement of 7.76%.The experiment replaces the base policy optimization with GSPO, demonstrating compatibility with another policy update scheme.

D.2. Ablation Study on Token-level Response Length

The ablation shows that longer token budgets are not universally beneficial in multi-turn RL: T2PO maintains efficiency with a 500-token limit while supporting progressively deeper interaction-driven reasoning.

  • Response-length effects: Increasing the maximum response length from 500 to 700 leaves the final model’s average output length nearly unchanged, while a 300-token limit clips many trajectories.These observations indicate that excessive token budgets are often unnecessary, but overly small limits constrain responses.
  • Token efficiency: 20% fewer tokens: Under the final 500-token setting, T2PO produces fewer average tokens than GiGPO.This establishes higher token efficiency for T2PO in the comparison shown in Figure 6.
  • Token efficiency: In the last 50 training steps, T2PO rarely triggers maximum-length clipping, indicating reduced redundant or uninformative text and substantially less over-thinking.The result accompanies T2PO’s higher token efficiency under the 500-token limit.
  • Interaction-driven reasoning: During the first 20 training steps, T2PO’s output length gradually increases, whereas GiGPO’s sharply decreases during the first 50 steps.The contrasting dynamics suggest progressively enhanced reasoning depth for T2PO and rapidly suppressed exploration for GiGPO.

D.3. Sensitivity Analysis on α · D.4. Efficiency Analysis on Alfworld

The fusion coefficient α is most effective at α = 0.4, which the paper adopts. On ALFWorld, T2PO improves exploration efficiency while using fewer tokens and interaction turns on successful trajectories than SOTA baselines.

  • D.3. Sensitivity Analysis on α: D.3: The sensitivity analysis varies α across 0.2, 0.4, 0.6, and 0.8 in the self-calibrated uncertainty signal.The signal balances entropy and confidence.
  • D.3. Sensitivity Analysis on α: D.3: α = 0.4 yields the best performance and is therefore selected.The coefficient fuses entropy and confidence in the uncertainty signal.
  • D.4. Efficiency Analysis on Alfworld: D.4: Successful T2PO trajectories on ALFWorld consume substantially fewer tokens than those generated by SOTA baselines.The token-consumption distribution is shown in the bar chart.
  • D.4. Efficiency Analysis on Alfworld: D.4: T2PO maintains consistently higher exploration efficiency than baselines on successful ALFWorld trajectories throughout training.The line plot tracks exploration efficiency during training.
  • D.4. Efficiency Analysis on Alfworld: D.4: T2PO completes ALFWorld tasks with approximately 16% fewer interaction turns at the turn level.This reduction is reported for the right-figure comparison during training.
  • D.4. Efficiency Analysis on Alfworld: D.4: The ALFWorld efficiency analysis jointly reports lower token consumption, higher exploration efficiency, and fewer interaction turns for T2PO successful trajectories.These findings come from the bar chart, line plot, and turn-level figure, respectively.

D.5. More Results on WebShop … A Failure Case Caused by Over-Thinking

The paper attributes WebShop gains to reinforcement learning and analyzes exploration failures at both token and turn levels. It also provides scalable implementation details, formalizes hierarchical control algorithms, and documents over-thinking failures that motivate intervention and resampling.

  • D.5. More Results on WebShop: WebShop decomposes reward into Title Score, reward type, reward attribute, and reward option, each measuring a distinct task-completion aspect.These components respectively assess product-title identification, category matching, attribute fulfillment, and final option selection.
  • D.5. More Results on WebShop: Prompting and instruction-tuned baselines perform poorly on fine-grained rewards, while PPO and GRPO improve all dimensions; GiGPO variants remain imbalanced.The passage attributes these patterns to insufficient supervised or in-context alignment and the benefits of reinforcement learning for long-horizon behavior.
  • E. Codebase: The implementation modifies vLLM and the agent interaction loop to support fine-grained uncertainty-aware decoding and rollout control within verl’s step-wise multi-turn training.The redesign preserves verl’s scalability, modularity, and compatibility with its training paradigm.
  • E. Codebase: The framework is framework-agnostic and extends to asynchronous reinforcement learning, enabling nonblocking rollout collection and parameter updates under distributed settings.It can be deployed on verl or other large-scale asynchronous RL infrastructures with minimal engineering overhead.
  • F. Algorithm Pseudo Code: Algorithm 1 terminates decoding after sustained stabilization of the self-calibrated uncertainty signal Mt by overwriting logits to emit a reasoning terminator token.This is the token-level thinking intervention mechanism.
  • F. Algorithm Pseudo Code: Algorithm 2 computes turn-level uncertainty aggregation and regenerates a turn when consecutive-turn variation Γk falls below a tolerance threshold.Regeneration occurs under the same environment state until the turn becomes sufficiently informative or the procedure’s limit is reached.
  • G. Case Study: In a vanilla GiGPO WebShop case, contradictory product constraints trigger verbose checking and speculation, causing reasoning to exceed the output budget before </think> and <action> are produced.The case shows that unconstrained reasoning, rather than the compact action space, becomes the dominant failure source.
  • A Failure Case Caused by Over-Thinking: Overthinking caused truncation through missing </think> and <action> tags, while repeated search loops reused unsuccessful actions without meaningful progress or adaptation.Together, these failures motivate explicit reasoning-length control, early exits, and improved exploration in multi-turn interaction.

A Failure Case with Repeated Failure Turns · A Successful Case with Valid Action from Our T2PO

The failure case shows the agent falling into repeated, redundant reasoning across interaction turns. In contrast, T2PO produces a valid, parsable action that is concise, non-redundant, and directed toward improving a mismatched shopping state.

  • A Failure Case with Repeated Failure Turns: After selecting a Nautica t-shirt priced at $16.05, the agent continues reconsidering whether the product satisfies the requested attributes.The response notes that the color matches but that the required 3x-large size is not specified.
  • A Failure Case with Repeated Failure Turns: The failure case repeats the same multi-constraint search reasoning across interaction turns instead of progressing the WebShop task.The reasoning repeatedly restates requirements for heather grey, classic fit, 3x large, and under $30, including the same proposed search query.
  • A Successful Case with Valid Action from Our T2PO: T2PO instead generates a valid, executable action from a structured state containing task constraints, recent memory, and current observations.The structured representation includes fine-grained attributes such as material, fit, color, size, and price.
  • A Successful Case with Valid Action from Our T2PO: The successful action is non-redundant and uses the memory context to select a single navigation operation that improves a mismatched result page.The example selects click[< prev] as an outcome-oriented control signal rather than repeating state tokens.
  • A Successful Case with Valid Action from Our T2PO: In another WebShop state, the agent recognizes that page-2 results do not match the requested shirt criteria and reasons about the next admissible action.The requirements include washability, materials, sleeve style, classic fit, brown color, women’s fit, x-large size, and a price below $40.
  • A Successful Case with Valid Action from Our T2PO: The generated response is successfully parsed as an action in the T2PO case.The figure explicitly reports successful action parsing after the reasoning and action-format output.
Loading 2605.02178v1…