PaperlayerView exact revision on arXiv

Source-linked AI summary

A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

Dingwei Chen, Zefang Zong, Zhipeng Ma, Leo Luo, Yang Li, Chengming Li, Peng Chen, Jie Jiang

arXiv:2605.06200v1cs.CL

TL;DR

Agentic reinforcement learning lacks fine-grained credit assignment for individual tool-calls, while existing Information Gain approaches face normalization, accumulation, and clipping challenges. A2TGPO redesigns these three components and consistently outperforms prior baselines across seven QA benchmarks and three backbones.

  • Problem

    Trajectory-level rewards do not distinguish tool-calls that advance toward an answer from those that merely prolong interaction, while Information Gain methods face turn-context, advantage-scale, and clipping challenges.

  • Method

    A2TGPO normalizes Information Gain within prompt–turn groups, variance-rescales discounted accumulation, and adapts turn-level clipping to normalized informativeness.

  • Results

    +1.75 on multi-hop and +1.69 on single-hop QA on average, with A2TGPO consistently outperforming prior strong baselines across seven benchmarks and three backbones.

  • Takeaways & Limitations

    A2TGPO provides fine-grained process credit assignment without an external evaluator or additional rollout for the evaluated QA settings.

  • Takeaways & Limitations

    A2TGPO relies on ground-truth answers for Information Gain, limiting direct applicability to tasks without verifiable outcomes.

Abstract

from arXiv · show

Reinforcement learning for agentic large language models (LLMs) typically relies on a sparse, trajectory-level outcome reward, making it difficult to evaluate the contribution of individual tool-calls within multi-turn interactions. Existing approaches to such process credit assignment either depend on separate external process reward models that introduce additional consumption, or tree-based structural rollout that merely redistributes the outcome signal while constraining trajectory diversity. A promising alternative leverages the per-turn change in the policy's predicted probability of the ground-truth, termed Information Gain (IG), as an intrinsic process signal without an external evaluator. However, prior work on leveraging IG signals within the RL training loop faces three systematic challenges: normalizing across turns that face heterogeneous positional contexts can distort the relative standing of individual turns, accumulating a variable number of terms causes advantage magnitudes to drift with trajectory depth, and a fixed clipping range governs policy updates identically for turns with vastly different IG signals. In this paper, we propose A$^2$TGPO (Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping), which retains IG as the intrinsic signal but re-designs how it is normalized, accumulated, and consumed: (i) turn-group normalization: normalizes IG within each (prompt, turn-index) group so that each turn is compared only against peers at the same interaction depth; (ii) variance-rescaled discounted accumulation: divides cumulative normalized IG by square root of accumulated terms to keep advantage magnitudes comparable across turn positions; and (iii) adaptive turn-level clipping: modulates each turn's clipping range based on its normalized IG, widening the update region for informative turns and narrowing it for uninformative ones.

1 Introduction

A2TGPO addresses the lack of per-turn credit assignment in agentic RL by retaining Information Gain as an intrinsic signal and redesigning its normalization, accumulation, and policy consumption. It introduces turn-group normalization, variance-rescaled discounted accumulation, and adaptive turn-level clipping, achieving average improvements of +1.75 on multi-hop and +1.69 on single-hop QA over existing RL methods.

  • Motivation: Existing agentic RL methods use a single trajectory-level outcome, providing no mechanism for evaluating individual tool-calls in multi-turn interactions.Process reward models add separately trained evaluators and reward-hacking risks, while tree-based methods redistribute outcome signals and constrain trajectory diversity.
  • Challenges: Prior IG-based process supervision pools heterogeneous turn positions, causing incomparable gains, depth-dependent advantage magnitudes, and uniform clipping for turns with different signals.These challenges distort turn rankings, make cumulative advantages drift with trajectory depth, and prevent policy updates from adapting to turn informativeness.
  • A2TGPO: Turn-group normalization evaluates each turn against peers within the same (prompt, turn-index) group, avoiding comparisons across different interaction depths.The method treats turn-index as a natural unit for normalization and credit assignment.
  • A2TGPO: Adaptive turn-level clipping modulates each turn’s clip range according to normalized IG, widening updates for informative turns and narrowing them for uninformative ones.Together with the rescaled accumulation, this allows policy updates to respond to turn-level signal strength.
  • A2TGPO: Variance-rescaled discounted accumulation divides cumulative normalized IG by the square root of accumulated terms, keeping advantage magnitudes comparable across turn positions.This addresses scale drift caused by summing variable numbers of normalized IG terms.
  • Results: +1.75 on multi-hop and +1.69 on single-hop QA are A2TGPO’s average improvements over existing RL methods across seven benchmarks and three backbones.The evaluation covers single-hop and multi-hop open-domain question answering.

2 Related Work

Related work builds from PPO-based RLHF toward critic-free, group-relative optimization and increasingly agent-specific methods. Agentic RL research also addresses sparse trajectory-level rewards through process reward models or tree-structured credit redistribution.

  • Reinforcement Learning in LLMs and Agents: PPO-based RLHF established a foundation for LLM reasoning and alignment, followed by critic-free methods including GRPO and DAPO.These methods estimate advantages from group-relative comparisons and refine clipping granularity with verifiable rewards.
  • Reinforcement Learning in LLMs and Agents: A growing line of work tailors reinforcement-learning optimization to the agentic paradigm.The passage identifies this as an extension of the PPO-based and critic-free RL foundation.
  • Credit Assignment in Agentic Reinforcement Learning: Outcome-driven agentic RL provides sparse trajectory-level rewards, making credit assignment across long multi-turn interactions difficult.This motivates methods that supply denser process-level credit.
  • Credit Assignment in Agentic Reinforcement Learning: Process reward models score process steps to provide dense rewards, but require a separately loaded reward model.The passage presents PRMs as one route for addressing coarse outcome-driven rewards.
  • Credit Assignment in Agentic Reinforcement Learning: Tree-structured rollouts offer another route by organizing trajectories and redistributing process credit.The supplied passage introduces this approach but does not provide further details.

3 Preliminaries

The preliminaries define agentic RL as multi-turn policy interaction with a tool environment receiving a trajectory-level outcome reward, then introduce turn-level Information Gain for finer-grained credit assignment. They describe how IG-based turn advantages replace trajectory-level advantages while standard clipping remains fixed across turns.

  • Task Definition: A policy πθ answers queries through multi-turn interaction with tool environment E, producing rollouts τ and trajectory-level reward R(τ) for final-answer correctness.The dataset consists of query–ground-truth pairs (q, a⋆), and the rollout concludes with prediction ˆa.
  • Multi-turn Rollout: At each turn, the policy generates a segment yt, the environment may return observation ot, and only model-generated tokens contribute to the policy gradient.Environment-produced observations are masked out during loss calculation.
  • Turn-level Information Gain: Turn-level Information Gain igi,t measures the change in the policy’s length-normalized probability assigned to the ground-truth answer after turn t.IG is computed from the policy’s own likelihoods with stop-gradient treatment; the first-turn gain uses the query-only baseline.
  • Policy Update with Turn-level Advantages: IGPO-style updates assign igi,t to nonfinal turns and R(τi) to the final turn, then jointly z-normalize rewards and propagate them through discounted accumulation.The discount factor satisfies γ ∈ (0, 1].
  • Policy Update with Turn-level Advantages: The resulting turn-level advantage replaces the trajectory-level advantage, providing finer-grained process credit than GRPO while retaining a fixed clipping range ε across turns and samples.The fixed clipping range applies identically across all turns and all samples in this formulation.

4 Methodology

A2TGPO assigns intrinsic information-gain credit at the turn level through prompt-and-turn-group normalization, variance-rescaled discounted accumulation, and adaptive clipping. It combines this process signal with normalized outcome rewards in a turn-level policy objective.

  • Turn-group normalization: A2TGPO normalizes each per-turn information gain within peers sharing the same prompt and turn index, preserving position-conditional comparisons.Rollouts ending before turn t are excluded; groups with at most one member receive zero normalized information gain and rely on outcome reward.
  • Variance-rescaled accumulation: Discounted backward accumulation propagates normalized information-gain credit from downstream turns toward earlier turns, with γ down-weighting distant contributions.The approach addresses the baseline’s inconsistent advantage magnitudes caused by summing different numbers of terms across trajectory depths.
  • Variance-rescaled accumulation: Dividing cumulative credit by √n_i,t approximately stabilizes variance across turn positions, keeping advantage magnitudes comparable across trajectory depths.This rescaling follows from the variance of the sum growing linearly in n_i,t under mild independence assumptions.
  • Turn-level advantage: The turn-level advantage combines rescaled backward process credit with per-prompt normalized outcome reward, while the final answer turn uses only the outcome signal.Process turns receive both components; the final answer turn has no defined information gain.
  • Adaptive turn-level clipping: Adaptive clipping widens policy-update ranges for turns with higher normalized information gain and narrows them for low or negative gain.A sigmoid scale c_i,t monotonically increases with normalized gain and remains bounded within (1 − β, 1 + β); turn-level ratios and advantages are shared across tokens.

5 Experiments

A2TGPO is evaluated across three backbones and seven open-domain QA benchmarks, achieving the highest sample-weighted average in every setting and larger gains on multi-hop tasks. Ablations and training analyses attribute these results to turn-group normalization, variance-rescaled discounted accumulation, adaptive clipping, and balanced exploration dynamics.

  • Experimental Setup: Evaluation covers three backbones and seven open-domain QA benchmarks, split into multi-hop and single-hop reasoning groups, using Exact Match as the primary metric.Backbones are Qwen3-4B, Qwen3-8B, and Qwen2.5-7B; multi-hop tasks include HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle, while single-hop tasks include NQ, TriviaQA, and PopQA.
  • Main Results: A2TGPO achieves the highest sample-weighted average on all benchmark settings, improving over existing RL methods by +1.75 on multi-hop and +1.69 on single-hop.The gains are consistently larger on multi-hop benchmarks, where longer tool-use trajectories amplify the benefit of per-turn credit assignment.
  • Main Results: A2TGPO matches or surpasses the strongest single-hop baseline, DAPO, without its higher rollout consumption, while GiGPO’s gains remain marginal in generative settings.DAPO benefits from a higher clip and dynamic sampling that triples the per-step rollout budget; GiGPO relies on identifying repeated states across rollouts.
  • Ablation Study: Additive ablations show that turn-group normalization resolves positional incomparability, variance-rescaled discounted accumulation propagates credit, and adaptive clipping completes the proposed component design.The study starts from IGPO on Qwen3-4B multi-hop benchmarks and compares against GRPO as a non-IG reference.
  • Training Dynamics Analysis: A2TGPO maintains a balanced entropy plateau, preserving stronger exploration than classic RLVR methods while remaining more stable than AEPO.Classic RLVR methods undergo rapid entropy collapse, whereas AEPO’s entropy climbs steadily during training.
  • Training Advantage Distribution Analysis: Variance rescaling compresses A2TGPO’s cumulative-advantage distribution into a narrow, consistent band across turn positions, unlike IGPO’s widely spread boxes and large outliers.Turn-group normalization also centers each position independently, eliminating location bias before advantage aggregation.

6 Conclusion … B.7 Hardware and Artifacts

A2TGPO reformulates information gain as an intrinsic per-turn signal through turn-group normalization, variance-rescaled discounted accumulation, and adaptive clipping, enabling fine-grained credit assignment without external evaluators or additional rollouts. The paper specifies its workflow, reward and prompt schema, QA benchmarks, training settings, baselines, search environment, and hardware artifacts.

  • 6 Conclusion: A2TGPO groups IG by prompt and turn index, rescales discounted advantages by accumulated-term variance, and adapts clipping to per-turn informativeness.The method is presented as stable, fine-grained process credit assignment without an external evaluator or additional rollout.
  • A Algorithm Workflow: Each training iteration generates multi-turn tool-use rollouts, computes IG from forward logits, constructs normalized advantages, and updates the policy with turn-level clipping.The workflow includes turn-group normalization, variance-rescaled discounted accumulation, and IG-based adaptive clipping.
  • B.1 Reward Design: The binary outcome reward combines exact-match correctness with a required response format containing <think>...</think>, <answer>...</answer>, and a boxed final answer.A trajectory receives maximal reward only when it satisfies the schema and exactly matches the reference answer.
  • B.2 Prompt Template: The prompt schema partitions search-augmented reasoning into tagged regions and extracts the canonical prediction from the boxed span inside the final answer.The format requirement supports reliable tool-call parsing and final-answer extraction while penalizing schema violations.
  • B.3 Datasets: Experiments evaluate search-augmented agentic reasoning on multi-hop benchmarks HotpotQA and 2WikiMultiHopQA and single-hop benchmarks NQ, TriviaQA, and PopQA.The multi-hop category tests multi-turn tool use and compositional reasoning, while the single-hop category tests single-step retrieval.
  • B.4 A2TGPO Settings: Training uses batch size 64, mini-batch size 8, response length 6192, rollout size 16, maximum tool usage 6, clipping thresholds 3e-3 and 4e-3, and γ = 1.0.The √n_t rescaling is stated to maintain consistent advantage magnitudes across turn depth.
  • B.5 Baseline Settings: Baselines include ReAct, GRPO, DAPO, GSPO, AEPO, Tree-GRPO, GIGPO, and IGPO, spanning prompting, policy optimization, sequence-level clipping, entropy-aware scheduling, tree search, and IG-based credit assignment.The baseline experiments use their respective RL recipes without an additional SFT phase.
  • B.6 Search Tool Environment: The system uses a Wikipedia snapshot with e5-base-v2 retrieval over approximately 21M entries, and runs on one node with 8× NVIDIA H20 GPUs using Qwen3-4B, Qwen3-8B, Qwen2.5-7B, and VeRL.The search configuration mirrors Search-R1, while the backbones are publicly released checkpoints selected for reasoning ability and agentic post-training compatibility.

B.8 Limitations and Future Work · C Extended Experimental Analysis · C.1 Computational Overhead

A2TGPO’s ground-truth-dependent Information Gain limits direct applicability to tasks without verifiable outcomes, while its added computation produces only modest training overhead. Future work targets broader intrinsic signals and more efficient IG computation for longer-horizon agentic tasks.

  • B.8 Limitations and Future Work: Ground-truth-dependent Information Gain limits A2TGPO’s direct applicability to open-ended creative generation and subjective evaluation tasks.The paper suggests self-consistency across rollouts and uncertainty reduction in internal representations as alternative intrinsic signals.
  • C.1 Computational Overhead: +2.9% net overhead: A2TGPO’s additional IG forward pass cost is largely offset by faster generation under rollout budget n = 16.The IG forward pass is the sole additional component, following an efficient single-pass implementation.
  • C.1 Computational Overhead: +164 s from the IG forward pass is largely offset by −86 s from faster generation, resulting in +15 s net overhead.
  • B.8 Limitations and Future Work: A2TGPO produces a tighter response-length distribution, with a higher minimum, comparable mean, and substantially lower maximum over 240 training steps.
  • B.8 Limitations and Future Work: More efficient IG computation, including amortized estimation or cached incremental updates, could reduce cost and support agentic tasks with dozens of interaction turns.
  • C.1 Computational Overhead: ∼525 s/step versus ∼511 s/step: A2TGPO averages this per-step wall-clock time against GRPO under matched rollout budget n = 16.

C.2 Ablation Study on Single-hop Benchmarks · C.3 Sensitivity to Adaptive Clipping Coefficient β

On single-hop benchmarks, all three A2TGPO components remain effective, with adaptive turn-level clipping providing the largest gain and variance-rescaled accumulation the smallest. Sensitivity analysis shows performance peaks at β=0.3 and remains stable across β ∈[0.2, 0.4].

  • C.2 Ablation Study on Single-hop Benchmarks: IG-adaptive turn-level clipping contributes the largest gain, approximately 50%, on single-hop benchmarks.Single-hop tasks involve fewer turns, making per-turn update-intensity modulation especially important.
  • C.2 Ablation Study on Single-hop Benchmarks: Turn-group normalization provides a moderate gain of approximately 35% on single-hop benchmarks.
  • C.2 Ablation Study on Single-hop Benchmarks: The ablation study evaluates A2TGPO components on single-hop benchmarks using Qwen3-4B.
  • C.2 Ablation Study on Single-hop Benchmarks: Variance-rescaled accumulation contributes approximately 15%, less than the other components.Single-hop trajectories typically contain 1 to 2 process turns, reducing the importance of cross-depth scale correction.
  • C.3 Sensitivity to Adaptive Clipping Coefficient β: The adaptive clipping coefficient β controls the clipping range ci,t ∈[1−β, 1+β], with β=0 corresponding to fixed clipping.The sensitivity sweep uses β ∈{0, 0.1, . . . , 0.5}.
  • C.3 Sensitivity to Adaptive Clipping Coefficient β: Performance improves monotonically up to β=0.3, reaching multi-hop 48.06 and single-hop 56.44.Performance degrades only mildly beyond β=0.3 and remains within 0.15 points of the optimum across β ∈[0.2, 0.4].
  • C.3 Sensitivity to Adaptive Clipping Coefficient β: At β=0, A2TGPO already outperforms GRPO and IGPO, supporting the standalone effectiveness of turn-group normalization and variance rescaling.

C.4 Turn Distribution Analysis

Tool-call distributions differ by task type: multi-hop rollouts use more calls on average and typically require 2–4 calls, while single-hop rollouts concentrate at 1–2 calls. Both settings also include zero-call rollouts.

  • Tool-call distributions: Multi-hop tasks average 3.45 tool calls per rollout, with 72% using 2–4 calls.The distribution reflects the need for multiple retrieval steps in multi-hop reasoning chains.
  • Tool-call distributions: Single-hop tasks average 2.05 tool calls per rollout, with 65% using 1–2 calls.Single-hop rollouts therefore concentrate at fewer tool calls than multi-hop rollouts.
  • Tool-call distributions: Zero-call rollouts account for 2.5% of multi-hop and 4.5% of single-hop rollouts.Both task settings exhibit a non-trivial proportion of rollouts without tool calls.

C.5 Context Similarity Analysis · D Theoretical Analysis

The analysis finds that same-position turns share comparable contexts, supporting turn-group normalization, and reports formal results addressing its bias, robustness, and variance-rescaling properties.

  • C.5 Context Similarity Analysis: At initial policy step 1, same-position rollouts show moderate per-turn Jaccard similarity of approximately 0.6–0.7, decreasing with interaction depth.Similarity is computed over thinking, query, and tool-response content across rollouts of the same prompt.
  • C.5 Context Similarity Analysis: Figure 10 reports the distribution of tool-call counts per rollout for multi-hop and single-hop benchmarks.The figure concerns rollout-length variation relevant to agentic tool-use trajectories.
  • C.5 Context Similarity Analysis: After training at step 250, intra-position similarity reaches 0.86 at turn 1 and 0.67 at turn 2.The result indicates more consistent tool calls at each position under the converged policy.
  • C.5 Context Similarity Analysis: Overall intra-position similarity is 0.62 versus 0.38 across positions, a 63% higher value supporting comparisons within turn groups.The comparison is presented as evidence that turn-group normalization evaluates turns under comparable contexts.
  • D Theoretical Analysis: Section D.1 shows that IGPO’s pooled normalization introduces systematic position-dependent bias, while turn-group normalization provably eliminates it.This formal result directly justifies normalizing within prompt-and-turn-index groups.
  • D Theoretical Analysis: Section D.2 establishes turn-group normalization’s unbiasedness and robustness, including behavior under small groups caused by variable rollout lengths.The analysis explicitly addresses group-size variation arising from trajectories with different lengths.
  • D Theoretical Analysis: Section D.3 analyzes dividing cumulative Information Gain by the square root of accumulated terms as the paper’s variance-rescaling mechanism.This is the theoretical component corresponding to variance-rescaled discounted accumulation.

D.1 Positional Bias under Pooled Normalization

Pooled IG normalization creates systematic positional bias when turn-level IG distributions differ across interaction depths. Because early positions dominate the pooled statistics, advantages can reflect positional characteristics rather than individual tool-call quality.

  • Heterogeneous per-position IG distributions: Turn positions have heterogeneous IG distributions because their contextual states differ, with means that can vary across positions.Rollouts also terminate at different turns, so deeper positions are reached by fewer rollouts.
  • Positional bias: Weighting by G_t makes early positions dominate the pooled mean, biasing normalization against sparse, deeper positions.The effect is stronger where fewer rollouts reach a position and its mean IG deviates from the grand mean.
  • Pooled normalization: IGPO pools information gains across all rollout positions before applying group-wise z-normalization.The pooled statistics are computed over every valid rollout-position pair.
  • Positional bias: Pooled-normalized signals have non-zero expected values whenever a position’s mean IG differs from the weighted grand mean.This establishes systematic positional bias independent of the individual tool-call’s quality.
  • Consequences: Pooled normalization inflates advantages when µ_t exceeds the grand mean and deflates them when µ_t falls below it, conflating position with action quality.This positional artifact distorts the advantage signal used by the optimizer.

D.2 Unbiasedness and Robustness of Turn-Group Normalization · D.3 Variance Homogeneity under Square Root Rescaling · D.4 Gradient Modulation under Adaptive Clipping

The appendix establishes that turn-group normalization removes positional bias while remaining unbiased for groups of at least two, square-root rescaling equalizes accumulated-signal variance across depths, and adaptive clipping allocates wider update regions to informative turns.

  • D.2 Unbiasedness and Robustness of Turn-Group Normalization: Turn-group normalization compares IG signals within each (prompt, turn-index) group rather than against a pooled distribution.This removes positional bias and places positions on a common unit-variance scale.
  • D.2 Unbiasedness and Robustness of Turn-Group Normalization: For any group size G_t ≥ 2, turn-group normalization yields an unbiased, zero-mean signal without systematic directional error.Smaller groups increase estimator variance but do not introduce positional bias; G_t = 1 provides no meaningful peer comparison.
  • D.2 Unbiasedness and Robustness of Turn-Group Normalization: Small G_t often marks deeper turns whose few surviving rollouts differ strongly from early-terminating rollouts, producing pronounced outcome-reward divergence.The outcome term in the advantage is estimated from all G rollouts and is independent of G_t.
  • D.3 Variance Homogeneity under Square Root Rescaling: Dividing the backward cumulative IG by √n_i,t corrects the larger unscaled advantage magnitudes produced by longer shallow-turn horizons.Without rescaling, cumulative terms increase with the variable count n_i,t = T_i − t.
  • D.3 Variance Homogeneity under Square Root Rescaling: When residual correlation is small, √n_i,t rescaling makes advantage variance approximately σ^2 across turn depths, independent of the number of accumulated terms.This prevents shallow turns from dominating updates solely because they aggregate longer horizons.
  • D.4 Gradient Modulation under Adaptive Clipping: The adaptive clip scale c_i,t increases monotonically with normalized IG and remains bounded in (1 − β, 1 + β).The sigmoid mapping converts each turn’s normalized IG into a per-turn effective clipping width.
  • D.4 Gradient Modulation under Adaptive Clipping: High-IG turns approach c_i,t → 1 + β and receive wider trust regions, whereas low- or negative-IG turns approach c_i,t → 1 − β and receive narrower ones.This allocates larger step budgets to informative process signals and suppresses updates from turns providing little or adverse evidence.

E Case Study

Tables 5–11 provide qualitative validation examples from all seven benchmarks, visualizing A2TGPO’s reasoning, searches, retrieved passages, and final answers. The examples show multi-hop behavior decomposing questions into sequential sub-goals and launching multiple searches, illustrated by chained evidence in the Space Needle example.

  • Qualitative Examples: Tables 5–11 visualize complete A2TGPO validation trajectories across all seven evaluation benchmarks.Color annotations distinguish model reasoning, search queries, retrieved passages, and final answers.
  • Multi-hop Behavior: On multi-hop benchmarks, A2TGPO decomposes questions into sequential sub-goals and launches multiple searches.
  • Qualitative Examples: In the Star Trek V example, the agent identifies William Shatner as director, searches his nationality, and concludes he is Canadian.The trajectory links the film’s director to search evidence stating that Shatner was born in Montréal and is Canadian.
  • Multi-hop Behavior: In the Space Needle example, the agent searches for Washington’s capital and combines the results to answer that it is Olympia.The trajectory first identifies Seattle, Washington, then retrieves evidence about Olympia as the state capital.
Loading 2605.06200v1…