A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

• Agentic RL’s trajectory-level outcome reward does not distinguish tool-calls that advance toward the answer from those that merely prolong interaction.Prior routes include external process reward models [15] [16], tree-based redistribution [17] [18] [19], and intrinsic signals such as GiGPO [11] and IGPO [20]; IGPO normalizes Information Gain across all turns and uses discounted accumulation.Source: §1 · p. 2
• Prior Information Gain methods pool turns across heterogeneous contexts, let accumulated advantage magnitudes drift with trajectory depth, and apply fixed clipping regardless of turn informativeness.Source: §1 · p. 2
• Same-position rollouts have greater contextual similarity than cross-position rollouts: similarity declines from 0.86 at turn 1 to 0.42 at turn 5, while overall intra-position versus cross-position similarity is 0.62 versus 0.38.The paper uses turn index as a comparison group because trajectories at the same interaction depth tend to occupy similar contexts and states, especially early in rollouts.Source: §1 · p. 2, Figure 1 · p. 2
• A2TGPO normalizes Information Gain within each prompt–turn-index group, variance-rescales discounted accumulation by the square root of accumulated terms, and widens or narrows turn-level clipping according to normalized Information Gain.It applies the importance-sampling ratio and clipping at the turn level rather than the token or sequence level.Source: §1 · p. 3
• A2TGPO consistently outperforms prior strong baselines across seven single-hop and multi-hop open-domain QA benchmarks and three backbones, improving over existing RL methods by +1.75 on multi-hop and +1.69 on single-hop on average.Source: §1 · p. 3

Reinforcement Learning in LLMs and Agents

• Search-R1, ARPO, and AEPO extend RL to tool use and entropy-guided rollout, but these outcome-driven approaches still rely on trajectory-level rewards.Critic-free methods such as GRPO and DAPO estimate advantages from group-relative comparisons and refine clipping granularity.Source: §2 · p. 3

Credit Assignment in Agentic Reinforcement Learning

• PRMs provide dense process rewards but require a separately loaded model, while tree-based rollout methods redistribute outcome rewards and constrain trajectory diversity.Entropy-guided branch expansion further improves exploration dynamics within the tree-based paradigm.Source: §2 · p. 3
• GiGPO and IGPO introduce intrinsic credit signals, but lack an objective comparison for estimating the relative importance of individual tool calls across turns.GiGPO groups same-state actions across trajectories; IGPO uses per-turn information gain to estimate tool-call advantages.Source: §2 · p. 3

Task Definition

• Agentic RL trains policy πθ to answer queries through tool-environment rollouts, using trajectory-level reward R(τ) for prediction correctness [4] [9] [12] [20].The objective is defined over a dataset of query–ground-truth answer pairs.Source: §3 · p. 3

Multi-turn Rollout

• Following ReAct [1], each turn produces a model-generated segment and, for tool calls, an environment observation; only generated tokens contribute to the policy gradient.Environment-produced observations are masked out of the loss calculation.Source: §3 · p. 4

Turn-level Information Gain

• IG measures the change in the policy’s probability of the ground-truth answer after a turn versus its preceding prefix, with the first turn compared against the query-only baseline [20].The signal uses the policy’s own likelihoods, is stop-gradient, and requires one additional forward pass per turn.Source: §3 · p. 4

Policy Update with Turn-level Advantages

• The update assigns IG to nonfinal turns and trajectory reward to the final turn, jointly z-normalizes turn rewards, then uses discounted accumulation as turn-level advantages.These advantages replace trajectory-level advantages in the clipped policy objective, whose clipping range remains fixed across turns and samples.Source: §3 · p. 4

4.1 IG-based Turn-Group Normalization

• A2TGPO z-normalizes information gain among rollouts sharing a prompt and turn index, setting the signal to zero when at most one rollout reaches that turn.This compares turns at the same interaction depth, where contexts are more similar, rather than pooling early- and late-turn signals with different distributions.Source: §4.1 · pp. 4–5

4.2 Discounted Cumulative Advantage with Variance Rescaling

• A2TGPO accumulates discounted normalized gains backward from downstream turns and divides by the square root of the accumulated-term count to keep advantage variance comparable across positions.The discount factor γ ∈ (0, 1] down-weights distant turns; the rescaled process credit is combined with the normalized outcome reward, while the final-answer turn receives only the outcome signal.Source: §4.2 · pp. 5–6

4.3 IG-based Adaptive Turn-level Clipping

• A2TGPO uses a length-normalized geometric mean of token ratios for each turn and widens or narrows shared turn-level clipping bounds according to normalized information gain.A sigmoid scale bounded by (1 − β, 1 + β) adjusts asymmetric base bounds; because information gain is stop-gradient, the scale contributes no gradient with respect to policy parameters.Source: §4.3 · p. 6

5.1 Experiment Settings

• The reproducible evaluation uses three backbones, seven open-domain QA benchmarks, and Exact Match as the primary metric.Benchmarks cover four multi-hop and three single-hop datasets; the setup avoids proprietary APIs and heavyweight tool infrastructure.Source: §5.1 · pp. 6–7

5.2 Main Results of A2TGPO

• A2TGPO achieves the highest sample-weighted average across all settings, improving over existing RL methods by +1.75 on multi-hop and +1.69 on single-hop benchmarks.Gains are consistently larger on multi-hop benchmarks, where longer tool-use trajectories amplify per-turn credit assignment. IGPO’s pooled normalization conflates positional contexts, while A2TGPO refines how IG is normalized, accumulated, and consumed.Source: Table 1 · p. 7, §5.2 · p. 8

5.3 Analysis

• The additive Qwen3-4B multi-hop ablation finds distinct gains from turn-group normalization, variance-rescaled discounted accumulation, and IG-adaptive turn-level clipping.Turn-group normalization already surpasses GRPO; adaptive clipping contributes most on the longer-horizon MuSiQue and Bamboogle benchmarks.Source: Table 2 · p. 7, §5.3 · p. 8

5.1 Experiment Settings

• Comparisons include non-RL ReAct [1], RLVR methods, and agentic RL baselines; Tree-GRPO [18] results are reported only on Qwen2.5-7B after frequent Qwen3-family training crashes.RLVR baselines are GRPO, DAPO [6], and GSPO [7]; agentic baselines include Tree-GRPO [18], GiGPO [11], IGPO [20], and AEPO [10].Source: §5.1 · p. 7

5.3 Analysis

• A2TGPO maintains a balanced entropy plateau and leads baselines in validation accuracy throughout training, more stable than AEPO and more exploratory than classic RLVR methods.Classic RLVR methods undergo rapid entropy collapse, whereas AEPO’s entropy rises steadily.Source: Figure 3 · p. 8, §5.3 · p. 8
• Turn-group normalization removes position bias, while variance rescaling compresses A2TGPO’s cumulative advantages into a consistent band across turn positions.IGPO’s pooled normalization misreads the naturally higher information gain of the first tool call as superiority, distorting credit for later turns; its cumulative aggregation also produces widely spread values and large outliers.Source: §5.3 · pp. 8–9, Figure 4 · p. 9
• Across 240 training steps, A2TGPO keeps its advantage envelope comparable to GRPO’s bounded range, while IGPO shows wider minimum/maximum swings and slower mean convergence toward zero.A2TGPO retains per-turn process credit signals that GRPO lacks while maintaining distributional stability.Source: Figure 5 · p. 9, §5.3 · p. 9

6 Conclusion

• A2TGPO consistently outperforms general-purpose RLVR methods and specialized agentic baselines across seven QA benchmarks and three backbones while maintaining a balanced entropy regime.The paper leaves extension to broader tool suites and longer-horizon agentic tasks for future work.Source: §6 · p. 9

A Algorithm Workflow

• Each A2TGPO iteration generates multi-turn tool-use rollouts, computes per-turn information gain from forward logits, constructs turn-group-normalized and variance-rescaled advantages, then updates the policy with IG-adaptive clipping.The final answer turn uses outcome advantage only.Source: A Algorithm Workflow · p. 13

B.1 Reward Design / Prompt Template

• The binary rule-based reward combines exact-match correctness with a required response format, awarding maximal reward only when both are satisfied.Responses must include <think> and <answer> tags, with the answer enclosed in \boxed{}; format violations receive no credit regardless of correctness.Source: §B.1 · p. 13, Prompt Template · p. 14

B.2 Prompt Template

• The prompt schema separates reasoning, search queries, retrieved results, and final answers with dedicated tags, and extracts the canonical prediction from the boxed answer span.Search queries appear inside <search> tags, while tool feedback is marked with <result> tags; the template adapts the schema used in [4] [10] [12].Source: §B.2 · p. 14

B.3 Datasets

• The evaluation covers multi-hop QA requiring compositional evidence across retrieved passages and single-hop QA assessing retrieval-regime performance.Multi-hop datasets include HotpotQA [28], 2WikiMultiHopQA [29], MuSiQue [30], and Bamboogle; single-hop datasets include NQ, TriviaQA [33], and PopQA.Source: §B.3 · p. 15

B.4 A2TGPO Settings / B.5 Baseline Settings / B.6 Search Tool Environment / B.7 Hardware and Artifacts

• Implementation uses a batch size of 64, rollout size 16, maximum tool usage 6, γ = 1.0, and β = 0.3; baselines omit additional SFT, and runs use a Wikipedia retrieval corpus on 8× NVIDIA H20 GPUs.A2TGPO clipping thresholds are 3e-3 and 4e-3, with adaptive scale bounded within (0.7, 1.3); evaluation reports average EM at the highest-scoring checkpoint. Retrieval uses e5-base-v2 over approximately 21M Wikipedia entries, and the stack uses VeRL with Qwen3-4B, Qwen3-8B, and Qwen2.5-7B backbones.Source: §B.4 · p. 15, §B.5 · p. 15, §B.6 · p. 16

B.8 Limitations and Future Work

• A2TGPO requires a ground-truth answer to compute information gain, limiting direct applicability to tasks without verifiable outcomes, such as open-ended creative generation or subjective evaluation.The passage identifies this as a limitation on the method’s direct applicability.Source: §B.8 · p. 16
• Self-consistency across rollouts or uncertainty reduction in policy representations are proposed as alternative intrinsic signals to broaden A2TGPO’s scope.These are suggested directions for tasks where ground-truth answers are unavailable.Source: §B.8 · p. 16

C.1 Computational Overhead

• A2TGPO averages ∼525 s/step versus GRPO’s ∼511 s/step under rollout budget n = 16, a net overhead of only +2.9%.Figure 7 reports the matched-budget comparison.Source: Figure 7 · p. 17, §C.1 · p. 17
• The IG forward pass adds ∼164 s, largely offset by faster generation (−86 s), for a net overhead of only +15 s.The forward pass uses the efficient single-pass implementation of Wang et al. [20].Source: Figure 7 · p. 17, §C.1 · p. 17

B.8 Limitations and Future Work

• Amortized estimation or cached incremental updates are proposed to reduce information-gain computation cost and enable scaling to agentic tasks with dozens of interaction turns.The passage describes the current IG forward-pass overhead as modest, while noting further efficiency strategies could reduce it.Source: §B.8 · pp. 16–17

C.1 Computational Overhead

• A2TGPO initially produces a higher minimum response length, indicating that rollouts explore substantially rather than terminate prematurely.This early-stage exploration accounts for elevated training time during the first ∼50 steps.Source: §C.1 · p. 17, Figure 8 · p. 17
• After the transient exploration phase, A2TGPO’s maximum response length drops from ∼6000 to ∼4000 tokens, while GRPO continues to hit the ∼6192-token limit.Because parallel generation is bottlenecked by each batch’s longest sequence, shorter maxima yield progressively faster generation for A2TGPO.Source: §C.1 · p. 17, Figure 8 · p. 17

C.2 Ablation Study on Single-hop Benchmarks

• On single-hop benchmarks, all three A2TGPO components remain effective, with adaptive clipping contributing ∼50%, turn-group normalization ∼35%, and variance-rescaled accumulation ∼15%.The ablation attributes the largest gain to adaptive clipping because single-hop tasks involve fewer turns.Source: §C.2 · pp. 17–18, Table 4 · p. 18
• IG-adaptive turn-level clipping contributes the largest single-hop gain (∼50%), where per-turn update intensity becomes the primary factor.The passage links this effect to single-hop tasks involving fewer turns.Source: §C.2 · p. 17
• Turn-group normalization provides a moderate gain (∼35%) on single-hop benchmarks.Source: §C.2 · p. 17, Table 4 · p. 18
• Variance-rescaled accumulation contributes less (∼15%) because single-hop trajectories typically contain 1 to 2 process turns, reducing the need for cross-depth scale correction.Source: §C.2 · p. 18, Table 4 · p. 18

C.3 Sensitivity to Adaptive Clipping Coefficient β

• With β=0, which removes adaptive clipping, A2TGPO still outperforms GRPO and IGPO, supporting the standalone effectiveness of turn-group normalization and variance rescaling.The clipping range is ci,t ∈[1−β, 1+β].Source: §C.3 · p. 18, Figure 9 · p. 18
• Performance improves monotonically through β=0.3, reaching multi-hop 48.06 and single-hop 56.44, then degrades only mildly beyond it.Source: §C.3 · p. 18, Figure 9 · p. 18
• Performance remains within 0.15 points of the optimum across β∈[0.2, 0.4], and the paper fixes β=0.3 in all experiments.Figure 9 sweeps β∈{0, 0.1, . . . , 0.5}.Source: §C.3 · p. 18, Figure 9 · p. 18

C.4 Turn Distribution Analysis

• At validation step 240, multi-hop rollouts average 3.45 tool calls (72% use 2–4), versus 2.05 for single-hop (65% use 1–2), with zero-call rates of 2.5% and 4.5%.The paper states that these variable-length distributions make turn-group normalization critical for unbiased credit assignment.Source: §C.4 · p. 18

C.5 Context Similarity Analysis

• At step 1, same-position rollouts show moderate Jaccard similarity of ∼0.6–0.7, decreasing with depth.Similarity is computed across per-turn thinking, query, and tool-response content for rollouts of the same prompt.Source: §C.5 · p. 18
• Overall intra-position similarity is 0.62, exceeding cross-position similarity of 0.38 by 63%.The authors interpret this as evidence that turn-group normalization compares turns under comparable contexts, unlike pooled normalization, which conflates quality differences with positional heterogeneity.Source: §C.5 · p. 19
• After training at step 250, intra-position similarity reaches 0.86 at turn 1 and 0.67 at turn 2.The paper interprets this increase as indicating higher-quality, more consistent tool calls at each position.Source: §C.5 · p. 19

D Theoretical Analysis

• Section D.1 shows that IGPO’s pooled normalization introduces systematic position-dependent bias, while turn-group normalization provably eliminates it.This formal result addresses the positional heterogeneity that motivates turn-group normalization.Source: D Theoretical Analysis · p. 19
• Section D.2 establishes turn-group normalization’s unbiasedness and robustness, including for small groups caused by variable rollout lengths.Source: D Theoretical Analysis · p. 19
• Section D.3 proves that dividing cumulative IG by the square root of accumulated terms equalizes advantage variance across turn depths.Source: D Theoretical Analysis · p. 19
• Section D.4 characterizes how IG-adaptive clipping modulates the gradient.Source: D Theoretical Analysis · p. 19

D.1 Positional Bias under Pooled Normalization

• The analysis assumes position-specific IG distributions have heterogeneous means; different contextual states, such as accumulated evidence or remaining sub-question specificity, motivate this assumption.The number of rollouts reaching a position is non-increasing with depth; positions reached by only one rollout receive zero process signal in implementation.Source: §D.1 · p. 19
• IGPO pools information gains across valid rollout-position pairs and applies group-wise z-normalization.Source: §D.1 · p. 19
• In the population limit, pooled-normalized signals have nonzero expected values at positions whose mean IG differs from the weighted grand mean, regardless of individual tool-call quality.The resulting artifact can conflate positional characteristics with action quality.Source: §D.1 · p. 20
• Because more rollouts reach early turns, those positions weigh more heavily in the pooled mean, biasing normalization against sparse deeper positions whose mean IG may differ.Source: §D.1 · p. 20

D.2 Unbiasedness and Robustness of Turn-Group Normalization

• Turn-group normalization compares information gain within each turn-position group, anchoring average-quality actions at zero and avoiding pooled normalization’s positional bias.The paper reports per-turn mean drift of ΔĀt = 0.00 under turn-group normalization versus ΔĀt = 1.03 under pooled normalization.Source: §D.2 · pp. 20–21
• For any group size Gt ≥2, turn-group normalization is unbiased; smaller groups increase estimator variance without introducing systematic positional error.In the population limit, the normalized signal has zero mean and unit variance.Source: §D.2 · pp. 20–21
• At deeper positions with small Gt, the outcome term can dominate when outcome divergence is large, making the advantage less sensitive to noisier process-credit estimates.The sigmoid gate also bounds the adaptive clip scale to (1 − β, 1 + β), limiting the influence of an extreme normalized IG estimate.Source: §D.2 · p. 21

D.3 Variance Homogeneity under Square Root Rescaling

• Dividing cumulative normalized IG by √ni,t yields approximately constant advantage variance across turn depths when residual pairwise covariance is small.Under the stated weak-dependence assumption, the normalized signals have mean zero and common variance, with pairwise covariance bounded by ρσ².Source: §D.3 · p. 22
• Without √ni,t rescaling, shallow turns can dominate because variance grows with accumulated terms; dividing by ni,t instead would over-correct and compress shallow-position signals.The square-root divisor balances these extremes while retaining long-horizon credit propagation.Source: §D.3 · pp. 21–22

D.4 Gradient Modulation under Adaptive Clipping

• The adaptive clip scale ci,t = 1 + β (2σ(bigi,t) − 1) increases monotonically with normalized IG and remains bounded in (1 − β, 1 + β).The scale adjusts the effective per-turn clipping bounds in the policy objective.Source: §D.4 · p. 22
• High-IG turns receive wider clipping ranges, while low- or negative-IG turns receive narrower ranges; the scale depends on normalized IG, not advantage magnitude.This separates relative informativeness among positional peers from the overall gradient magnitude, avoiding compounding a large advantage with a wide clipping range.Source: §D.4 · p. 23

E Case Study

• Multi-hop trajectories decompose questions into sequential sub-goals and use multiple tool-call turns, while single-hop tasks typically need one well-targeted query.The case study presents one A2TGPO validation trajectory for each of the seven evaluation benchmarks.Source: E Case Study · p. 23
• In a HotpotQA example, the agent identifies Star Trek V director William Shatner, searches his nationality, and concludes he is Canadian.The trajectory first retrieves the film’s director and then uses a separate search for Shatner’s nationality.Source: E Case Study · p. 25
• In a Bamboogle example, the agent finds the Space Needle in Seattle, searches for Washington’s capital, and concludes it is Olympia.The trajectory uses a second search to connect the Space Needle’s location to the requested state capital.Source: E Case Study · p. 27