PaperlayerView exact revision on arXiv

Source-linked AI summary

AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

Haotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S. -T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu

arXiv:2605.00425v3cs.AI

TL;DR

Sparse outcome-only rewards make credit assignment difficult in long multi-turn agent trajectories. AEM uses response-level entropy to adaptively rescale advantages, consistently improving strong RL baselines across three benchmarks, including a 1.4% gain on SWE-bench-Verified.

  • Problem

    Sparse outcome-based rewards provide nearly indistinguishable learning signals across steps in long multi-turn trajectories, leaving credit assignment ambiguous.

  • Method

    AEM is a supervision-free plug-in that uses response-level entropy and relative surprisal to adaptively rescale advantages, transitioning from exploration to exploitation.

  • Results

    AEM consistently improves strong group-based RL baselines across ALFWorld, WebShop, and SWE-bench-Verified, including a +1.4% improvement over DeepSWE.

  • Takeaways & Limitations

    Response-level entropy modulation provides a lightweight route to denser credit assignment without auxiliary models or additional supervision.

  • Takeaways & Limitations

    AEM assumes complete responses are the effective interaction units and uses a relative-surprisal proxy rather than reconstructing the exact state-specific entropy gap.

Abstract

from arXiv · show

Reinforcement learning (RL) has substantially improved the ability of large language model (LLM) agents to interact with environments and solve multi-turn tasks. However, effective agentic RL remains challenging: sparse outcome-only rewards provide limited guidance for assigning credit to individual steps within long interaction trajectories. Existing approaches often introduce dense intermediate supervision, such as process reward models or auxiliary self-supervised signals, which increases supervision and tuning complexity and may limit generalization across tasks and domains. We present AEM, a supervision-free credit assignment method that adaptively modulates entropy dynamics during RL training to improve the exploration-exploitation trade-off. Since in agentic RL the environment is typically affected by a complete response, rather than an individual token, our analysis lifts entropy dynamics from the token level to the response level, aligning uncertainty estimation with the effective action granularity of LLM agents and reducing sensitivity to token-level sampling noise. We further show that entropy drift under natural-gradient updates is governed by the interaction between the sampled-response advantage and its relative surprisal. Motivated by this result, AEM derives a practical response-level uncertainty proxy and uses it to rescale advantages, leveraging the evolving balance between positive and negative samples to naturally transition from exploration to exploitation. Extensive experiments on ALFWorld, WebShop, and SWE-bench-Verified with models ranging from 1.5B to 32B demonstrate that AEM consistently improves strong RL baselines, including a +1.4\% gain when integrated into a state-of-the-art software-engineering RL training framework.

1 Introduction

Multi-turn agentic RL is hindered by sparse outcome-based feedback, while existing dense-credit alternatives add supervision or computational overhead. AEM addresses this with supervision-free response-level entropy modulation, theoretically linking entropy drift to advantage and surprisal and empirically improving strong RL baselines.

  • Motivation: Multi-turn agentic RL receives sparse, outcome-based feedback, making credit assignment across long interaction trajectories fundamentally challenging.Agents repeatedly observe feedback, select actions, and refine behavior across sequential interactions.
  • Motivation: Existing credit-assignment methods require additional models or annotations for dense supervision, or incur high computational overhead through branching trajectories.Process reward models provide step-level supervision, whereas TreeGRPO and ATPO propagate credit through tree-structured optimization.
  • Method: Response-level entropy provides an intrinsic credit signal because high-entropy responses indicate exploration, low-entropy responses indicate confidence, and completed responses change the environment state.AEM therefore treats the completed response as the effective action granularity for credit modulation.
  • Theory: Entropy drift is governed by the interaction between sampled-response advantage and relative surprisal, establishing response-level uncertainty as a principled intrinsic signal for credit assignment.This response-level analysis motivates AEM’s entropy-derived uncertainty proxy.
  • Method: AEM is a supervision-free, lightweight, plug-in method that rescales response-level advantages with an entropy-derived uncertainty proxy to transition from exploration to exploitation.The transition follows the evolving balance between positive and negative samples during training.
  • Experiments: 8.8% peak gains occur on GRPO with Qwen2.5-1.5B on ALFWorld, while DeepSWE gains +1.4% on SWE-bench-Verified.Experiments span ALFWorld, WebShop, and SWE-bench-Verified with models from 1.5B to 32B, and AEM consistently improves multiple strong group-based RL baselines.

2 Related Work

Related work frames LLMs as interactive decision-makers trained with group-based RL methods, while highlighting sparse-reward credit assignment and entropy-aware optimization as central directions. Existing approaches derive credit from external supervision or sampled trajectories and use entropy to promote exploration and training stability.

  • From LLMs to Agentic RL: ReAct and Toolformer established LLMs as agents that interleave reasoning, actions, and external tool calls, motivating RL for interactive decision-making.Group-based methods including RLOO and GRPO have emerged as a dominant approach for training such agents.
  • Credit Assignment in Agentic RL: Credit assignment in agentic RL remains challenging because rewards are delayed and sparse, with methods differing in how and where they derive step-level signals.External signals such as value functions or step-level supervision add modeling and scaling overhead, while other methods derive credit internally from sampled trajectories.
  • Entropy-Aware Policy Optimization: Entropy has traditionally regularized RL to promote exploration and improve training stability, while recent work incorporates entropy into optimization objectives and advantage scaling.The cited approaches include entropy-regularized policy optimization and entropy-guided advantage scaling.

3 Theoretical Analysis

The analysis establishes response-level entropy as a structurally faithful uncertainty measure for agentic RL, matching the environment’s complete-response interaction granularity. It further shows that entropy drift is governed by the interaction between sampled-response advantage and relative surprisal, motivating response-level advantage modulation.

  • Response-level entropy: Because environments typically react after complete responses, response-level entropy aligns uncertainty estimation with the effective interaction unit and is less sensitive to token-level sampling variation.Response-level modulation induces corresponding changes in policy entropy.
  • Response-level entropy: Response-level entropy is the expectation of token-level entropy sums, while policy entropy is the expectation of response-level entropy sums.Theorem 3.2.1 formalizes the relationship among token, response, and policy entropy.
  • Entropy drift: Under natural-gradient updates, entropy drift is determined by the sampled response’s advantage multiplied by its relative surprisal.The result is derived on the policy simplex equipped with the Fisher-Rao metric and aggregated over visited states.
  • Entropy drift: If sgn(A(a, s)(S(a | s) −Hresp)) > 0, entropy increases; if it is < 0, entropy decreases.Thus, advantage and relative surprisal jointly determine the direction of entropy change.
  • Implication for AEM: Rescaling response advantages by relative surprisal can induce entropy-increasing or entropy-decreasing pressure without changing the underlying RL optimization backbone.State-level entropy regularization and KL penalties do not change this response-dependent modulation principle.

4 AEM: Adaptive Entropy Modulation

AEM is a supervision-free, response-level advantage modulation method that uses relative surprisal to regulate entropy dynamics. Its self-calibrated modulation shifts training from exploration toward exploitation while mitigating early entropy collapse and improving convergence.

  • Response-level modulation: AEM applies a scalar coefficient uniformly across each completed response span, rescaling response-level advantages on top of a base advantage estimator.This aligns modulation with the response granularity that affects the environment rather than individual sampled tokens.
  • Adaptive entropy dynamics: AEM increases entropy pressure for negative responses and decreases it for positive responses, naturally transitioning from exploration to exploitation as positive responses become more prevalent.Negative responses are relatively prevalent early in training, while positive responses become relatively prevalent later.
  • Self-calibrated uncertainty proxy: The coefficient uses normalized relative surprisal: α > 1 upweights lower-surprisal responses, whereas α < 1 downweights higher-surprisal responses within a response group.AEM sets α = 1 when within-group proxy variation is below 0.1, reducing sensitivity to sampling noise.
  • Self-calibrated uncertainty proxy: AEM estimates response uncertainty with a length-normalized, predictable surprisal proxy and maps it monotonically to a response-uniform coefficient.The proxy reduces sensitivity to particular sampled tokens while preserving response-level entropy scaling.
  • Empirical effects: AEM mitigates early entropy collapse, promotes more complete late-stage convergence, and improves final performance.Empirical analyses support the connection between α −1 and relative surprisal and show that advantage together with α −1 determines practical entropy dynamics.

5 Experiments

Experiments on ALFWorld, WebShop, and SWE-bench-Verified show that AEM consistently improves RL baselines across benchmarks and model scales. Mechanism analyses support its response-level uncertainty modulation, adaptive exploration–exploitation transition, and low computational overhead.

  • Performance on ALFWorld and WebShop: AEM consistently improves group-based RL baselines on ALFWorld and WebShop across benchmarks and model scales, sometimes reaching performance competitive with strong closed-source models.These results validate AEM as an effective plug-in mechanism for multi-turn agent training.
  • Performance on SWE-bench-Verified: 43.7% resolved rate: DeepSWE+AEM outperforms the reproduced DeepSWE baseline by 1.4% on SWE-bench-Verified with Qwen3-32B.The reproduced DeepSWE baseline achieves a 42.3% average resolved rate.
  • Analysis A: Consistency between α−1 and −(S − Hresp): Pearson correlation r = 0.63: α−1 shows a clear positive relationship with Monte Carlo relative surprisal, with matching signs in 55 of 64 states (85.9%).The probing study samples n = 64 states and K = 64 responses per state.
  • Analysis B: Validating the trend of entropy: Entropy dynamics are jointly determined by the sampled-response advantage A(a) and α−1, producing divergent trends under opposite gradient-masking strategies.The analysis visualizes entropy over the first 50 training steps during GRPO training.
  • Analysis C: AEM induces an exploration-exploitation transition: AEM preserves higher early-stage entropy and gradually reduces it later, adaptively transitioning from exploration toward exploitation as positive samples become more prevalent.The baseline instead undergoes abrupt early entropy collapse and remains relatively flat, while AEM uses increasing success rates and positive-sample proportions to shift entropy dynamics.
  • Computational overhead: 1.1%: AEM-specific computations account for only 1.1% of policy latency, while requiring neither extra rollouts nor additional policy or reference model forward passes.Response-level entropy values are obtained during the same recomputation pass used for old-policy log-probabilities.

6 Conclusions · A Pseudo-code Algorithm

AEM provides a supervision-free, response-level entropy method for credit assignment in multi-turn agentic RL, adaptively regulating exploration and exploitation. Algorithm 1 operationalizes this method by computing response uncertainty, deriving group-normalized modulation coefficients, and returning modulated advantages.

  • 6 Conclusions: AEM uses response-level entropy as an intrinsic signal for supervision-free credit assignment in multi-turn agentic RL.Its entropy dynamics are governed by the interaction between advantage and relative response surprisal.
  • 6 Conclusions: Adaptive entropy modulation regulates entropy dynamics and enables a natural transition from exploration to exploitation.The method is designed as a lightweight plug-in to existing advantage estimators.
  • A Pseudo-code Algorithm: Algorithm 1 illustrates the pseudo-code procedure for AEM.The algorithm takes batch responses, response entropies, temperature λ, and stability constant ε as inputs, and ensures modulated AEM advantages.
  • A Pseudo-code Algorithm: The procedure parses rollouts into environment-reactive agentic responses and computes a response-level uncertainty proxy for each response.This establishes the response-level quantities used in subsequent modulation.
  • A Pseudo-code Algorithm: For each response group, the algorithm finds group extrema and handles groups whose entropy range satisfies the stated threshold by setting α_i,t to 1.The threshold condition shown is H̄max_G − H̄min_G < 0.1.
  • A Pseudo-code Algorithm: Otherwise, AEM computes raw modulation coefficients using exp(−λ H̃_i,t) and then normalizes them by the group-average coefficient plus ε.The normalization is applied within each response group.
  • A Pseudo-code Algorithm: Finally, the algorithm applies response-level uniform modulation across rollouts and responses before returning the modulated AEM advantages.The returned object is {A^AEM_i,t}.

B Limitations · C Broader Impact

AEM’s response-level uncertainty proxy is a heuristic approximation rather than an exact estimator, while the method may reduce supervision and engineering costs but also heighten risks from more capable agents without oversight.

  • B Limitations: Hresp(s) is not directly computable for open-ended LLM policies because it requires summing over the entire response space.
  • B Limitations: AEM therefore approximates relative response surprisal using a group-based, length-normalized entropy proxy.
  • B Limitations: The proxy has statistical support from experiments but remains a heuristic surrogate rather than an exact estimator, so AEM does not guarantee optimal entropy modulation.
  • C Broader Impact: AEM is presented as a supervision-free, lightweight, plug-in method for entropy-aware response-level credit modulation in multi-turn agentic RL.
  • C Broader Impact: By improving credit assignment under sparse outcome-only rewards across different backbones, AEM may help make LLM agents more effective in long-horizon interactions.
  • C Broader Impact: Methods that improve training efficiency without additional dense supervision or auxiliary reward models may reduce engineering complexity and development costs.
  • C Broader Impact: More capable and efficiently trained long-horizon agents could amplify risks without sufficient oversight, including misuse in autonomous web interaction, large-scale automation, or software manipulation.
  • C Broader Impact: The authors conclude that AEM contributes a valuable tool for improving the reliability and sample efficiency of agentic RL research.

D Experimental Training Curves

The section presents training-curve analyses across ALFWorld, WebShop, and the R2E dataset, covering Qwen2.5 models from 1.5B to 7B and DeepSWE with and without AEM.

  • Qwen2.5-1.5B: Training curves are shown for Qwen2.5-1.5B on ALFWorld and WebShop.The ALFWorld and WebShop plots are presented separately.
  • Qwen2.5-7B: Training curves are also shown for Qwen2.5-7B on ALFWorld and WebShop.Both environments are evaluated with the 7B model.
  • DeepSWE: The section includes DeepSWE training reward curves comparing runs with and without AEM on the R2E dataset.This comparison examines reward trajectories under the two training configurations.

E Ablation Study · F Theoretical Details and Proofs

The ablation study shows that full AEM’s gains depend on correctly aligning entropy modulation with responses and using group normalization, while the theoretical section supplies mathematical details and proofs for the main algorithms, theorems, and properties.

  • E Ablation Study: The ablation compares GRPO, AEM, shuffled and reversed modulation, trajectory normalization, and batch normalization under controlled WebShop experiments.The study uses Qwen2.5-1.5B and holds training configuration, rollout budget, and evaluation protocol constant.
  • E Ablation Study: Full AEM achieves the best WebShop Score and Success Rate, whereas improper modulation strategies reduce or sometimes negate these gains.All variants use identical training configurations, rollout budgets, and evaluation protocols; only the construction or application of α differs.
  • E Ablation Study: AEMshuffle remains comparable to GRPO but is clearly worse than AEM, showing that entropy coefficients must remain aligned with their corresponding responses.Shuffling preserves α’s marginal distribution and scale while destroying response-specific uncertainty alignment.
  • E Ablation Study: AEMreverse performs substantially worse than GRPO, indicating that reversing the entropy-to-credit mapping actively harms performance.The reversed mapping tends to exacerbate entropy-related misassignment of credit.
  • E Ablation Study: Group normalization is the most suitable of the three tested choices because it aggregates multiple responses while avoiding task-mixing entropy bias.Trajectory normalization uses weaker statistics, whereas batch normalization can mix tasks; group responses share the same prompt, making entropy values more comparable.
  • E Ablation Study: Table 3 reports ablation performance as the mean and sample standard deviation over 3 runs.The table covers the WebShop benchmark ablation study.
  • F Theoretical Details and Proofs: The theoretical section provides mathematical details, theorem proofs, and proofs of properties related to the algorithms presented in the main text.This section formalizes the paper’s principal algorithmic claims rather than reporting additional experiments.

F.1 Proof of Theorem 3.2.1 · F.2 Policy Simplex

The proof establishes that response-level entropy equals the conditional expectation of pathwise token-entropy sums and, under on-policy rollouts, aggregates exactly to policy entropy over visited states. The policy-simplex section equips finite-action policies with the Fisher–Rao metric, which measures update size in local KL-trust-region units.

  • F.1 Proof of Theorem 3.2.1: Step 1 identifies response-level entropy as the conditional expectation of the pathwise token-entropy sum.The argument uses the autoregressive policy factorization and the tower property.
  • F.1 Proof of Theorem 3.2.1: Step 2 shows that policy entropy is the expected sum of response-level entropies over on-policy visited states.This step assumes on-policy rollouts and applies conditional expectations at each visited state.
  • F.1 Proof of Theorem 3.2.1: Under on-policy rollouts, policy entropy is exactly the expected aggregation of response-level entropies over visited states.Combining the two steps completes the proof.
  • F.2 Policy Simplex: For a fixed state with finite action space |A_s| = m, the policy π = (π_θ(a|s))_{a∈A_s} lies on the policy simplex.The simplex is defined over the finite action set at that state.
  • F.2 Policy Simplex: The policy simplex is equipped with the Fisher–Rao metric, forming a Riemannian manifold.The metric is defined on tangent vectors within the interior of the simplex.
  • F.2 Policy Simplex: The Fisher–Rao metric is the infinitesimal quadratic form induced by KL divergence.It evaluates tangent perturbations in the policy simplex.
  • F.2 Policy Simplex: The Fisher–Rao metric measures the local size of a policy update in the same units as a local KL trust region.This connects geometric update magnitude with KL-based locality.

F.3 State and Proof of the Generalized Version of Theorem 3.2.2 … G.1 Base RL Methods Used in Experiments

The appendices generalize response-level entropy-drift theory, decompose response surprisal, and account for parameterization effects, while G.1 describes the base RL methods used experimentally. Together, these sections connect AEM’s theoretical analysis to PPO, GRPO, DAPO, and GSPO baselines.

  • F.3 State and Proof of the Generalized Version of Theorem 3.2.2: F.3 decomposes regularized response-level entropy drift into reward-driven, entropy-expanding, and reference-alignment effects.With β = γ = 0, the result reduces to Theorem 3.2.2; entropy and KL regularization act as state-level modulation terms, unlike the reward-driven term.
  • F.3 State and Proof of the Generalized Version of Theorem 3.2.2: The reward-driven term shows that sampled-action advantage and relative surprisal jointly determine entropy dynamics even without entropy or KL regularization.The KL contribution additionally includes a positive variance term and a covariance-related term whose sign is generally not fixed.
  • F.4 Doob’s decomposition of fixed-length response surprisal: F.4 applies Doob’s decomposition to express fixed-length response surprisal as cumulative conditional entropy deviations plus a terminal martingale term.For a realized response a = (Y1, . . . , YL), the decomposition relates S(a | s) − Hresp(s) to token-level entropy deviations and ML.
  • F.5 Parametrized Version of Entropy Drift: F.5 shows that parameter-space entropy drift contains a kernel-weighted baseline Bker(a;s) arising from shared parameterization across responses.The Euclidean drift therefore differs from the simplex-level result by a cross-response residual.
  • Proof of Theorem F.5.1: The proof of Theorem F.5.1 derives a general regularized parameter-space entropy-drift identity before specializing it to entropy and KL regularization.The derivation uses the response-level entropy gradient, a general smooth regularizer, and a split separating the sampled response from the task-driven part.
  • G.1 Base RL Methods Used in Experiments: PPO uses clipped actor-critic updates with token-level advantages, whereas GRPO replaces the critic with within-group relative rewards while retaining clipped optimization.PPO requires an additional critic/value model, while GRPO eliminates that critic for large-scale LLM reinforcement learning.
  • G.1 Base RL Methods Used in Experiments: DAPO aggregates advantages at token level and adds asymmetric clipping, dynamic sampling, token-level policy-gradient loss, and overlong reward shaping.The original formulation also removes the explicit KL term and targets long-form reasoning trajectories.
  • G.1 Base RL Methods Used in Experiments: GSPO moves importance weighting and clipping to sequence level, using length-normalized ratios to align optimization with sequence-level rewards and improve training stability.It retains the normalized group-based advantage used by GRPO while keeping ratio scales comparable across response lengths.

G.2 Implementation Details · G.3 Prompts

The implementation uses rule-based outcome rewards, fixed group-based rollout settings, and task-specific training configurations, while prompts structure environment interaction, reasoning, admissible-action selection, and programming-tool calls. AEM experiments use fixed normalization parameters and results averaged over three random seeds.

  • G.2 Implementation Details: Rule-based rewards assign 10 for successful ALFWorld and WebShop trajectories, 0 for failures, and −0.1 for invalid actions, while SWE uses binary rewards.Across group-based RL methods, the rollout group size is fixed to N = 8.
  • G.2 Implementation Details: ALFWorld and WebShop training uses verl-agent with actor learning rate 1 × 10−6, rollout temperature 1.0, validation temperature 0.4, and KL coefficient 0.01.The setup samples 16 groups per rollout, yielding 128 environments in total.
  • G.2 Implementation Details: SWE training uses rLLM for Qwen3-32B with learning rate 1 × 10−6, prompt and response limits of 4096 and 65536 tokens, and batch size 64.Rejection sampling uses a 2× oversampling ratio and rejects rollout groups whose rewards are all 0 or all 1.
  • G.2 Implementation Details: All reported results average 3 random seeds, with AEM parameters fixed to λ = 1 and ϵ = 10−8.λ controls the modulation coefficient range, while ϵ prevents numerical instability in normalization procedures.
  • G.3 Prompts: ALFWorld prompts provide the task, recent observation-action history, current step and observation, and admissible actions before requiring step-by-step reasoning.The response must enclose reasoning in <think> </think> tags and the selected action in <action> </action> tags.
  • G.3 Prompts: WebShop prompts similarly include task context, interaction history, current observations, and available actions, then require reasoning before selecting an action advancing the shopping goal.The prompt enforces the same <think> </think> reasoning format and <action> </action> action format.
  • G.3 Prompts: R2E Training prompts define a programming-agent task with repository access and expose file_editor, execute_bash, search, and finish functions.Function calls must follow the specified format without a suffix.
  • G.3 Prompts: SWE-bench-Verified evaluation prompts use the same repository-based programming-agent framing and four-function interface, with function calls constrained to the prescribed format.The functions are file_editor, execute_bash, search, and finish.
Loading 2605.00425v3…