PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents

Jiawei Wang, Jiacai Liu, Yuqian Fu, Yingru Li, Xintao Wang, Yuan Lin, Yu Yue, Lin Zhang, Yang Wang, Ke Wang

arXiv:2509.09265v1cs.LGcs.CL

TL;DR

Long-horizon LLM agents struggle to assign credit from sparse outcome rewards, while policy-gradient magnitude is coupled to step uncertainty. EMPG re-calibrates updates with entropy-aware scaling and a future-clarity bonus, substantially outperforming strong policy-gradient baselines across WebShop, ALFWorld, and Deep Search.

  • Problem

    Sparse outcome-based rewards make it difficult to assign credit to crucial intermediate steps in long-horizon LLM-agent tasks.

  • Method

    EMPG combines uncertainty-based Self-Calibrating Gradient Scaling with a Future Clarity Bonus to re-calibrate step-level learning signals.

  • Results

    EMPG substantially outperforms strong policy-gradient baselines across WebShop, ALFWorld, and Deep Search.

  • Takeaways & Limitations

    Intrinsic step-wise uncertainty can provide a scalable, dense learning signal from minimal external feedback for complex decision-making.

  • Takeaways & Limitations

    The framework uses policy entropy as its uncertainty estimator, while alternative estimators remain a future direction.

Abstract

from arXiv · show

In long-horizon tasks, recent agents based on Large Language Models (LLMs) face a significant challenge that sparse, outcome-based rewards make it difficult to assign credit to intermediate steps. Previous methods mainly focus on creating dense reward signals to guide learning, either through traditional reinforcement learning techniques like inverse reinforcement learning or by using Process Reward Models for step-by-step feedback. In this paper, we identify a fundamental problem in the learning dynamics of LLMs: the magnitude of policy gradients is inherently coupled with the entropy, which leads to inefficient small updates for confident correct actions and potentially destabilizes large updates for uncertain ones. To resolve this, we propose Entropy-Modulated Policy Gradients (EMPG), a framework that re-calibrates the learning signal based on step-wise uncertainty and the final task outcome. EMPG amplifies updates for confident correct actions, penalizes confident errors, and attenuates updates from uncertain steps to stabilize exploration. We further introduce a bonus term for future clarity that encourages agents to find more predictable solution paths. Through comprehensive experiments on three challenging agent tasks, WebShop, ALFWorld, and Deep Search, we demonstrate that EMPG achieves substantial performance gains and significantly outperforms strong policy gradient baselines. Project page is at https://empgseed-seed.github.io/

1 Introduction

Long-horizon LLM agents receive sparse outcome feedback, making intermediate credit assignment difficult. EMPG re-calibrates policy-gradient updates using step-wise uncertainty and reports gains over strong baselines on demanding agent tasks.

  • Sparse end-of-task rewards make it difficult for reinforcement learning algorithms to identify crucial intermediate steps in long-horizon tasks.
  • Policy-gradient magnitude increases with entropy, causing small updates for confident actions and large, potentially destabilizing updates for uncertain actions.
  • EMPG uses Self-Calibrating Gradient Scaling to amplify confident correct actions and attenuate uncertain steps for stable exploration.
  • A Future Clarity Bonus encourages actions leading to more predictable states, complementing uncertainty-based gradient modulation.
  • EMPG substantially outperforms strong baselines including GRPO and DAPO on WebShop, ALFWorld, and Deep Search.

2 Related Work

Related work addresses sparse rewards through internal feedback, uncertainty signals, and self-consistency, but entropy has conflicting interpretations. Existing entropy-based methods mainly target single-turn reasoning, leaving long-horizon multi-step credit assignment underexplored.

  • Prior methods use internal uncertainty or self-consistency signals, often quantified by policy entropy, to create denser training feedback.
  • Entropy minimization treats lower uncertainty as a proxy for correctness, while other approaches use entropy to modulate learning signals.

3 Preliminaries

The paper formulates long-horizon agent training as sparse-reward reinforcement learning and reviews policy-optimization strategies for reducing variance and improving data efficiency. It then motivates EMPG by linking gradient magnitude to entropy and combining gradient scaling with a future-clarity signal.

  • Long-horizon trajectories provide zero reward at non-terminal steps, making credit assignment and exploration fundamental reinforcement-learning challenges.
  • PPO stabilizes updates with trust-region clipping, while GRPO uses multiple sampled responses to estimate comparative advantages and DAPO curates informative training data.
  • Trajectory-level outcome rewards treat interactive action sequences as monolithic blocks and cannot pinpoint critical actions in long sequences.
  • Proposition 1 characterizes the expected score-function norm as a direct function of Rényi-2 entropy, establishing monotonic coupling between gradient magnitude and policy entropy.
  • Self-Calibrating Gradient Scaling adjusts update magnitude, while the Future Clarity Bonus encourages actions leading toward low-entropy, predictable future states.

4 Entropy-Modulated Policy Gradients

EMPG recalibrates sparse-reward policy gradients using step-level entropy, reweighting confident and uncertain decisions differently while adding a future-clarity signal.

  • Step-level uncertainty: The step-level entropy proxy averages token-level entropy over a single reason-then-act step, with lower entropy indicating greater confidence.Token probabilities come from the LLM policy over its vocabulary.
  • Modulated advantage: EMPG transforms a trajectory-level advantage into step-specific updates using entropy-based scaling and a future-clarity bonus.The first mechanism reweights the shared trajectory advantage, while the second encourages actions leading to predictable future states.
  • Normalization: Gradient scaling is normalized within each mini-batch so modulation redistributes rather than uniformly inflates or deflates the learning signal.Entropy values are batch-normalized before scaling, and the resulting modulated advantages undergo final zero-mean normalization.
  • Self-calibrating gradient scaling: Low-entropy confident steps receive amplified gradients, whereas high-entropy uncertain steps receive attenuated gradients to stabilize exploration.Amplification accelerates confident correct decisions and penalizes confident errors; attenuation limits noisy updates from uncertain steps.
  • Algorithm: EMPG computes trajectory advantages, step entropies, scaling factors, future-clarity bonuses, and normalized modulated advantages before updating the policy.These operations are summarized in the EMPG training algorithm.
  • Future clarity: The future-clarity bonus rewards low entropy in the next step and guides trajectories toward less ambiguous decisions.The bonus is weighted by ζ > 0 and provides an intrinsic signal for predictable solution paths.

5 Experiments

EMPG is evaluated across three sparse-reward agent benchmarks and consistently improves policy-gradient baselines across tasks, model scales, and domains. Analyses associate these gains with complementary exploitation and uncertainty-regularization mechanisms, improved training stability, and step-level entropy effects.

  • Main Results: +8.1 points for GRPO and +7.3 points for DAPO improve ALFWorld on Qwen2.5-1.5B, while DAPO reaches 82.7% success on WebShop with Qwen2.5-7B.Table 1 reports averages over three random seeds and includes ALFWorld subtask and overall success rates plus WebShop score and success rate.
  • Main Results: +3.3 points raises DAPO’s Deep Search overall average from 62.0 to 65.3, including +3.1 ID and +3.9 OOD gains.Table 2 covers domain-partitioned Deep Search results, with the larger OOD improvement indicating robust generalization within the reported evaluation.
  • Main Results: EMPG consistently improves policy-gradient baselines across WebShop, ALFWorld, and Deep Search, across different tasks, model scales, and RL algorithms.The authors describe the method as a versatile and scalable enhancement with consistent gains across all three benchmarks.
  • Ablation Study and Generalization Analysis: The Future Clarity Bonus improves in-domain performance by +2.6 points, while Self-Calibrating Gradient Scaling regularizes uncertain steps and supports out-of-domain robustness.The ablation analysis assigns complementary roles: exploitation of known sequences for ID tasks and attenuation of high-entropy updates for OOD generalization.
  • Enhancing Training Stability: EMPG maintains stable KL Loss while DAPO becomes erratic after approximately 240 steps, indicating more stable policy updates during training.Both methods use the checkpoint at 220 steps for final evaluation, while the EMPG-enhanced run remains stable throughout the displayed training period.
  • Step-Level vs. Token-Level Entropy Dynamics: Step-level analysis over more than 9,000 ALFWorld steps finds substantial entropy changes even among initially low-entropy steps, unlike token-level findings.The result supports analyzing entropy at the reason-act step level rather than relying only on token-level observations.

6 Conclusion

EMPG addresses long-horizon credit assignment by using step uncertainty to recalibrate policy gradients from sparse outcome rewards. Experiments show substantial gains over strong baselines, while the authors position intrinsic uncertainty as a scalable self-supervision signal.

  • EMPG dynamically recalibrates policy gradients to amplify confident correct actions, penalize confident errors, and attenuate uncertain steps.The method uses intrinsic uncertainty to improve credit assignment under sparse end-of-task rewards.
  • Across WebShop, ALFWorld, and Deep Search, EMPG achieves substantial performance gains over strong baselines including GRPO and DAPO.
  • The authors frame EMPG as a general-purpose method for variance reduction and credit assignment using policy uncertainty as an adaptive, step-level baseline.
  • The findings suggest intrinsic uncertainty can provide a scalable self-supervision signal from minimal external feedback.The authors identify embodied AI and multi-agent collaboration as future application areas.

A Proof of Proposition 1

The proof establishes how policy entropy relates to score-function gradient magnitude and motivates EMPG’s composite objective. The resulting formulation combines entropy-based extrinsic scaling with an intrinsic reward for predictable future states.

  • For a softmax policy, the expected score-function gradient norm is linked monotonically to policy entropy, so uncertain actions produce larger gradients than confident actions.The standard estimator weights each step uniformly by trajectory return even though score-function norms vary with entropy.
  • The standard sparse-reward policy-gradient estimator applies the trajectory return uniformly across all action steps.This uniform weighting leaves each step’s contribution coupled to its entropy-dependent score-function norm.
  • EMPG is formulated as gradient ascent on a composite objective combining a re-weighted extrinsic objective with an intrinsic objective favoring higher-certainty states.
  • The entropy-dependent weighting function acts as an adaptive learning rate that counteracts entropy-driven gradient magnitudes.Its state-dependent reweighting interpretation justifies the self-calibrating gradient-scaling update.
  • The Future Clarity Bonus rewards transitions leading to predictable future states and corresponds to the gradient of an intrinsic objective.The intrinsic reward is based on the entropy of the subsequent state’s policy.
  • Combining both components yields the full EMPG gradient and a principled optimization method that reshapes learning toward salient updates and robust solution paths.

C Experimental Settings

The appendix documents experimental settings, hardware configurations, and hyperparameters for the three benchmark experiments. WebShop/ALFWorld and Deep Search use separate settings because their training frameworks and environments differ.

  • Experimental settings cover hardware configurations and hyperparameter choices across the three main benchmarks.The appendix separates WebShop/ALFWorld from Deep Search because their frameworks and environments differ.

C.1 WebShop and ALFWorld Experiments

WebShop and ALFWorld experiments use the Verl-Agent framework for scalable long-horizon reinforcement learning with customizable per-step agent inputs. Baselines were retrained under matched hardware, seeds, and settings for comparability.

  • WebShop and ALFWorld experiments use Verl-Agent, an extension of veRL designed for long-horizon, multi-turn LLM-agent reinforcement learning.The framework supports customizable per-step inputs, history management, memory modules, and multiple RL algorithms.
  • All baselines were re-trained with the same hardware, seeds, and settings to ensure strict comparability.The experiments used A100 GPUs because the original H200 setup was unavailable; model sizes received different GPU allocations.
  • Table 3 summarizes the key hyperparameters for the WebShop and ALFWorld experiments.

C.2 Deep Search Experiments

Deep Search training uses a proprietary RL framework with search and web-viewing tools, supported by a curated 17,000-instance dataset filtered for tool-dependent, challenging questions.

  • Deep Search training used Bing Search and a web viewer capable of reading and summarizing long articles.
  • The training dataset contained 17,000 instances filtered from WebWalker, HotpotQA, 2WikiMultiHopQA, NaturalQuestions, and TriviaQA.
  • Questions answerable directly by sampled results were removed so agents would learn to use search tools rather than rely on memorized answers.
  • Instances with consistently correct sampled search workflows were removed to focus reinforcement learning on harder cases and improve training efficiency.
  • Key reinforcement-learning hyperparameters for Deep Search are reported in Table 4.

D Analysis of Learning Dynamics

Learning curves show EMPG-enhanced agents continuing to improve while baseline agents plateau, producing higher final success rates across WebShop and ALFWorld settings.

  • Baseline agents stagnated early at a performance ceiling, whereas EMPG agents sustained learning momentum during training.
  • Figure D.1 compares EMPG-enhanced agents with GRPO and DAPO baselines on WebShop and ALFWorld learning curves.
  • EMPG-enhanced agents achieved significantly higher final success rates across all WebShop and ALFWorld settings than baseline agents.

E Algorithm Implementation Details

EMPG computes step-level entropy, uses normalized uncertainty to modulate advantages, adds a future-clarity bonus, and normalizes the resulting advantage for policy updates.

  • Algorithm 2 provides a PyTorch-style implementation of EMPG advantage calculation using trajectory responses, entropy values, advantages, and hyperparameters.
  • EMPG first identifies assistant response segments and averages token-level entropies to compute one entropy value H_t for each agent action step.
  • Collected step entropies are min-max normalized across the batch, then used to compute self-calibrating scaling and future-clarity components.
  • Each outcome-based advantage is scaled by g(H_t) and augmented by the future-clarity term ζ · g′(H_{t+1}) when a next step exists.
  • The resulting modulated advantages are normalized to zero mean across the batch before computing the policy gradient.
Loading 2509.09265v1…