PaperlayerView exact revision on arXiv

Source-linked AI summary

EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning

Wujiang Xu, Wentian Zhao, Zhenting Wang, Yu-Jhe Li, Can Jin, Mingyu Jin, Kai Mei, Kun Wan, Dimitris N. Metaxas

arXiv:2509.22576v2cs.LGcs.CL

TL;DR

Multi-turn agent training with sparse terminal rewards makes exploration and exploitation difficult to balance when turns share policy parameters. EPO combines trajectory-aware entropy regularization, historical smoothing, and adaptive phase weighting. It reports improvements on ScienceWorld and ALFWorld, and concludes that multi-turn agents require different entropy control from traditional RL.

  • Problem

    Multi-turn episodes can span 30+ turns with rewards only at task completion, while shared policy parameters prevent independent entropy control across turns.

    Source: §1 · p. 1

  • Method

    EPO combines trajectory-aware entropy computation, entropy smoothing against historical averages, and adaptive phase-based weighting.

    Source: §4.1 · p. 4, §7 · p. 8

  • Results

    EPO reports up to 152% performance improvement on ScienceWorld and 19.8% on ALFWorld.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that multi-turn LLM agent training requires fundamentally different entropy control than traditional RL.

    Source: §7 · p. 9

    Extending EPO to multi-turn vision-language agents remains future work, where visual and textual entropy may require different bounds.

    Source: E. Limitation and Future Work · p. 30

Abstract

from arXiv · show

Training LLM agents in multi-turn environments with sparse rewards, where completing a single task requires 30+ turns of interaction within an episode, presents a fundamental challenge for reinforcement learning. We identify a critical failure mode unique to this setting: the exploration-exploitation cascade failure. This cascade begins with early-stage policy premature convergence, where sparse feedback causes agents to commit to flawed, low-entropy strategies. Subsequently, agents enter late-stage policy collapse, where conventional entropy regularization becomes counterproductive, promoting chaotic exploration that destabilizes training. We propose Entropy-regularized Policy Optimization (EPO), a general framework that breaks this failure cycle through three synergistic mechanisms: (1) adopting entropy regularization in multi-turn settings to enhance exploration, (2) an entropy smoothing regularizer that bounds policy entropy within historical averages to prevent abrupt fluctuations, and (3) adaptive phase-based weighting that balances exploration and exploitation across training. Our analysis justifies that EPO guarantees monotonically decreasing entropy variance while maintaining convergence. EPO achieves up to 152% performance improvement on ScienceWorld and up to 19.8% on ALFWorld. Our work demonstrates that multi-turn sparse-reward settings require fundamentally different entropy control than traditional RL, with broad implications for LLM agent training.

1. Introduction

Multi-turn agents face an exploration–exploitation challenge because shared policy parameters couple turns while sparse rewards provide no intermediate correction. EPO combines trajectory-aware entropy regularization, smoothing, and adaptive weighting; it reports up to 152% performance improvement on ScienceWorld.

1. Introduction

  • Multi-turn episodes can span 30+ turns with rewards only at task completion, leaving early actions without corrective feedback.This makes exploration–exploitation balance a critical challenge compared with single-turn methods that receive immediate rewards.Source: §1 · p. 1
  • Shared policy parameters prevent independent per-turn entropy control, while sparse rewards make training dynamics unstable; the paper calls this the exploration-exploitation cascade failure.Figure 1 shows per-turn weighting leaves early- and late-turn entropy curves nearly identical, while PPO fluctuates with stagnant rewards and EPO maintains stable entropy and improving rewards.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • EPO combines multi-turn entropy regularization, historical-entropy smoothing, and adaptive phase-based weighting to stabilize exploration and support convergence.Its weighting applies stronger smoothing early, relaxes it mid-training, and maintains regularization later.Source: §1 · p. 2
  • Up to 152% performance improvement on ScienceWorld is reported for EPO, alongside more stable training dynamics.The framework is described as transforming unstable sparse-reward scenarios into smoothly converging optimization.Source: §1 · p. 2

2. Related Work

Prior LLM reinforcement-learning work addresses entropy collapse and proposes entropy-control methods, while multi-step agent research tackles sparse rewards through several training paradigms. The authors identify multi-turn entropy dynamics as an overlooked issue in that agent literature.

2.1. Reinforcement Learning for LLMs

2.2. Reinforcement Learning for LLM Agents

  • Multi-step agent research includes hierarchical RL, autonomous learning, off-policy Q-learning, supervised fine-tuning, multi-turn GRPO, and RLVMR’s intermediate meta-reasoning rewards.The authors argue these methods overlook entropy dynamics: shared parameters couple early-turn exploration and late-turn exploitation under sparse rewards.Source: §2.2 · p. 3

3. Preliminary

The preliminary section outlines on-policy optimization and its clipped PPO and group-relative GRPO variants, then formalizes multi-turn tasks as trajectories with sparse terminal rewards. Because all turns share that outcome and losses accumulate across turns, credit assignment spans the trajectory.

3.1. On-policy Optimization

  • On-policy optimization improves a policy using trajectories sampled from that current policy; PPO stabilizes updates with a clipped surrogate, and GRPO uses group-relative advantages.The group-relative advantage normalizes returns using the mean and standard deviation within a group, with a small constant for numerical stability.Source: §3.1 · p. 3

3.2. Problem formulation

  • In the ALFWorld and SciWorld settings, a task has sparse binary reward only at the final turn, and all turns share that outcome, creating a credit-assignment challenge.The formulation assumes an undiscounted return and accumulates losses across all turns before parameter updates.Source: §3.2 · pp. 3–4

4. Methodology

EPO adapts entropy regularization to multi-turn trajectories by aggregating entropy across turns and trajectories. It smooths entropy relative to historical averages and varies smoothing strength across training phases.

4.1. Entropy Regularization

  • EPO computes entropy across all turns within each trajectory and averages it across the batch, rather than computing it per step.This trajectory-level aggregation is designed to capture temporal structure under shared policy parameters.Source: §4.1 · p. 4
  • Token-level entropy is computed from the model’s vocabulary distribution, and the trajectory entropy aggregates token entropies across turns and trajectories.The aggregation uses batch size, turns per trajectory, token positions, and each turn’s sequence length.Source: §4.1 · p. 4

4.2. Entropy Smoothing Regularizer

  • The smoothing regularizer anchors current entropy to the historical mean and penalizes token entropies outside an acceptable corridor around that reference.The history stores average entropy across trajectories for previous RL steps; the corridor is set by κ_l and κ_r.Source: §4.2 · p. 4
  • The adaptive coefficient β_k applies stronger smoothing early, relaxes it around mid-training, and retains regularization later for smooth convergence.The combined objective uses λ for overall entropy regularization and β_k for phase-dependent smoothing strength.Source: §4.2 · p. 4

5. Experiments

Across ScienceWorld and ALFWorld, EPO improves benchmark performance and stabilizes training. The experiments assess its results, components, and alternative entropy-control strategies.

5.1. Experiments Setup

  • The experiments use sparse-reward ScienceWorld and ALFWorld, evaluate IID and OOD success, and compare EPO with prompting, imitation, and RL baselines.The authors implement PPO, GRPO, and EPO across three random seeds; other baseline results are sourced from the RLVMR paper.Source: §5 · p. 4, §5.1 · pp. 5–6, Table 1 · p. 5

5.2. Performance Comparison

  • PPO+EPO improves ScienceWorld IID Succ. by 152.1%, while GRPO+EPO improves ALFWorld IID performance by 19.8%.The comparison uses averaged success rate (Succ.) to reduce variance across individual runs.Source: Table 1 · p. 5, §5.2 · p. 6
  • EPO reaches ScienceWorld IID and OOD success rates above 0.8 within 40 steps; baselines struggle to exceed 0.4 after 100 steps.On ALFWorld, OOD baselines often fall below 0.2 while EPO variants remain above 0.4.Source: §5.2 · p. 6, Figure 2 · p. 6

5.3. Ablation Study

  • Removing entropy smoothing leaves ScienceWorld EPO-Base at 0.5–0.6 success versus EPO’s 0.8–1.0; constant β slows ALFWorld’s early convergence.The constant-β variant has comparable final ALFWorld performance but increased training variance.Source: Figure 3 · p. 7, §5.3 · p. 7

5.4. Model Study

  • Turn-wise decay suppresses exploration and drops late-step entropy, while decoupled EPO outperforms entropy-based advantage reshaping on ScienceWorld.The cumulative entropy window adapts quickly early and stabilizes later; the authors say a sliding window could conflict with adaptive βk. EPO reaches near-perfect success while EA plateaus at 0.5–0.6.Source: §5.4 · pp. 7–8

6. Theoretical Analysis

The analysis frames EPO as constrained optimization that limits entropy drift across turns. It relates that stability to convergence and identifies conditions under which the corridor adds no stability cost.

Theoretical Analysis

  • EPO’s smoothing loss acts as an exact penalty for violating the entropy corridor, with adaptive weighting enforcing the constraint more strongly early and relaxing it later.The analysis connects this schedule to curriculum-style enforcement as entropy stabilizes.Source: §6 · p. 8
  • Under the corridor, cumulative entropy deviation is O(T), versus O(T^2) in the standard-regularization worst case; the gap grows with dialogue length.The analysis attributes compounding drift to entropy deviations affecting later turns’ contexts, and links stability to lower gradient variance and trajectory diversity.Source: §6 · p. 8
  • When the optimal policy’s entropy lies within the corridor, EPO’s stability cost is zero and it matches standard bounds while retaining stability guarantees.The paper says sparse-reward multi-turn tasks typically require moderate-entropy optimal policies, satisfying this condition.Source: §6 · p. 8

7. Conclusion

EPO addresses the exploration-exploitation cascade failure with trajectory-aware entropy computation, entropy smoothing, and adaptive phase-based weighting. The authors report improved benchmark performance and more stable training dynamics.

7. Conclusion

  • EPO combines trajectory-aware entropy computation, entropy smoothing regularization, and adaptive phase-based weighting.The authors say these mechanisms prevent severe entropy oscillations that destabilize training under shared policy parameters.Source: §7 · pp. 8–9
  • 152% on ScienceWorld and 19.8% on ALFWorld are the reported performance improvements with EPO.The authors also report smoother convergence and prevention of severe entropy oscillations under shared policy parameters.Source: §7 · p. 9
  • The authors conclude that multi-turn LLM agent training requires entropy control different from traditional RL.They describe this as opening new directions for training agentic LLMs.Source: §7 · p. 9

Impact Statement

The appendix describes a finite-horizon dialogue setup and assumptions for entropy smoothing, then interprets the EPO objective as an adaptive penalty for violating an entropy corridor. The Impact Statement frames the contribution as methodological work on training dynamics and calls for standard responsible AI practices in deployment.

Impact Statement

  • The Impact Statement presents the work as improving multi-turn LLM-agent training stability and advises standard responsible AI practices when deploying agents trained with the method.It notes potential misuse without safeguards and says the work focuses on training dynamics rather than fundamentally new capabilities.Source: Impact Statement · p. 10

A.1. Notation and Setup

  • Dialogue generation is modeled as a finite-horizon MDP, with conversation histories as states and generated tokens as actions.The setup names the initial query s0, maximum horizon H, turn count T, and per-turn token count Lt.Source: §A.1 · p. 13

A.2. Assumptions

  • The assumptions use a fixed positive entropy reference, require policy entropy to remain in a corridor, and bound the smoothing weight to preserve concavity.The stated bound is α < 0.5; the experiments use α = 0.1.Source: §A.2 · p. 13

A.3. Key Definitions

  • The appendix names stability cost, concentrability coefficient, EPO-optimal policy, and cumulative entropy deviation as formal quantities.The supplied passages give the definition headings but not the full definitions.Source: §A.3 · pp. 13–14

A.4. Preliminary Lemmas

  • The corridor penalty is zero exactly inside the entropy corridor, and its gradient pushes entropy toward that corridor when outside it.Within the corridor, the stated penalty gradient is zero.Source: §A.4 · p. 14

A.5.1. EXACT PENALTY INTERPRETATION

  • The EPO objective is an exact penalty formulation for the entropy-corridor constraint, with an adaptive penalty weight ρk = λβkα.The appendix characterizes the schedule as weaker early for exploration and stronger later for exploitation.Source: §6 · p. 8, §A.5.1 · p. 15

A.5.2. MULTI-TURN STABILITY

This section examines EPO’s entropy stability in multi-turn settings, including its cumulative deviation bound, possible relevance to convergence, and conditions under which its stability cost can vanish, alongside benchmark and evaluation contexts. It also describes two baseline approaches and reports negligible EPO training overhead under the tested hardware configurations.

A.5.2. MULTI-TURN STABILITY

  • Under the corridor constraint, EPO limits cumulative entropy deviation to O(T), versus O(T^2) in the worst case without corridor enforcement for standard entropy regularization; the gap is 5.5× at 10 turns and 10.5× at 20.Relevant conditions include sparse rewards observed only at dialogue end and entropy-collapse dynamics with positive feedback; EPO’s advantage grows with dialogue length.Source: §A.5.2 · pp. 15–16

A.5.3. CONNECTION BETWEEN ENTROPY STABILITY AND CONVERGENCE

  • Entropy stability may improve convergence by limiting importance-weight variance and preserving trajectory diversity, and EPO is reported to achieve lower gradient variance and faster convergence empirically.The authors explicitly do not formally prove a bound relating cumulative entropy deviation to suboptimality.Source: §A.5.3 · p. 16

A.5.4. PERFORMANCE BOUND

  • Under mild assumptions, Proposition 6.3 states an EPO performance bound, but the supplied passage does not include its expression.Source: §A.5.4 · p. 16

A.5.5. ZERO STABILITY COST CONDITION

  • When optimal policies have moderate entropy within the corridor, EPO’s stability cost is zero and it matches standard bounds; for α = 0.1, the optimization factor is approximately 1.25.A narrow corridor corrects deviations quickly but risks over-constraining learning, whereas a wide corridor lowers stability cost but may fail to prevent cascade failure.Source: §A.5.5 · pp. 17–18

A.6. Comparison with Standard Entropy Regularization

  • For α = 0.1 and βk ∈ [1, 2], the multi-turn stability advantage outweighs EPO’s per-step overhead by T ≥ 2, with a 5× to 10× reduction in cumulative entropy deviation for T ≥ 5.Source: §A.6 · p. 19

B.1.1. BENCHMARK

  • ScienceWorld tests open-ended scientific reasoning in a text-based lab, while ALFWorld tests embodied reasoning and long-horizon action planning in simulated homes.Source: §B.1.1 · p. 19

B.1.2. EVALUATION SETTING

  • Evaluation reports both maximum success across seeds and converged success; ALFWorld convergence is averaged from step 125, while ScienceWorld uses per-run windows that indicate earlier EPO convergence.Training curves are smoothed with a running-average window of 10.Source: §B.1.2 · pp. 19–20

B.1.3. BASELINES

  • The baselines span prompting, trajectory imitation and agent platforms, general RL, and agent-specific RL; PPO uses terminal-reward credit assignment, while GRPO evaluates trajectories at episode level.The supplied passages also characterize ReAct as single-pass prompting without parameter optimization, and SFT and AgentEvol as dependent on demonstration trajectories.Source: §B.1.3 · pp. 20–21
  • GIGPO (Feng et al., 2025) uses revisited anchor states for micro-level credit assignment, while RLVMR (Zhang et al., 2025) uses teacher-annotated meta-reasoning tags for process-level supervision.Both approaches are guided by episode-ending reward or teacher-provided annotations, respectively.Source: §B.1.3 · p. 21

B.1.4. IMPLEMENTATION DETAILS

  • EPO-enhanced PPO and GRPO training times were effectively identical to their respective baselines on H100s and A100s.On SciWorld, PPO took approximately 23 hours on H100s and 31 hours on A100s; GRPO took 16 and 20 hours, respectively. On ALFWorld, PPO took 23 and 31 hours, while GRPO took 18 and 25 hours.Source: §B.1.4 · p. 21

B.2. Performance Comparison

EPO improves training stability and validation performance across ScienceWorld and ALFWorld. The gains include stronger ScienceWorld success rates and more consistent ALFWorld OOD performance.

B.2. Performance Comparison

  • GRPO+EPO converges around step 60 with higher peak rewards than GRPO, while ALFWorld PPO+EPO accumulates rewards more consistently with fewer oscillations.The paper attributes the improved stability to entropy regularization guiding exploration toward productive policy regions.Source: §B.2 · p. 22
  • ScienceWorld GRPO+EPO exceeds 0.8 success within 40 steps, while GRPO struggles to surpass 0.6; ALFWorld PPO+EPO OOD success stays above 0.6 versus PPO’s frequent drops below 0.4.ALFWorld IID results are comparable between PPO+EPO and PPO.Source: §B.2 · p. 22
  • Across both environments, EPO variants show tighter confidence intervals and more reliable convergence, with stability particularly benefiting OOD scenarios.The reported results indicate simultaneous improvements in generalization capability and convergence reliability.Source: §B.2 · p. 22

B.3. Ablation Study

The ablations examine entropy smoothing, adaptive weighting, scheduled entropy decay, and comparison with entropy-based advantage shaping. Smoothing and EPO outperform their respective alternatives, while the value of smoothing varies with environment feedback structure.

B.4. Model Study

  • Figure 5 contrasts training rewards and IID and OOD success for GRPO versus GRPO+EPO on ScienceWorld and PPO versus PPO+EPO on ALFWorld.The caption reports improved training stability and performance across both evaluation settings.Source: Figure 5 · p. 23

B.3. Ablation Study

  • Adaptive β_k weighting strengthens stabilization when oscillations are detected and relaxes constraints during natural convergence.The paper reports consistent smoothing improvements across GRPO and PPO in sparse settings.Source: §B.3 · pp. 22–23
  • On ScienceWorld, EPO reaches 7–8 rewards and 0.7–0.85 success rates, versus EPO-Base near 2 rewards and 0.6 success; on ALFWorld, both variants finish near 12.5 reward and 0.8 success, with EPO converging 20 episodes faster.The reported ScienceWorld improvement is 40–50%; the authors describe smoothing as beneficial for speed but not essential for ALFWorld convergence.Source: §B.3 · p. 22, Figure 6 · p. 24

B.4.1. STUDY OF ENTROPY REGULARIZATION

  • PPO+EPO-Decay underperforms PPO+EPO-Base across episodic reward and IID and OOD success rates; its schedule prematurely suppresses exploration in early episode turns.Figure 7 compares episodic rewards and success rates, and examines entropy in the first versus last 10 tokens.Source: §B.4.1 · p. 24, Figure 7 · p. 25

B.4.2. STUDY OF ENTROPY-SHAPED ADVANTAGE

  • PPO+EPO reaches near-optimal success of almost 1.0 on ScienceWorld, while PPO+EA plateaus at 0.5–0.6.The paper contrasts EPO’s direct entropy objective and historical smoothing with EA’s detached entropy term and hard-clipped advantage bonus.Source: §B.4.2 · p. 25, Figure 8 · p. 26

C. Core Implementation Pseudocode

The appendix presents pseudocode for calculating EPO’s training loss, then documents the system prompts used for ALFWorld and ScienceWorld agents. Each environment has prompt versions with and without history.

C. Core Implementation Pseudocode

  • Algorithm 2 combines the base multi-turn policy loss, entropy regularization, and entropy smoothing into the EPO objective for a single policy update.The pseudocode takes the policy, batch, current epoch, and hyperparameters as inputs and obtains token-level log probabilities and entropy in a forward pass.Source: C. Core Implementation Pseudocode · p. 27
  • The loss calculation forms a historical average entropy, generates a token-wise penalty mask, calculates the smoothing loss and dynamic β_k, then combines the entropy and smoothing terms.The final EPO loss combines these terms with the base multi-turn loss.Source: C. Core Implementation Pseudocode · p. 27

D. System Prompts

  • System prompts are provided for both environments with and without history; placeholders such as {current_observation} are filled with environment-specific information at runtime.The appendix says the history versions include previous actions and observations, while the initial-turn versions omit that context.Source: D. System Prompts · p. 28, §D.1 · p. 28

D.1. ALFWorld Prompts

  • ALFWorld prompts present the task, current observation, admissible actions, and—when history is included—recent observations and corresponding actions.The prompts ask the agent to reason in <think> tags and provide an admissible action in <action> tags.Source: §D.1 · p. 28

D.2. ScienceWorld Prompts

  • ScienceWorld prompts provide the task, current observation, and available actions; history prompts also include prior steps, recent observations, and actions.The history-free prompt is identified as the initial-turn version.Source: §D.2 · pp. 29–30

E. Limitation and Future Work

The paper identifies limited use of memory systems as an avenue for extending EPO and discusses future application to multi-turn visual agents. It also specifies that LLMs supported editing and minor coding tasks, not research decisions or results.

E. Limitation and Future Work

  • EPO uses historical entropy for regularization but does not explicitly store or retrieve successful past trajectories for reuse.The authors suggest memory could help recall behavioral patterns, especially from rare positive-reward trajectories.Source: E. Limitation and Future Work · p. 30

F. Use of Large Language Models

  • The authors used LLMs for manuscript editing and minor boilerplate coding, not to generate experimental results, annotations, ground truth, or methodological decisions.They state that authors reviewed and tested generated outputs and take sole responsibility for the paper.Source: F. Use of Large Language Models · p. 30
Loading 2509.22576v2…