PaperlayerView exact revision on arXiv

Source-linked AI summary

Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning

Woongyeong Yeo, Minki Kang, Chanuk Lee, Sangwoo Park, Jinheon Baek, Sung Ju Hwang

arXiv:2609.33781v1cs.LGcs.AIcs.CL

TL;DR

Outcome-level RLVR rewards do not distinguish the contributions of individual reasoning decisions, while existing finer-grained methods can require extra resources or information. EAPO uses advantage-sign-dependent entropy to redistribute credit across tokens; evaluations report stronger overall performance and broader coverage and diversity.

  • Problem

    RLVR evaluates whole trajectories despite differing contributions by individual decisions, and existing finer-grained credit-assignment approaches can require auxiliary models, additional sampling, or privileged information.

    Source: §1 · p. 1

  • Method

    EAPO couples normalized policy entropy with response-advantage sign to reinforce high-entropy successful decisions and penalize low-entropy failed decisions while attenuating penalties on uncertain alternatives.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Across reasoning tasks and base and reasoning backbones, EAPO achieves the best overall performance in average accuracy and problem coverage, with analyses reporting broader coverage and greater answer diversity.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The findings support asymmetric credit assignment as a way to reinforce surprising success and correct repeated failure while sustaining exploration.

    Source: §7 · p. 10

    Entropy-guided allocation reflects learned preferences and confidence, so its reliability depends on how well model uncertainty aligns with individual decisions' contributions to the final outcome.

    Source: Limitations and Future Directions · p. 10

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.

1. Introduction

The introduction identifies a gap in token-level credit assignment: existing approaches can require extra models, sampling, or privileged information, while shared entropy preferences can penalize uncertain failed decisions. EAPO uses advantage-sign-dependent entropy weighting, and the paper reports stronger overall performance and exploration outcomes.

1. Introduction

  • Finer-grained credit assignment is motivated because outcome-level rewards evaluate whole trajectories despite differing contributions by individual decisions.Existing approaches to obtaining token-level feedback can require auxiliary models, additional sampling, or privileged information.Source: §1 · p. 1
  • A shared entropy preference can concentrate failure penalties on uncertain positions that may retain alternative paths to recovery.This motivates treating uncertainty differently under reinforcement and penalization.Source: §1 · p. 2
  • EAPO couples policy entropy with advantage sign, reinforcing high-entropy decisions after success and penalizing low-entropy decisions after failure.It attenuates penalties at uncertain positions in unsuccessful responses to preserve alternative reasoning paths.Source: §1 · p. 2
  • Across reasoning tasks and base and reasoning backbones, EAPO achieves the best overall performance in average accuracy and problem coverage.The analyses also report broader problem coverage across test-time sampling budgets and greater answer diversity on challenging problems.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe response-level reward advantages in GRPO and policy entropy as uncertainty over the next-token distribution. They also connect entropy tails with different word-use patterns.

2. Preliminaries

  • GRPO normalizes rewards for a sampled response group into response-level advantages, which are shared uniformly across tokens and provide no position-specific weighting.Thus, token decisions within a response receive equal attribution under this baseline formulation.Source: §2 · p. 2
  • Exploratory expressions are enriched in high-entropy positions in both correct and incorrect responses, while conclusion markers are enriched at low-entropy positions in incorrect responses.Figure 2 compares words with higher relative frequencies in the top and bottom 10% entropy tails of each response.Source: Figure 2 · p. 3
  • Policy entropy measures uncertainty over the full next-token distribution, unlike surprisal, which measures the unexpectedness of one sampled token.Higher entropy indicates greater uncertainty among continuations; lower entropy reflects more concentrated preferences.Source: §2 · p. 3

3. Surprising Success, Repeated Failure

Resampling analyses relate entropy to repeatability and recovery: high-entropy success is less likely to recur, while high-entropy positions in failed responses retain alternatives. Word-frequency patterns are consistent with exploratory behavior at uncertain positions.

3. Surprising Success, Repeated Failure

  • Resampling high-entropy windows in originally incorrect responses yields higher accuracy, indicating retained alternatives to recovery; high-entropy windows also have lower continuation overlap across outcomes.The authors use these observations to motivate preserving uncertain positions in failed responses while correcting low-entropy repeated failures.Source: §3 · pp. 3–4
  • Originally correct responses are less likely to succeed again from high- than low-entropy windows, while originally incorrect responses tend to fail again from low-entropy windows.The table compares continuation resampling by original outcome and window entropy; its Δr̄ is the reward difference for High − Low.Source: Table 1 · p. 3, §3 · p. 3
  • In 2,376 Qwen3-4B-Base responses, exploratory expressions are enriched at high-entropy positions in correct responses, while conclusion markers are more prevalent at low-entropy positions in incorrect responses.Exploratory expressions also appear at high-entropy positions in incorrect responses, consistent with retained alternatives in failed responses.Source: §3 · p. 4

4. Entropic Advantage Policy Optimization

EAPO redistributes response advantages across tokens using normalized policy entropy and advantage sign. Its theoretical interpretation relates this allocation to KL anchoring and analyzes how negative updates affect unchosen alternatives.

4. Entropic Advantage Policy Optimization

  • EAPO redistributes response advantages across completion tokens using policy entropy, assigning stronger reinforcement to uncertain positive-advantage decisions and stronger penalties to confident negative-advantage decisions.It derives token-level credit from the policy's own entropy and existing response advantage.Source: §4 · p. 4

4.1. Asymmetric Advantage Reweighting via Sign–Entropy Coupling

  • Shared percentile normalization within each rollout batch addresses entropy variation across responses and fluctuations within a response that could make absolute-entropy weighting noisy.The entropy values are evaluated under the old policy, over valid completion positions excluding prompts and padding.Source: §4.1 · p. 4
  • Signed uncertainty reverses the entropy preference by advantage sign: positive advantages favor high-entropy tokens, while negative advantages favor low-entropy tokens.Exponential weighting concentrates allocation according to κ, and within-response mean normalization redistributes rather than rescales the response-level advantage.Source: §4.1 · pp. 4–5
  • EAPO replaces the response-level advantage in the policy objective with token-level credit while leaving the rest of the optimization procedure unchanged.The method uses existing entropy and response advantage without auxiliary models, token-level supervision, or substantial additional computation.Source: §4.1 · p. 5

4.2. Theoretical Interpretation

  • The theoretical interpretation treats uniform token credit as a reference allocation and signed uncertainty as utility; increasing κ strengthens the entropy preference, while κ→0 recovers uniform credit.This operationalizes the allocation as KL anchoring transferred to token space.Source: §4.2 · p. 5
  • In a single-step logit model, stronger negative updates lower the sampled-token probability, reduce entropy among unchosen alternatives, and increase their KL distortion.Reducing penalty strength at a position limits this sharpening and distortion, supporting attenuated penalties at uncertain positions; Table 2 covers six mathematical benchmarks and base and reasoning models.Source: §4.2 · p. 5, Table 2 · p. 6

5. Experiments

Across mathematical and broader reasoning benchmarks, EAPO achieves the strongest overall results among the compared methods. Its training and exploration analyses show gains in coverage and answer diversity, while ablations support asymmetric entropy preferences for reinforcement and penalization.

5.1. Experimental Setup

  • The evaluation spans six competition-level mathematics benchmarks, with avg@32 and pass@32, and compares EAPO with the initial backbone and five RLVR methods.The study includes base and reasoning backbones; optimization-based methods use DAPO-Math-17k-Processed and a DAPO-style training configuration.Source: §5.1 · pp. 6–7

5.2. Main Results

  • 31.0% and 34.0% mean avg@32 with Qwen3-4B-Base and Qwen3-8B-Base, respectively, exceed the strongest entropy-based baselines by 5.6 and 4.3 percentage points.EAPO also has the highest mean avg@32 and pass@32 across all four backbones, and beats RLRT on both metrics for both base backbones.Source: §5.2 · p. 7
  • 34.89% and 43.02% macro-averaged avg@16 on Reasoning Gym improve on the respective best baselines by 1.46 and 3.02 percentage points.The results cover eight logical, spatial, and algorithmic tasks beyond the mathematical optimization problems.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Training Dynamics

  • EAPO maintains higher training rewards than baselines over most of training and longer responses as training progresses, but its final average entropy is below EntropyAdv and HAPO.Additional continuations do not close its accuracy gap at comparable mean response lengths; EAPO achieves stronger reasoning performance without a substantial entropy increase.Source: Figure 3 · p. 7, §5.3 · p. 8

5.4. Analysis of Exploration

  • EAPO achieves the highest pass@k across sampling budgets from 1 to 256, and at k=256 solves one problem that no baseline solves.EAPO also has higher normalized answer entropy and lower collision rate than other RLVR methods on challenging, low-accuracy problems, and the highest epistemic-marker frequency across six mathematics benchmarks.Source: §5.4 · pp. 8–9, Figure 4 · p. 8, Table 4 · p. 9

5.5. Ablation Studies

  • High-entropy reinforcement paired with low-entropy penalization achieves the highest avg@32 and pass@32; avg@32 exceeds uniform token credit by 6.52 percentage points.With high-entropy reinforcement fixed, shifting penalization from high to low entropy improves accuracy by 5.64 percentage points.Source: §5.5 · p. 9, Table 5 · p. 9
  • Larger κ generally improves overall performance but can make training less stable; the default κ=log 4 is selected based on observed training stability.As κ approaches zero, token allocation converges to uniform credit and performance drops.Source: §5.5 · p. 9, Table 6 · p. 9

6. Related Work

Prior work addresses finer-grained credit assignment and exploratory reasoning through learned signals, policy-based signals, and diverse exploration strategies. EAPO is distinguished by coupling policy entropy with advantage sign without auxiliary models, extra rollouts, or privileged information.

Fine-Grained Credit Assignment

  • Fine-grained RLVR credit signals have come from learned process rewards, intermediate value estimates, teacher- or oracle-conditioned likelihoods, and policy-internal signals.Other methods reweight token advantages using selected-token probabilities or surprisal.Source: §6 · p. 10

RLVR for Exploratory Reasoning

  • Exploration methods allocate more samples to difficult problems, revisit promising states, branch over continuations, encourage diversity, regulate policy concentration, or preserve low-probability alternatives.Token-level approaches also use reversed teacher signals or focus updates on high-entropy tokens.Source: §6 · p. 10
  • EAPO couples policy entropy with the advantage sign for asymmetric credit assignment, without auxiliary models, additional rollouts, or privileged information.This contrasts with prior approaches that rely on those additional resources.Source: §6 · p. 10

7. Conclusion

EAPO uses advantage-sign-dependent entropy weighting to reinforce uncertain successful decisions and penalize confident unsuccessful ones. Evaluations report consistent gains across reasoning tasks and models, with broader coverage and response diversity.

Conclusion

  • EAPO redistributes response advantage across tokens, reinforcing uncertain decisions in positive-advantage responses and penalizing confident decisions in negative-advantage responses.Penalties are attenuated on uncertain alternatives.Source: §7 · p. 10
  • Evaluations show consistent overall improvements over exploration-oriented RLVR baselines across base and reasoning models.The analyses also report broader problem coverage and increased response diversity.Source: §7 · p. 10
  • The findings highlight asymmetric credit assignment as a way to reinforce surprising success and correct repeated failure while sustaining exploration.Source: §7 · p. 10

Limitations and Future Directions

EAPO uses policy entropy for token-level credit assignment without auxiliary models, additional information, or extra compute, but its reliability depends on whether uncertainty tracks decisions’ contributions. The analysis focuses on alternative continuations from one reasoning state and leaves combining findings across states or trajectories unexplored.

Limitations and Future Directions

  • Entropy-guided credit assignment may be sensitive to the model’s learned preferences and biases when uncertainty does not align with decisions’ actual contributions.The paper notes that improved reasoning performance does not itself guarantee reliable or safe outputs.Source: Limitations and Future Directions · p. 10
  • The mechanism analysis covers alternative continuations from a given reasoning state, not explicit combination of findings across states or trajectories.The paper identifies iterative discovery settings as a direction for future work.Source: Limitations and Future Directions · p. 10

Ethics Statement

The paper calls for safeguards when applying EAPO beyond the reasoning tasks studied. Its related-work discussion distinguishes EAPO by conditioning token-level credit allocation on both entropy and response advantage sign.

Ethics Statement

  • Models trained with EAPO may retain underlying model and data biases, and improved reasoning does not guarantee reliable or safe outputs.The authors encourage safeguards beyond the studied reasoning tasks and in real-world applications.Source: Ethics Statement · p. 10

A. Extended Related Work

  • Earlier uncertainty-guided approaches weight or filter tokens without distinguishing reinforcement from penalization, whereas EAPO reverses entropy preference by advantage sign.EAPO also attenuates penalties at uncertain positions to preserve exploratory alternatives.Source: A. Extended Related Work · p. 15
  • EAPO redistributes full-vocabulary entropy-based advantage across token positions while preserving within-response mean, without privileged context or additional teacher passes.The related-work discussion contrasts this with outcome-conditioned methods that may leave token credit uniform or require extra teacher computation.Source: A. Extended Related Work · p. 15

B.1. Additional Details on the Resampling Experiment

This material details the resampling and benchmark protocols, shared training settings, and an entropy-versus-surprisal ablation. The reported analyses describe window selection, benchmark coverage, and the distinct training-reward behavior of the two uncertainty signals.

B.1. Additional Details on the Resampling Experiment

  • The resampling experiment selects entropy-contrasting non-overlapping 32-token windows in correct and incorrect responses, then samples 64 continuations from each fixed prefix.It uses moderately difficult problems with 6–14 correct responses among 32 attempts and measures continuation overlap and final-answer accuracy.Source: §B.1 · p. 16
  • Selected high- and low-entropy windows start near the response midpoint on average, with neither consistently earlier than the other.Table 7 reports mean and standard deviation of starting positions as percentages of response length.Source: Table 7 · p. 16, §B.1 · p. 16

B.2. Benchmark Details

  • The mathematical evaluation covers 297 problems across six benchmarks, samples 32 responses per problem, and reports avg@32 and pass@32 with equal benchmark weighting.The set includes AIME24/25/26, HMMT26, AMC23, and the level-5 subset of MATH500.Source: §B.2 · p. 16
  • The out-of-distribution evaluation uses eight Reasoning Gym tasks, with 50 problems and 16 responses per task, and reports equally weighted macro-average avg@16.Tasks cover logical, spatial, and algorithmic reasoning, with specified settings for graph coloring, mini sudoku, and zebra puzzles.Source: §B.2 · p. 16

B.3. Training Hyperparameters

  • Shared training uses a DAPO-style configuration on DAPO-Math-17k-Processed, with LoRA and AdamW for optimization-based methods unless otherwise stated.Training uses TRL and vLLM, and all experiments run on a single NVIDIA H200 GPU; Table 8 summarizes shared hyperparameters.Source: §B.3 · p. 16, Table 8 · p. 17

C.1. Entropy vs. Surprisal

  • Entropy-based EAPO continues improving training reward, while its sampled-token surprisal variant declines after a few steps and eventually collapses to zero reward.The authors suggest surprisal can conflate rare sampled tokens with uncertain decisions, destabilizing credit allocation.Source: Figure 5 · p. 18, §C.1 · p. 18

C.2. Extended Ablation on Sign–Entropy Coupling

Across these additional analyses, EAPO’s reported advantage persists across model sizes and fine-tuning strategies, while ablations support its asymmetric entropy preferences. Matched-token-budget and multi-seed analyses further examine response length and consistency.

C.2. Extended Ablation on Sign–Entropy Coupling

  • With high-entropy reinforcement fixed, shifting penalties from high to low entropy improves avg@32 by 4.25 percentage points; with low-entropy penalties fixed, shifting reinforcement from low to high improves avg@32 by 6.22 percentage points.The default configuration (b+ = +1, b− = −1) achieves the highest avg@32 and pass@32 among nine combinations.Source: Table 9 · p. 18, §C.2 · p. 18

C.5. LoRA vs. Full Fine-Tuning

  • LoRA and full fine-tuning differ by less than one percentage point on average in macro-averaged avg@32, and EAPO remains best under both strategies.Its margin over the strongest baseline exceeds the shifts between fine-tuning strategies.Source: Table 10 · p. 19, §C.5 · p. 20

C.3. Performance under Matched Token Budgets

  • EAPO reaches 20.2% avg@32 at 5.3k tokens, while RLRT peaks at 15.7% with approximately 6.2k tokens under budget forcing.The authors report that extra continuations do not necessarily improve exploration and that most approaches lose accuracy as responses extend further.Source: §C.3 · p. 19

C.4. Scaling with Model Size

  • Across Qwen3-Base models from 1.7B to 14B parameters, EAPO has the highest mean macro-averaged avg@32 across six mathematical benchmarks at every scale.Performance improves with model size, while relative trends among methods remain broadly consistent.Source: §C.4 · p. 19

C.6. Performance across Random Seeds

  • Across three random seeds, EAPO maintains higher mean training reward than GRPO, EntropyAdv, and HAPO over most of training, with a particularly pronounced final-stage gap.The authors report that the separation is substantial relative to observed variation across seeds.Source: Figure 8 · p. 20, §C.6 · p. 20

D. Proofs and Additional Theoretical Analysis

The appendix develops theoretical rationales for EAPO’s entropy-guided allocation and penalty attenuation, then illustrates surprising success and repeated failure in a MATH500 example.

D.1. KL-Regularized Uncertainty Allocation

  • The KL-regularized uncertainty allocation has a unique global maximizer, corresponding to the exponential allocation in Equation (4).The allocation favors signed uncertainty while anchoring credit to a uniform reference.Source: §D.1 · p. 21

D.2. Concentration of Unchosen Alternatives

  • For a single negative logit update with finite logits, increasing update strength lowers sampled-token probability, does not increase alternative entropy, and does not decrease KL divergence.When all alternatives start with the same probability, their entropy and KL divergence remain unchanged.Source: §D.2 · pp. 21–22

D.3. Implications for Entropy-Guided Allocation

  • In a binary decision model, expected feedback information rises with entropy on success but falls with entropy on failure, matching EAPO’s asymmetric allocation.The analysis also shows that EAPO weights feedback information above uniform allocation when normalized entropies are nonuniform and κ > 0.Source: §D.3 · pp. 22–23

E. Example of Surprising Success, Repeated Failure

  • In 32 rollouts, 28 AM–GM attempts failed while one of four Lagrange-multiplier attempts succeeded on the MATH500 problem.Splitting 4y² into 2y² + 2y² makes the AM–GM approach reach the correct minimum of 16; the rare Lagrange-multiplier success also reaches it.Source: E. Example of Surprising Success,… · p. 24
Loading 2609.33781v1…