Source-linked AI summary
Correlation-Aware Exploration for Long-Trace Reasoning
Junxing Hu, Zicheng Zhang, Zhiqian Zhang, Ai Han, Yifeng Zhang, Pengzhang Liu, Ling Li
TL;DR
Long-trace GRPO reasoning faces policy collapse, while standard entropy bonuses repeatedly push probability toward the low-probability vocabulary tail and oppose entropy reduction. Info-GRPO correlates latent seeds with whole reasoning trajectories, and experiments report consistent gains over GRPO baselines across benchmarks and model architectures.
Problem
In long-trace reasoning, GRPO can collapse onto a narrow set of strategies, while token-level entropy bonuses accumulate pressure on low-probability tokens and are difficult to tune.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Latent–trajectory correlation carries diversity across reasoning paths while allowing low-entropy, confident policies within each path.
Source: §1 · p. 2, §5 · p. 5The longest evaluated trace budget is 8,192 tokens (4,096 for Qwen2.5); performance at 32K or 128K budgets remains unverified.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse. Entropy regularization is the standard remedy, but it works by pushing probability mass away from the few tokens that carry it and toward the vast low-probability tail of the vocabulary, and it applies this force once for every token of the reasoning trace. This puts the bonus in direct conflict with the entropy decay that convergence requires, and makes its coefficient difficult to set in the long-trace regime. To resolve this tension, we propose Info-GRPO, an information-theoretic framework that cultivates correlation between the policy and a latent prior. Rather than raising entropy everywhere, Info-GRPO builds on the mutual information between a latent variable and the trajectory it induces: augmenting prompts with latent seeds during training lets the model explore diverse policies correlated with the prior, while the seed-conditioned entropy is guided toward convergence. Diversity is therefore carried across whole reasoning traces rather than by per-token randomness, and exploration no longer fights entropy reduction. Experiments show that Info-GRPO outperforms vanilla GRPO and entropy-regularized GRPO across diverse reasoning benchmarks. On AIME24, it improves $Avg@8$ over GRPO by 3.75%, 1.66%, 4.16%, and 4.17% on four backbones, and it reaches the same majority-vote accuracy with $8\times$ fewer samples. Code and models will be publicly released.
1 Introduction
Long-trace reasoning makes GRPO vulnerable to policy collapse, while token-level entropy bonuses can conflict with convergence. Info-GRPO instead correlates latent seeds with reasoning trajectories to support diverse, confident policies.
1 Introduction
- GRPO reinforces existing high-reward paths and can collapse onto a narrow set of strategies, despite the need for exploration in long reasoning traces.The context a policy must manage is largely self-generated as its reasoning trace grows.Source: §1 · p. 1
- Figure 1 reports validation results on Qwen2.5-Math-7B in panel (a), contrasting Info-GRPO with the GRPO baseline.Its other panels depict entropy patterns for original versus seed-augmented prompts and seed–output correlation.Source: Figure 1 · p. 2
- Long-trace entropy bonuses repeatedly push probability toward low-probability tokens, and anomalous tokens can shape every later step in an 8K derivation.The authors argue long-horizon reasoning needs diversity between solution paths and low uncertainty within each path.Source: §1 · pp. 1–2
- Info-GRPO conditions policies on latent priors and maximizes mutual information, placing diversity across reasoning trajectories while allowing each conditioned policy to become confident.The latent prior is used during training; inference runs on the original prompt.Source: §1 · p. 2
- Info-GRPO frames naive entropy regularization as tail-sensitive in large vocabularies and reorients learning toward conditional entropy reduction.The contribution describes this reduction as consistent with convergence.Source: §1 · p. 2
- Across diverse benchmarks and models, Info-GRPO consistently outperforms GRPO baselines.Latent-conditioned outputs also exhibit distinct, interpretable reasoning patterns.Source: §1 · p. 2
2 Related Work
Related work addresses GRPO’s exploration limits through changes to training objectives, entropy shaping, and mutual-information-based structure. Info-GRPO builds on latent-policy correlation to carry diversity across reasoning traces.
2 Related Work
- GRPO is a value-model-free method that replaces PPO’s critic with group-relative advantage estimation, and later variants refine its training pipeline.The passage also identifies offline preference-based policy optimization as a parallel line of work.Source: §2 · p. 2
- GRPO-based approaches suffer from insufficient exploration and policy collapse; this work extends GRPO to improve exploration and exploitation concurrently.Prior variants modify advantage estimation, curricula, clipping and sampling, value modeling, or chain-of-thought length.Source: §2 · pp. 2–3
- Entropy methods support exploration in RL and LLM reasoning, while ETR shapes uncertainty to decrease along the trace.The authors note that uncertainty introduced for exploration can weaken policy confidence and accumulate at each trace step.Source: §2 · p. 3
- Prior LLM work uses mutual information to reduce redundant GRPO completions, analyze trajectory dynamics, and optimize preference alignment.This paper instead maximizes MI conditioned on a latent variable so correlation carries diversity across whole reasoning traces.Source: §2 · p. 3
3 Background
The background models RLVR generation as a policy emitting tokens from a prompt-conditioned history, with the resulting trajectory representing the reasoning trace. GRPO estimates trajectory advantages relative to a sampled group, while policy entropy accumulates across generation steps.
3 Background
- In RLVR, the prompt is the initial state, each state records the action history, and the policy emits the next token; the resulting sequence is the reasoning-trace trajectory.Source: §3 · p. 3
- GRPO estimates a trajectory’s advantage against a sampled group and inserts that estimate into a PPO-style clipped objective.Source: §3 · p. 3
- Trajectory entropy accumulates policy uncertainty over its generation steps; high entropy corresponds to a uniform, uncertain policy, while low entropy indicates a deterministic, confident policy.Successful RLVR training reduces entropy as the model commits to high-quality reasoning paths; a sharp early drop can foreclose exploration.Source: §3 · p. 3
4 Why Entropy Bonuses Are Fragile at Long Trace Lengths
Entropy bonuses add a tunable exploration force to GRPO, but their pressure on low-probability vocabulary tails accumulates with trace length. The resulting instability makes direct entropy maximization sensitive to its coefficient in long-trace settings.
4 Why Entropy Bonuses Are Fragile at Long Trace Lengths
- The entropy bonus coefficient β arbitrates between reward-driven entropy reduction and exploration-driven entropy increase, but small values barely explore while large values destabilize training.The passage characterizes this sensitivity as structural rather than merely a tuning issue.Source: §4 · p. 3
- Direct entropy maximization can push a policy toward a uniform distribution, and its logarithmic factor is largest across the vast low-probability vocabulary tail.In Qwen2.5-7B’s 152,064-token vocabulary, fewer than five tokens exceed probability 10^-6.Source: Figure 2 · p. 4
- Total Tail Instability measures the summed magnitude of entropy-specific scalar factors over low-probability tokens, diagnosing how entropy allocates optimization pressure.The realized parameter update also depends on the policy Jacobian and optimizer.Source: §4 · p. 4
- Because trajectory entropy applies its per-step quantity once per token, misallocated pressure grows as O(T · TTI) with trace length and vocabulary size.The passage contrasts short generations with long-form reasoning traces thousands of tokens long.Source: §4 · p. 4
- Entropy maximization amplifies near-zero probabilities toward uniformity, whereas entropy minimization suppresses negligible probabilities; the authors use this asymmetry to motivate conditional-entropy exploration.Source: §4 · p. 4
5 The Info-GRPO Framework
Info-GRPO uses mutual information between latent seeds and reasoning trajectories to preserve diversity across paths while encouraging each seed-conditioned policy to become confident. Its practical objective combines weighted marginal entropy with reduced seed-conditioned entropy, using an unseeded anchor and a coefficient below 1.
5 The Info-GRPO Framework
- Figure 3 contrasts vanilla GRPO’s entropy decrease and entropy regularization’s broad increase with Info-GRPO’s mild increase plus stronger seed-conditioned decrease.The caption presents this balance as following the mutual-information decomposition between seeds and outcomes.Source: Figure 3 · p. 5
- The objective combines weighted marginal trajectory entropy with negative seed-conditioned entropy, while α controls how much marginal-entropy pressure remains.A small positive α anchors exploration in the original policy; values near 1 restore tail sensitivity.Source: §5 · p. 5
- Info-GRPO reframes exploration as mutual information between latent seeds and reasoning paths, preserving prompt-level diversity while encouraging confident seed-conditioned policies.Diversity is carried between trajectories rather than between tokens.Source: §5 · pp. 4–5
- The practical objective estimates marginal entropy from unseeded prompts and uses α < 1; the authors set α = 0.5 throughout.The unseeded branch anchors training to inference, which uses the original prompt; α = 1 restores tail-sensitive pressure, while an overly small α weakens the anchor.Source: §5 · p. 5
- Info-GRPO preserves GRPO’s loop and sampling budget by splitting G = 16 generations between base-policy and seeded-policy responses.Seeds are drawn uniformly from a digital set and appended using a fixed template; the two entropy terms are estimated by Monte Carlo from base and augmented trajectories.Source: §5 · p. 5
6 Experiments
Across the evaluated backbones and benchmarks, Info-GRPO outperforms GRPO, with majority-vote gains on AIME24 and ablations examining its entropy, trace-budget, and latent-seed choices.
6.1 Setup and Metrics
- Training covers four backbones and six benchmarks, reporting Pass@1, Pass@8, and Avg@8; semantic diversity measures pairwise distances between reasoning-path embeddings.The reported setup uses 8 rollouts per prompt for evaluation and sets λ = 0.005 and α = 0.5 throughout.Source: §6.1 · p. 6
6.2 Comparative Experiments
- 3.75%, 1.66%, 4.16%, and 4.17% AIME24 Avg@8 gains over GRPO across four backbones; Info-GRPO also beats the base model and GRPO on all code metrics.It achieves the best overall performance on Qwen and Llama backbones, although it is nearly tied with GRPO on Qwen2.5-7B.Source: §6.2 · p. 7
- 80.00 at N = 8 on DeepSeek-R1-Distill-Qwen-7B matches the accuracy GRPO reaches only at N = 64, an 8× larger sample budget.On Qwen2.5-Math-7B, GRPO does not reach Info-GRPO’s peak of 30.00 even at N = 128; the benchmark contains 30 problems.Source: §6.2 · p. 7, Table 3 · p. 7
6.3 Ablation Study
- The full-vocabulary entropy bonus lowers Avg@8 from 57.92 to 53.23, while restricting it to head tokens restores 58.13; Info-GRPO reaches 62.08.The compared entropy configurations differ only in whether the vocabulary tail participates.Source: §6.3 · p. 7, Table 4 · p. 7
- AIME25 Avg@8 rises monotonically from 37.08 to 45.83 as the trace budget grows from 2K to 8K, and AIME24 is best at 8K.The λ sweep finds that smaller values constrain exploration and improve accuracy; λ = 0.005 is used across backbones and benchmarks.Source: §6.3 · pp. 7–8, Table 5 · p. 7, Table 6 · p. 7
- At α = 1.0, training is unstable and AIME24 accuracy falls to 52.50, below vanilla GRPO; α = 0.5 performs best among the tested settings.At α = 0.05, the anchor is too weak.Source: §6.3 · p. 8, Table 7 · p. 8
- Appending seeds without MI regularization lowers Avg@8 from 45.04% to 42.23%, while gains stabilize around 10 seeds with higher inter-group and lower intra-group diversity.Replacing neutral digital seeds with semantic heuristics marginally increases diversity but degrades efficacy.Source: §6.3 · p. 8, Figure 4 · p. 8, Table 8 · p. 8
- Info-GRPO shows faster entropy reduction and steeper reward growth than GRPO, while exploration is carried by trajectory diversity rather than token-level randomness.The analysis describes this pattern as indicating more stable optimization.Source: §6.3 · p. 8
7 Conclusion
The paper attributes the conflict between entropy regularization and convergence in long-trace reasoning to tail sensitivity in vast token spaces. It proposes Info-GRPO and reports improvements over GRPO across benchmarks and architectures.
7 Conclusion
- Naive entropy regularization is tail-sensitive in vast token spaces, with its cost applied once per generated token and quantified by the TTI diagnostic.A controlled tail-masking ablation verifies the tail sensitivity of direct marginal entropy maximization.Source: §7 · p. 9
- Info-GRPO uses latent-variable augmentation and a generalized mutual-information-guided regularizer to ease the conflict between exploration and convergence.The paper identifies seed-conditioned reasoning modes and stable training dynamics as evidence for correlation-aware exploration.Source: §7 · p. 9
- Experiments show consistent improvements over GRPO baselines across multiple benchmarks and model architectures.The conclusion attributes Info-GRPO’s advantage to correlation-aware exploration.Source: §7 · p. 9
Limitations
The paper’s claims are bounded by its trace lengths, evaluation settings, and latent-space design. Several aspects of the seed’s contribution and use remain unresolved.
Limitations
- Claims are supported up to 8K tokens; the predicted effect at 32K or 128K remains unverified because retraining at those budgets exceeded available compute.The longest budget was 8,192 tokens, or 4,096 for the Qwen2.5 series.Source: Limitations · p. 9
- Avg@8 lacks a dispersion estimate because independent-seed repetitions of full RL training were prohibitively expensive.On 30-problem benchmarks, a one-problem difference is roughly 3.3 points; conclusions rely on consistency across backbones, benchmarks, metrics, and larger test sets.Source: Limitations · p. 9
- The gain attributable specifically to varying the latent seed remains open: the ablations do not separate it from the conditional-entropy term.A matched control keeping the seed fixed across augmented prompts would measure that effect; seed scheduling and per-problem selection are also unexplored.Source: Limitations · p. 9
- The flat, unstructured discrete latent space limits control granularity, while hierarchical or continuous spaces for compositional control remain future work.Neutral digital seeds avoid the alignment tax incurred by semantic heuristics.Source: Limitations · p. 9
- Evaluation is restricted to mathematics and code with strict verifiable rewards; generalization to tool use, varying reward sparsity, and open-ended reasoning remains untested.The restriction isolates the algorithmic effect of the mutual-information objective from noisy or hackable reward models.Source: Limitations · p. 9
- Info-GRPO does not help when a backbone cannot produce valid reasoning paths to diversify over; adaptive exploration modulation for such backbones is future work.On Qwen2.5-7B, it achieved the highest trajectory diversity on AIME25 without the highest accuracy.Source: Limitations · p. 9
A Appendix
The appendix provides proofs, related-work discussion, implementation details, further comparisons and ablations, qualitative analyses, and an AI-assistant disclosure.
A Appendix
- Sections A.1–A.3 provide theoretical proofs and derivations, extended related work, and additional training hyperparameters and implementation details.These topics are listed as supplementary support for the main manuscript.Source: A Appendix · p. 14
- Sections A.4–A.6 contain additional comparative results, extended ablations, and qualitative case analyses of reasoning diversity.The comparisons cover further metrics, baselines, and model series.Source: A Appendix · p. 14
- Section A.7 discloses the use of AI assistants in manuscript preparation.Source: A Appendix · p. 14
A.1 Proof
The proof bounds the size of the low-probability tail and the entropy-specific factor it accumulates. It frames this tail pressure as an objective-level allocation, while noting limits to what the diagnostic establishes.
A.1 Proof
- The high-probability head contains at most 1/δ tokens, independent of vocabulary size V, so the tail contains at least V − 1/δ tokens.For large V, the tail approaches the vast majority of the vocabulary.Source: §A.1 · pp. 14–15
- TTI aggregates entropy-specific logarithmic factors over the tail and is lower-bounded by a quantity that grows linearly with V when V > 1/δ.The bound follows from the magnitude of each tail token’s factor and the lower bound on tail-set size.Source: §A.1 · p. 15
- The objective allocates optimization pressure across Θ(V) low-probability tail tokens, but this does not establish that the realized parameter update diverges.The diagnostic omits policy Jacobians, and the realized update also depends on the optimizer, clipping, and numerical stabilization.Source: §A.1 · p. 16
- Info-GRPO exploits the asymmetry by reducing the marginal branch’s weight with α < 1 while its conditional-entropy branch drives probabilities toward the well-behaved end of the logarithm.Source: §A.1 · p. 16
- The entropy update aggregates many large-magnitude tail factors, directing a growing share of its signal toward tokens carrying almost no probability mass rather than the head tokens that determine the reasoning path.Source: §A.1 · p. 16
- Gradient clipping and numerical stabilization remain complementary safeguards: they do not remove the objective-level pressure to spread probability mass across the tail.The experiments use identical clipping and stabilization settings for baselines and Info-GRPO.Source: §A.1 · p. 16
A.2 Related Work
Related work situates GRPO and RLVR within reasoning-focused reinforcement learning, while noting that GRPO methods still face exploration and solution-diversity challenges. Prior entropy- and mutual-information-based approaches motivate Info-GRPO’s correlation-aware exploration.
A.2 Related Work
- GRPO uses trajectory-level comparisons instead of value networks, reducing computational costs and improving robustness over PPO; RLVR supplies outcome-based rewards for reasoning tasks.Prior work also links reasoning behaviors, rather than correct answers alone, to performance gains in RLVR.Source: §A.2 · p. 16
- GRPO-based approaches still struggle with insufficient exploration and low diversity in generated solutions.The paper frames its work as extending GRPO to improve exploration and exploitation.Source: §A.2 · p. 17
- Entropy-based methods encourage exploration but can make it difficult to balance exploration with the model’s confidence.Prior approaches target both policy-level collapse and individual token choices.Source: §A.2 · p. 17
- Mutual information has been used to promote diversity in learned representations and reasoning outputs, motivating its use with reinforcement learning in Info-GRPO.Examples include InfoGAN and InfoVAE for latent representations, and submodular mutual information for diverse reasoning completions [33].Source: §A.2 · p. 17
A.3 Full Experimental Settings
The experimental settings use a largely shared training pipeline, with method-specific exceptions for released checkpoints and official configurations. The accompanying tables cover training hyperparameters and comparisons across backbones and benchmarks.
A.3 Full Experimental Settings
- Most methods use the same training pipeline and core settings, with identical gradient clipping, loss scaling, and log-probability stabilization across methods.Methods marked † use released checkpoints and are evaluated as published; the reported differences are attributed to the objectives.Source: §A.3 · p. 18
- Info-GRPO uses mutual-information regularization, while KL-Cov and Clip-Cov use covariance-based clipping strategies and DAPO adds asymmetric clipping and overlong-response buffering.DAPO, Clip-Cov, and KL-Cov follow their official implementations and configurations.Source: §A.3 · p. 18
- Tables 11 and 12 compare methods across multiple backbones and benchmarks using Pass@1 and Pass@8, respectively.Table 13 compares with ETR on DeepSeek-R1-Distill-Qwen-7B using Pass@1 under an identical decoding protocol.Source: Table 11 · p. 19, Table 12 · p. 19, Table 13 · p. 20
A.4 Comparative Experiments
This section presents comparative experiments, ablations, and case analyses of performance, stable settings, and seed-conditioned reasoning, including cases where diversity does not improve accuracy when base-model capability is limiting. It also discloses that LLMs were used only for grammar checking.
A.4 Comparative Experiments
- Against ETR [39], Info-GRPO shows a downward uncertainty trend without an explicit trajectory penalty, ties on AIME24, loses on AMC23, and leads on MATH500, 94.8 vs. 90.6.The methods have a comparable overall average, and their mechanisms are described as complementary.Source: §A.4 · p. 20
- Pass@1 gains over GRPO average 4.72% on Qwen2.5-Math-7B and 6.75% on DeepSeek-R1-Distill-Llama-8B; Pass@8 gains average 2.99% on Qwen2.5-7B and 2.85% on DeepSeek-R1-Distill-Qwen-7B.Info-GRPO has the best average performance across base models, though its single-attempt advantage diminishes on strong distilled backbones near saturation.Source: §A.4 · p. 20
- On AIME24, Info-GRPO reaches 80.0 at N=8 versus GRPO at N=64 on DeepSeek-R1-Distill-Qwen-7B; on Qwen2.5-Math-7B it reaches 30.0 at N=8, which GRPO misses even at N=128.The Qwen2.5-Math-7B comparison is a one-problem gap on the 30-problem benchmark.Source: §A.4 · p. 20
- Across model settings, Info-GRPO leads on most benchmarks against KL-Cov and Clip-Cov [16], has the best mean versus Light-R1 [25], and generalizes beyond Qwen to DeepSeek-R1-Distill-Llama-8B.Dr. GRPO [23] scores highest on AIME24 but has a substantially lower overall average; Info-GRPO is reported as more stable across tasks and model families.Source: §A.4 · pp. 20–21
- On AIME25, Info-GRPO reaches 8.75 at its diversity peak, below GRPO, DAPO, and Clip-Cov, while Skywork-OR1 scores 7.5; diversity alone does not ensure accuracy.The analysis attributes this limitation to insufficient base-model capability: added diversity can increase low-quality or unstable reasoning paths.Source: Table 14 · p. 21, §A.4 · p. 21
A.5 Ablation Study
- Lower λ values, especially 0.005, improve generation quality and stability across disjoint benchmarks and another architecture, so 0.005 is shared across backbones and benchmarks without per-benchmark tuning.The same λ ordering appears on AMC and Minerva as well as the headline AIME sets.Source: §A.5 · pp. 21–22, Table 15 · p. 22, Table 16 · p. 22
- Increasing the response budget from 2K to 8K tokens consistently improves both AIME benchmarks; the maximum length is fixed by backbone, with pilot runs beyond 8K saturating at sharply higher inference cost.Limits are 4,096 tokens for Qwen2.5 and 8,192 for DeepSeek-R1-Distill, applied equally across methods on each backbone.Source: §A.5 · p. 22
- Ten latent seeds match the best Pass@1 and Avg@8 scores, while per-seed intra-group diversity generally declines and stabilizes across training, supporting convergence within seed-conditioned policies.Across 3, 10, and 30 seeds, performance is comparable and semantic diversity varies only slightly; the downward dynamics hold across all 10 seeds despite mild early fluctuations in some.Source: §A.5 · p. 22, Figure 5 · p. 23, Table 17 · p. 23
A.7 Usage of LLMs
- LLMs were used exclusively for grammar checking, not writing assistance, literature retrieval, or research ideation.Source: §A.7 · p. 28