PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang, Haitao Mi, Dong Yu

arXiv:2509.15194v3cs.LGcs.CL

TL;DR

Label-free LLM self-improvement can reinforce majority solutions, causing entropy collapse and reduced reasoning diversity. EVOL-RL combines majority-based selection with semantic novelty rewards, and it consistently improves in-domain and out-of-domain performance over majority-only training.

  • Problem

    Existing label-free self-improvement signals reinforce self-confirming, majority-favored solutions, degrading entropy, pass@n, and reasoning diversity.

  • Method

    EVOL-RL uses majority-voted answers as a stability anchor and rewards semantic novelty among concurrently generated reasoning traces within GRPO.

  • Results

    EVOL-RL consistently improves pass@1 and pass@16 over TTRL across settings, with gains frequently exceeding 20 percentage points on challenging benchmarks.

  • Takeaways & Limitations

    EVOL-RL prevents diversity collapse while improving reasoning performance and out-of-domain generalization without external labels.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly trained with reinforcement learning from verifiable rewards (RLVR), yet real-world deployment demands models that can self-improve without labels or external judges. Existing self-improvement approaches primarily rely on self-confirmation signals (e.g., confidence, entropy, or consistency) to generate rewards. This reliance drives models toward over-confident, majority-favored solutions, causing an entropy collapse that degrades pass@n and reasoning complexity. To address this, we propose EVOL-RL, a label-free framework that mirrors the evolutionary principle of balancing selection with variation. Concretely, EVOL-RL retains the majority-voted answer as an anchor for stability, but adds a novelty-aware reward that scores each sampled solution by how different its reasoning is from other concurrently generated responses. This majority-for-stability + novelty-for-exploration rule mirrors the variation-selection principle: selection prevents drift, while novelty prevents collapse. Evaluation results show that EVOL-RL consistently outperforms the majority-only baseline; e.g., training on label-free AIME24 lifts Qwen3-4B-Base AIME25 pass@1 from baseline's 4.6% to 16.4%, and pass@16 from 18.5% to 37.9%. EVOL-RL not only prevents in-domain diversity collapse but also improves out-of-domain generalization (from math reasoning to broader tasks, e.g., MMLU-Pro and BBEH). The code is available at: https://github.com/YujunZhou/EVOL-RL.

1 Introduction

Label-free self-improvement can collapse solution diversity when rewards reinforce the model’s existing majority preferences. EVOL-RL restores exploration by combining majority-based stability with novelty-based variation, improving accuracy and generalization.

  • Motivation: Majority-only self-improvement reinforces conformity, progressively impoverishes reward signals, and drives entropy and reasoning diversity downward.This collapse can reduce pass@n even when pass@1 rises.
  • Proposed approach: EVOL-RL combines majority-voted answers for stable selection with semantic novelty rewards that promote diverse reasoning paths.The method explicitly balances refinement of known solutions with discovery of new ones.
  • Results: 16.4% versus 4.6% AIME25 pass@1 and 37.9% versus 18.5% pass@16 follow label-free AIME24 training of Qwen3-4B-Base with EVOL-RL versus TTRL.These results are reported as representative out-of-domain gains.
  • Contributions: The paper frames diversity collapse as premature convergence and reports gains in pass@n, reasoning chains, and out-of-domain accuracy.Its stated contributions include a novelty-aware reward and a theoretical analysis of entropy stabilization.

2 Related Works

Prior work has substantially advanced LLM reasoning through reinforcement learning with verifiable rewards, which uses automated verification of final answers.

  • Enhancing Reasoning in LLMs: RLVR fine-tunes language models on tasks such as mathematics and coding when an automated verifier can confirm final-answer correctness.The related-work passage situates this paradigm as a major driver of recent reasoning progress.

3 Method

EVOL-RL uses GRPO to update a policy from grouped responses, assigning rewards that prioritize majority correctness while promoting semantic novelty. Supporting entropy regularization and asymmetric clipping further preserve diverse, high-quality reasoning.

  • Optimization with GRPO: GRPO evaluates each sampled response relative to peers from the same prompt and updates the policy with a clipped objective and KL regularization.It performs policy-gradient optimization without a separate value function.
  • Framework overview: EVOL-RL generates response groups, identifies majority answers, scores reasoning novelty, and combines both signals into GRPO rewards.Figure 2 depicts answer-based grouping with colors and semantically distinct reasoning paths with marker shapes.
  • Selection and variation: Majority correctness supplies stable selection, while semantic novelty supplies exploratory pressure that maintains diverse reasoning strategies.The novelty incentive applies to both majority-aligned and minority solutions.
  • Reward formulation: Majority and minority labels are assigned from valid boxed numeric answers, then mapped with nonoverlapping reward bands so correctness remains primary.Within each group, novelty refines the reward without overtaking the majority signal.
  • Reward formulation: Novelty is computed from cosine similarities among reasoning embeddings, using intra-group mean similarity and global maximum similarity to penalize redundancy.Scores are min-max normalized separately within majority and minority groups.
  • Supporting mechanisms: Entropy regularization and asymmetric clipping preserve exploration and prevent promising novel solutions from being prematurely suppressed.The total objective combines the GRPO loss with a token-level entropy regularizer.
  • Theoretical support: Theoretical analysis states that correctness-only optimization can collapse onto one solution, whereas the diversity-augmented objective spreads probability across correct reasoning modes.This provides a formal rationale for entropy stabilization.

4 Experiments

Across model scales and training datasets, EVOL-RL improves both single-shot and multi-sample reasoning over TTRL while preserving generalization. Its novelty reward is especially important on easier data, whereas entropy regularization and asymmetric clipping enable exploration on harder tasks.

  • Main Results: EVOL-RL consistently improves pass@16 over TTRL, often by more than 20 percentage points, while also producing more consistent pass@1 gains.+24.2% pass@16 on AIME24 for the 4B model is one reported example.
  • Main Results: EVOL-RL’s gains hold across 4B and 8B models and across MATH-TRAIN, MATH-500, and AIME24 training datasets.The reported robustness spans substantially different model capacities and data volumes.
  • Main Results: For the 4B model trained on MATH-500, EVOL-RL reaches nearly the same AIME24/AIME25 performance as direct AIME24 training, while 8B MATH-TRAIN training reaches 26.0% and 21.6% pass@1.These comparisons are presented as evidence of transferable rather than purely overfit reasoning skills.
  • Main Results: EVOL-RL improves non-mathematical transfer, achieving +7 to +15% pass@16 over TTRL on GPQA across training configurations.TTRL consistently degrades GPQA pass@16 relative to the base model, whereas EVOL-RL recovers and surpasses it.
  • Ablation Study: Removing novelty causes the largest pass@16 degradation on MATH-500, where majority-only training can lock the model into repetitive reasoning templates.Novelty prevents template lock-in and promotes generalizable skills.
  • Ablation Study: On harder AIME24 training, removing entropy regularization or asymmetric clipping lowers pass@16 because these mechanisms supply varied paths and preserve rare high-value learning signals.The components act as enablers for the novelty reward when task difficulty already induces more exploration.
  • Training Dynamics: Training dynamics show an initial entropy and response-length collapse for both methods, followed by EVOL-RL recovery after an evolving point.During recovery, entropy, response length, and out-of-domain accuracy rise together, while TTRL remains trapped in a low-complexity state.

5 Conclusion

EVOL-RL addresses entropy collapse by combining majority-vote stability with semantic-novelty exploration, improving both in-domain and out-of-domain performance.

  • EVOL-RL balances majority-vote selection with an explicit semantic-novelty incentive to prevent diversity collapse during label-free evolution.The majority signal anchors learning, while novelty encourages exploration and preserves reasoning complexity.

A Implementation Details

The implementation uses GRPO with grouped rollouts, majority-based validity signals, and novelty scores computed from normalized reasoning embeddings.

  • Implementation Setup: Training generates grouped responses, extracts valid numeric answers, and assigns majority labels from the extracted final answers.The setup uses Qwen3-4B-Base and Qwen3-8B-Base with 64 candidate rollouts per problem.
  • Novelty Calculation: Novelty is estimated from semantic similarity between generated reasoning traces using L2-normalized Qwen3-4B-Embedding vectors.The similarity matrix is computed as S = VV^T, with each entry representing pairwise cosine similarity.
  • Normalization and Settings: Raw novelty scores are min-max normalized within each response group, using 10^-8 to avoid division by zero when scores coincide.The experiments largely follow TTRL settings, with general and EVOL-RL-specific hyperparameters reported separately.
  • Compute: All experiments were conducted on a single server equipped with 8 NVIDIA H20 GPUs.

B.1 Effectiveness on Different Model Architectures

Across architectures and label-free baselines, EVOL-RL consistently improves multi-path reliability while avoiding the entropy-collapse pattern observed in majority- or confidence-only methods.

  • Entropy-Collapse Contrast: On OctoThinker, TTRL raises MATH pass@1 from 33.8% to 63.8% but lowers AIME24 pass@16 from 13.5% to 10.8%.TTRL also lowers GPQA pass@16 from 85.7% to 71.5%, illustrating improved single-shot accuracy alongside degraded multi-path performance.
  • Effectiveness Across Architectures: +19.5% on AIME24 and +11.4% on AIME25 are EVOL-RL’s pass@16 gains over TTRL on OctoThinker-8B-Hybrid-Base.EVOL-RL produces large pass@16 improvements across benchmarks while achieving comparable MATH pass@1 gains.
  • Comparison with Baselines: On AIME24, TTRL and Self-Consistency reach 23.2% and 30.4% pass@16, respectively, versus 32.4% for the Base Model.The cited comparison reports no consistent pass@16 improvement for single-signal baselines.
  • Comparison with Baselines: EVOL-RL is the only compared method reported to improve both pass@1 and pass@16 across all benchmarks, especially on out-of-domain tasks.On AIME24, it reaches 20.7% pass@1 and 47.6% pass@16, exceeding the next-best pass@1 baselines and the Base Model’s pass@16.

B.3 Analysis of the Majority Vote Signal

Training dynamics show TTRL plateauing at a consensus solution, whereas EVOL-RL later resumes improving majority-vote accuracy and stabilizes at a higher level.

  • TTRL Dynamics: TTRL initially improves maj@16 but quickly reaches a plateau, indicating that consensus-only training becomes locked into a local optimum.
  • EVOL-RL Dynamics: EVOL-RL’s maj@16 accuracy diverges from the TTRL plateau and undergoes a sustained second ascent to a higher stable level.Its early trajectory often resembles TTRL before the later divergence associated with the evolving point.

B.4 Training Dynamics of 8B Models

Across three training datasets, EVOL-RL escapes the low-entropy, low-complexity state that traps TTRL and recovers entropy, response length, and AIME25 accuracy at 8B scale.

  • EVOL-RL’s 8B training dynamics largely mirror its 4B behavior, indicating that its core mechanisms remain robust to scale.
  • Across MATH-TRAIN, MATH-500, and AIME24, both methods initially lose policy entropy and response length under majority pressure.
  • TTRL remains trapped in a low-entropy, low-complexity state, whereas EVOL-RL later enters a sustained recovery phase.
  • EVOL-RL’s entropy recovery is followed by longer responses and rising out-of-domain AIME25 accuracy, while TTRL stagnates.

B.5 Analysis of the Computational Overhead from the Novelty Reward

The novelty reward adds embedding and similarity computation to training, with costs increasing for longer reasoning but remaining a minor share of total runtime.

  • Novelty reward calculation requires B × N embedding calls and O(BN^2) similarity computation in addition to rollout generation and model updates.
  • As reasoning length grows, both novelty calculation and total training time increase because longer responses take longer to embed and generate.
  • The novelty reward remains a small portion of runtime, stabilizing around 100 seconds despite its dependence on response length.
  • The reward protects rare exploratory errors but strongly penalizes common minority failure modes when both global and within-group similarity are high.
  • Theoretical analysis states that, under its assumptions, the novelty objective favors broader coverage across correct reasoning trajectories than correctness-only optimization.

D.2 Step 1: Global maximizers concentrate on the correct set

Under bounded dissimilarity and a sufficiently small similarity weight, both correctness-only and similarity-augmented objectives place optimal policy mass entirely on correct trajectories.

  • The proof uses the unit reward gap between correct and incorrect trajectories to show that, for small λ, replacing incorrect mass with correct mass improves the objective.
  • Correctness-only optimization also has global maximizers supported on G, with every distribution over G achieving the same optimal value.
  • For sufficiently small λ, every global maximizer of the similarity-augmented objective is supported on the correct set G.
  • The result assumes a nonempty correct set, bounded dissimilarity, multiple reasoning modes, and larger dissimilarity across modes than within modes.

D.4 Step 3: Coverage structure and entropy of optimal policies on G

Correctness-only optimization permits collapsed policies over correct trajectories, whereas the similarity-augmented objective selects coverage across reasoning modes and can yield the maximum-entropy correct policy.

  • For fixed mode masses, uniformity within each mode minimizes the relevant sum-of-squares term, uniquely when within-mode dissimilarity is positive.
  • When all modes have equal size and within-mode dissimilarity is positive, the optimal policy is uniform over all correct trajectories and maximizes entropy.
  • Correctness-only optimization is flat over the correct set, so degenerate policies concentrated on one mode are globally optimal.
  • The similarity-augmented objective requires positive probability on every correct reasoning mode and uniform allocation within each mode.
  • Thus, under the stated binary-reward assumptions, similarity-augmented training favors dispersed correct solutions while correctness-only training does not enforce coverage.
Loading 2509.15194v3…