PaperlayerView exact revision on arXiv

Source-linked AI summary

DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

Zhongwei Wan, Yun Shen, Zhihao Dou, Donghao Zhou, Yu Zhang, Xin Wang, Hui Shen, Jing Xiong, Chaofan Tao, Zixuan Zhong, Peizhou Huang, Mi Zhang

arXiv:2602.19895v1cs.LGcs.CL

TL;DR

RLVR reasoning policies can collapse onto a few patterns, limiting deep exploration and weakening learning signals when group rewards become nearly constant. DSDR couples correct-only global trajectory diversity with length-invariant local entropy regularization, allocating stronger local regularization to distinctive correct trajectories. Across multiple reasoning benchmarks, it reports consistent gains in accuracy and pass@k, supporting dual-scale diversity as a framework for exploration in RLVR.

  • Problem

    RLVR policies often concentrate on a small set of reasoning patterns, limiting deep exploration, pass@k improvement, and informative group-based learning signals.

  • Method

    DSDR couples global diversity among correct trajectories with positive-only, length-invariant token-level entropy on correct trajectories through a global-to-local allocation mechanism.

  • Results

    DSDR consistently improves accuracy and pass@k across diverse reasoning benchmarks, including average performance of 25.4 / 25.6 on Qwen2.5-Math-1.5B.

  • Takeaways & Limitations

    Dual-scale diversity can support deep exploration while preserving correctness and maintaining informative learning signals in group-based RLVR optimization.

Abstract

from arXiv · show

Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large language model (LLM) reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few reasoning patterns and prematurely stop deep exploration, while conventional entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM reasoning into global and coupling components. Globally, DSDR promotes diversity among correct reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant, token-level entropy regularization restricted to correct trajectories, preventing entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.

1. Introduction

RLVR improves LLM reasoning but often concentrates probability on a few homogeneous reasoning patterns, limiting deep exploration and weakening group-based learning signals. DSDR addresses this tension with correctness-aligned global and local diversity coupled across reasoning trajectories and tokens.

  • Motivation: RLVR policies can improve pass@1 while collapsing onto homogeneous reasoning patterns, limiting pass@k gains and generalization.The problem is especially pronounced on out-of-domain or more compositional reasoning tasks.
  • Motivation: Token-level entropy regularization mainly adds local randomness, whereas trajectory-level diversity methods may not jointly preserve correctness and distinct reasoning paths.Existing approaches therefore leave deep exploration and correctness insufficiently coordinated across trajectory and token scales.
  • Method: DSDR distinguishes global diversity across solution modes from local diversity within each mode, treating the two scales as complementary.The framework allocates local regularization according to global distinctiveness rather than treating all correct trajectories identically.
  • Method: DSDR couples global diversity among correct trajectories with positive-only, length-invariant local entropy regularization through a global-to-local allocation mechanism.This design promotes diverse correct trajectories while preventing entropy collapse within individual reasoning modes.
  • Contributions: The framework provides theoretical support for correctness preservation, informative group-normalized learning signals, and principled global-to-local coupling.These properties are supported together with consistent empirical gains.

2. Related Work

Prior RLVR and diversity-control methods improve reasoning or local stochasticity but commonly converge to narrow reasoning patterns. DSDR instead structures diversity across global and local scales within policy optimization without modifying rollout procedures.

  • RLVR and Exploration in LLMs: RLVR can elicit verification and self-reflection, but policies often converge early to narrow reasoning patterns and reach performance plateaus.This motivates methods that improve exploration during reasoning-oriented reinforcement learning.
  • Diversity and Entropy Control for LLM Reasoning: DSDR structures diversity directly within policy optimization across global-to-local scales, enabling deep exploration without modifying rollout procedures.Its design addresses the exploration limitation through policy updates rather than changes to rollout generation.
  • Diversity and Entropy Control for LLM Reasoning: Token-level entropy methods can alleviate premature collapse but primarily increase local action randomness rather than explicitly diversifying complete reasoning paths.Their control operates at the local policy-action level.

3. Methodology

DSDR augments group-based RLVR with global diversity among correct trajectories and local entropy regularization, coupling both scales to preserve informative exploration while maintaining correctness.

  • 3.1. Preliminaries: GRPO samples grouped outputs, normalizes rewards into relative advantages, and optimizes a clipped token-level surrogate whose signal depends on within-group reward variation.The verifier reward is typically binary, so nearly constant group rewards can weaken preference gradients.
  • 3.2. Global-scale diversity signals: DSDR adds trajectory-level diversity bonuses to correct rollouts, using bounded signals that distinguish solution paths without rewarding incorrect responses.The diversity score is designed to reflect trajectory differences, remain inexpensive relative to rollout generation, and stay safely scaled for reward shaping.
  • 3.2.1. Global-scale diversity signals: Global diversity combines semantic embedding dissimilarity with formula-level uniqueness, capturing complementary variation in reasoning meaning and symbolic manipulation.Embedding distances are averaged across group comparisons for stability, while formula uniqueness is averaged over extracted expressions and assigns no novelty when none are detected.
  • 3.2.2. Global-to-local coupling over correct trajectories: A diversity-weighted softmax over correct responses allocates local regularization, assigning stronger entropy incentives to globally distinctive trajectories.This coupling focuses local exploration on underrepresented regions while avoiding redundant expansion around already-common correct solutions.
  • 3.2.3. Local positive-sample regularization: The local entropy surrogate reuses group-sampled rollouts through per-token importance ratios, remains differentiable, and applies only to correct trajectories.The allocation weight couples local and global scales, while the correct-only restriction prevents entropy regularization from encouraging noise on incorrect solutions.
  • 3.2.4. DSDR objective: DSDR optimizes the group-relative objective with augmented rewards, and Theorem 3.1 derives its global-to-local softmax coupling from a correct-only diversity-tilted objective.The augmented rewards incorporate clipped global diversity bonuses before group advantages are formed.

4. Experiments

Across diverse mathematical reasoning benchmarks, DSDR consistently improves accuracy, pass@k, and exploration stability over Backbone, GRPO, and DAPO. Ablations and training analyses attribute these gains to coordinated global trajectory diversity and local entropy regularization.

  • Main Results: DSDR achieves the best average Pass@1 / Avg@16 performance across the evaluated model scales and benchmarks.Reported averages are 25.4 / 25.6 for Qwen2.5-Math-1.5B, 36.8 / 36.8 for Qwen3-1.7B, and 48.0 / 46.8 for Qwen3-4B.
  • Pass@k Evaluation: DSDR consistently outperforms Backbone and DAPO across Pass@k values, with widening gains on AIME2024, AIME2025, and Olympiad as k increases.It maintains an advantage on Minerva and stable improvements on MATH500 without degradation at large k.
  • Ablation Study: Removing global diversity or global-to-local coupling causes performance drops across model sizes, confirming that both components contribute to DSDR.The drops are especially evident on AIME and Olympiad when coupling is removed.
  • Training Dynamics: During training, DSDR maintains higher AIME2024 Avg@16 while balancing entropy and sustaining lower semantic similarity and symbolic diversity.Without global diversity, entropy rises excessively; without coupling, exploration diminishes later in training.
  • Diversity Analysis: With 32 test-time rollouts, DSDR produces higher judged response diversity and higher pass@32 than DAPO across four benchmarks.Diversity is evaluated by an LLM judge on a 1–10 scale.
  • Hyperparameter Sensitivity: Moderate regularization is stable, whereas λℓ values above 0.01 destabilize training; λd = 0.001 gives the best and most stable Avg@16.These sensitivity results are reported on Qwen3-1.7B for AIME2024/2025.

5. Conclusion

The paper introduces DSDR, a correctness-aligned framework that coordinates trajectory-level and token-level diversity in RLVR. Its analysis and experiments support stable exploration with consistent gains in accuracy, pass@k, and training stability.

  • Conclusion: DSDR promotes diversity among correct trajectories and applies length-invariant token-level entropy regularization exclusively to correct solutions.A global-to-local allocation mechanism focuses local regularization on globally distinctive correct trajectories.
  • Conclusion: The analysis shows that bounded local regularization preserves correctness while correct-only global shaping maintains informative group-based learning signals.The framework explicitly coordinates trajectory- and token-level exploration.
  • Conclusion: Experiments across reasoning benchmarks demonstrate consistent gains in accuracy, pass@k, and training stability.The conclusion attributes these results to coordinating the two exploration scales in RLVR.

6. Impact Statements

DSDR promotes correctness-aligned exploration at both trajectory and token levels for RLVR-based LLM reasoning, aiming to improve robustness, stability, and sample efficiency.

  • DSDR is a dual-scale diversity regularization framework for reinforcement learning with verifiable rewards in LLM reasoning.It promotes exploration at both trajectory and token levels.

A. Implementation Details

The experiments use VERL with specified hardware, default optimization settings, binary verifier rewards, and sweeps over diversity and coupling coefficients.

  • All models are trained using VERL and deployed on 8× NVIDIA A100 GPUs (40GB).
  • Unless otherwise specified, experiments use rollout group size n = 8, learning rate 1 × 10−6, and binary verifier rewards.
  • Hyperparameter sweeps vary global diversity scaling λd, local regularization λℓ, and the coupling configuration.The passage specifies λd ∈{0.001, 0.01, 0.1} and λℓ∈{0.001, 0.002, 0.01}.
  • Table 2 summarizes training details, while Table 3 reports evaluation settings.

B.1. Additional Training Dynamics Analysis

Training dynamics show DSDR sustaining gains in accuracy and diversity-related behavior, while removing global diversity destabilizes learning and DAPO saturates earlier.

  • DSDR steadily improves Avg@16 and Pass@16 on AIME2025 throughout training.The reported gains are accompanied by sustained exploration rather than convergence to a narrow solution mode.
  • GRPO and DAPO grow more slowly and saturate earlier than DSDR during training.
  • Removing global diversity causes large policy-gradient-loss fluctuations, indicating unstable learning when trajectory-level diversity is absent.
  • With 16 test-time rollouts, DSDR produces 7 correct solutions versus 2 for DAPO, using distinct reasoning processes.The examples illustrate more diverse valid solution paths under DSDR.

B.2.1. SAMPLES GENERATED BY DSDR

The DSDR samples illustrate distinct solution strategies for the digit-divisibility problem. Both admissible candidates are checked, and the greatest valid number yields Q + R = 699.

  • SAMPLES GENERATED BY DSDR: Two DSDR responses use different solution strategies for the same digit-divisibility problem.The sample presentation highlights the differing strategies in the generated responses.
  • SAMPLES GENERATED BY DSDR: The modular system yields two admissible digit pairs, (C, D) ∈{(2, 4), (9, 4)}, corresponding to 5624 and 5694.Each candidate is explicitly checked against all four digit-replacement conditions.
  • SAMPLES GENERATED BY DSDR: The solution formulates four modular divisibility conditions for the digits A, B, C, and D, then eliminates variables to derive candidate relations.The derivation uses congruences modulo 7 and successive subtraction.
  • SAMPLES GENERATED BY DSDR: For the greatest valid number 5694, Q = 5 and R = 694, so Q + R = 699.

B.2.2. SAMPLES GENERATED BY DAPO

The DAPO samples apply digit-wise divisibility constraints to a four-digit problem, but both solutions introduce unsupported assumptions and reach an incorrect result. The section also defines a three-dimensional rubric for scoring rollout diversity.

  • Generated samples: Both generated solutions formulate divisibility-by-7 constraints by replacing each digit of N with 1.The samples represent N as digits a, b, c, d and enumerate the resulting expressions or congruences.
  • Generated samples: Answer 2 sets a = 1 without justification and then proceeds by ad-hoc guessing and trial-based arguments.The passage identifies this assumption as the point where the reasoning becomes unsupported.
  • Generated samples: Answer 1 incorrectly assumes all four expressions share one residue class and reports Q + R = 897 instead of the ground truth 699.The subsequent digit enumeration is therefore based on a constraint not implied by the original system.

C. Theoretical Analysis of DSDR

The theoretical analysis formalizes DSDR’s correct-only diversity reward and positive-only local entropy regularization. It argues that the two mechanisms target different diversity scales, preserve optimal correctness under bounded regularization, and maintain non-degenerate group signals.

  • Design and setup: DSDR combines a correct-only global diversity reward with a length-invariant, positive-only token-level entropy regularizer.The framework is analyzed under verifier rewards in {0, 1}, with stabilized group normalization for group-based training.
  • Inter- and intra-mode diversity: Global shaping targets inter-mode coverage, while local entropy targets intra-mode variation within reasoning modes.The distinction follows the decomposition H(O | q) = I(O; Z | q) + H(O | Z, q).
  • Correctness preservation: With sufficiently small bounded local regularization, no correctness-suboptimal policy can maximize the regularized population objective.The result means the regularizer can break ties among correctness-optimal policies without sacrificing optimal correctness.

C.4. GRPO Signal Preservation via Correct-Only Global Diversity Reward

The analysis explains how DSDR preserves informative GRPO signals when verifier rewards are constant within a group and gives a principled interpretation of its diversity-based local allocation. The allocation is uniquely determined by an entropy-regularized optimization problem.

  • GRPO signal preservation: Verifier-only rewards can yield uninformative group advantages when sampled groups contain only successes or only failures.For i.i.d. Bernoulli verifier rewards, the mixed-group probability is 1 − p^G − (1 − p)^G.
  • GRPO signal preservation: A positive correct-only diversity bonus keeps group-relative advantages non-degenerate in solve-all groups when correct trajectories have unequal diversity scores.Under this condition, the augmented rewards are not all identical and their variance is positive.
  • Global-to-local allocation: The global-to-local coupling allocates local entropy across correct trajectories using diversity-softmax weights, with temperature τ controlling concentration versus exploration.The paper presents this allocation as the unique solution of an entropy-regularized resource-allocation problem.
  • Global-to-local allocation: The resulting allocation has weights proportional to exp(τ d̄_i) over correct rollouts.The derivation follows from stationarity and normalization on the simplex.
Loading 2602.19895v1…