PaperlayerView exact revision on arXiv

Source-linked AI summary

The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward

Long Li, Zhijian Zhou, Jiaran Hao, Jason Klein Liu, Yanting Miao, Wei Pang, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, Yuan Qi

arXiv:2509.07430v4cs.LGcs.AI

TL;DR

RLVR can improve single-attempt accuracy while reducing multi-attempt diversity, and the role of its divergence term remains under-examined. DPH-RL uses mass-covering f-divergences as a rehearsal mechanism, with experiments showing improved diversity preservation and performance across SQL and mathematical reasoning tasks.

  • Problem

    RLVR often improves Pass@1 while degrading Pass@k, but the role of its divergence term in preserving solution diversity remains under-examined.

  • Method

    DPH-RL uses mass-covering f-divergences, such as forward-KL and JS divergence, to preserve broad solution coverage during online RLVR training.

  • Results

    Across SQL and mathematical reasoning experiments, DPH-RL preserved or improved Pass@k and related performance in in-domain and cross-domain settings, with DPH-JS exceeding GRPO by 4.3% on Bird.

  • Takeaways & Limitations

    Choosing a mass-covering divergence can make the divergence term an active mechanism for mitigating diversity collapse and catastrophic forgetting in RLVR.

  • Takeaways & Limitations

    The theoretical guarantee relies on an assumption that reference-policy actions have expected advantage at least δ for every training state.

Abstract

from arXiv · show

A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance (Pass@k) despite improvements in single-attempt accuracy (Pass@1). This is often accompanied by catastrophic forgetting, where models lose previously acquired skills. While various methods have been proposed, the choice and function of the divergence term have been surprisingly unexamined as a proactive solution. We argue that standard RLVR objectives -- both those using the mode-seeking reverse KL-divergence and those forgoing a divergence term entirely -- lack a crucial mechanism for knowledge retention. The reverse-KL actively accelerates this decay by narrowing the policy, while its absence provides no safeguard against the model drifting from its diverse knowledge base. We propose a fundamental shift in perspective: using the divergence term itself as the solution. Our framework, Diversity-Preserving Hybrid RL (DPH-RL), leverages mass-covering f-divergences (like forward-KL and JS-divergence) to function as a rehearsal mechanism. By continuously referencing the initial policy, this approach forces the model to maintain broad solution coverage. Extensive experiments on math and SQL generation demonstrate that DPH-RL not only resolves the Pass@k degradation but improves both Pass@1 and Pass@k in- and out-of-domain. Additionally, DPH-RL is more training-efficient because it computes f-divergence using generator functions, requiring only sampling from the initial policy and no online reference model. Our work highlights a crucial, overlooked axis for improving RLVR, demonstrating that the proper selection of a divergence measure is a powerful tool for building more general and diverse reasoning models.

1 INTRODUCTION

RLVR improves single-attempt correctness but can collapse solution diversity and worsen forgetting, with reverse-KL’s role largely overlooked. The paper introduces DPH-RL, which uses alternative f-divergences to preserve broad solution coverage and the model’s original knowledge.

  • RLVR-tuned models improve Pass@1 while suffering degraded multi-attempt performance, reflecting a solution-diversity collapse.
  • Prior responses target entropy, Pass@k optimization, or training setups, while the influence of the divergence term remains largely under-examined.
  • Reverse-KL is mode-seeking and can concentrate the policy on a single high-probability solution, suppressing diversity and contributing to catastrophic forgetting.
  • DPH-RL addresses diversity collapse by applying mass-covering f-divergences, including forward-KL and Jensen-Shannon divergence, to preserve coverage of reference-policy solutions.Forward-KL acts as an anchor dataset that continuously rehearses the model’s original knowledge base.
  • The paper identifies reverse-KL as a primary cause of diversity collapse and links it to lower Pass@k, catastrophic forgetting, and poor out-of-domain generalization.

2 RELATED WORK

Related work addresses RLVR diversity collapse through entropy control and explores generalized f-divergences for policy optimization. This work differs from f-PO by applying mass-covering f-divergences to online RLVR with verifiable rewards rather than offline preference alignment.

  • RLVR is a recent strategy for improving LLM reasoning in mathematics, coding, and analysis, with research addressing diversity collapse through direct entropy control.
  • Generalized f-divergences have been used to replace reverse-KL in offline DPO, whereas this work applies mass-covering f-divergences to online RLVR with verifiable rewards to address Pass@k diversity collapse.

3 PRELIMINARIES

This section introduces f-divergences as a unified family of distribution differences and establishes the MDP and surrogate-objective foundations underlying policy optimization.

  • f-divergences: f-divergences provide a unified framework for measuring differences between probability distributions, with KL and JS divergence as special cases defined by generator functions.The condition f(1) = 0 makes the divergence zero exactly when the two distributions are equal.
  • MDP formulation: The reinforcement-learning setting is a discounted MDP in which a stochastic policy selects actions to maximize expected cumulative discounted reward.The MDP is specified by state and action spaces, transition and reward functions, an initial-state distribution, and discount factor γ.
  • Policy optimization: Because the new policy’s state-visitation distribution depends on that policy, TRPO and PPO optimize a surrogate objective using the old policy’s visitation distribution.This approximation is reliable when the new and old policies remain close.

4 METHOD

DPH-RL partitions queries by reference-policy performance, disabling regularization for exploration while using Forward-KL or JS divergence to preserve near-perfect capabilities. Its analysis provides a stronger policy-improvement guarantee on expert-like data while recovering the standard TRPO bound during exploration.

  • Divergence variants: The method instantiates two variants—DPH-F with Forward-KL and DPH-JS with JS divergence—to regularize near-perfect samples while allowing exploration elsewhere.Forward-KL is defined as DKL(πref||πθ), and the framework computes divergence from reference-policy samples rather than online reference-model inference.
  • DPH-RL objective: DPH-RL partitions queries into near-perfect and exploration sets, applying unrestricted reward optimization to challenging examples and divergence regularization to capability-preserving examples.The method uses pre-sampled reference-policy responses to classify queries and trains with separate objectives for Dpef and Dexp.
  • Implementation: Pre-sampling the reference policy enables static-dataset divergence estimation, eliminating reference-model inference during online training.The total batch loss combines the appropriate Dpef or Dexp objective according to each sample’s partition.
  • Policy-improvement analysis: The theoretical framework uses a conditional reference policy that equals the current policy on Dexp, making its divergence penalty zero and enabling unrestricted exploration.On Dpef, the conditional reference policy stores near-perfect reasoning behavior for capability retention.
  • Policy-improvement analysis: On Dpef, the enhanced bound includes a positive ϵf bonus under effective regularization, while on Dexp it reduces to the standard TRPO guarantee.The result formally supports accelerated convergence from known expert behavior and preserves the conventional guarantee where regularization is disabled.

5 EXPERIMENTS

Experiments across SQL and mathematical reasoning tasks show that DPH-RL preserves diversity and improves generalization, especially under out-of-domain evaluation and larger sampling budgets. Results with Llama and Qwen models support the effectiveness of forward- and JS-divergence variants across model capabilities.

  • SQL Results: On Bird, DPH-JS exceeds GRPO and DAPO in Pass@8 by 4.3% and 3.3%, while both baselines fall below the base model.DPH-F and DPH-JS surpass the base model, indicating stronger diversity preservation on the in-domain SQL task.
  • Ablation Study: Increasing the f-divergence weight η steadily raises Pass@16, while minimal η approximates DAPO trained only on Dpef.The ablation evaluates the plug-and-play f-divergence loss and compares Generator and Divergence implementations.
  • Performance on OOD Tasks: DPH-F and DPH-JS outperform DAPO on average out-of-domain mathematical performance by 8.35% and 7.6%, respectively.GRPO and DAPO overfit the SQL training domain, whereas DPH methods retain stronger performance across five OOD mathematical datasets.
  • Performance with Massive Sampling: At k = 256, all methods converge, but GRPO and DAPO plateau near 75% while the DPH family substantially outperforms GRPO.DAPO performs better than GRPO at small k values but saturates early, limiting its scaling to larger numbers of samples.
  • Mathematical Tasks: DPH-JS preserves Llama’s original Pass@k while significantly improving mean@k on AIME, demonstrating exploration and knowledge preservation across model strengths.The mathematical experiments compare Pass@k and Mean@k across model families with different capabilities.

6 ANALYSIS

The analysis attributes higher Pass@k to balancing knowledge retention with additional exploration, while threshold choices trade diversity against greedy performance. DPH-JS’s data-partition threshold shows negligible differences between the strongest settings but reduced diversity at a lower threshold.

  • Knowledge retention and exploration: The analysis frames Pass@k gains through two factors: retaining solutions already found by the base model and discovering additional solutions.The study measures these as keep rate on previously correct samples and additional exploration rate on previously incorrect samples.
  • Knowledge retention and exploration: GRPO and DAPO both reduced keep rates, whereas the KL method mainly maintained higher keep rates; DAPO explored more in-domain but lost substantial out-of-domain retention.RKL maintained high retention on Bird but dropped to a level similar to GRPO on Spider.
  • Threshold analysis: The ‘7 of 8’ and ‘8 of 8’ DPH-JS thresholds produced negligible performance differences, while ‘6 of 8’ increased greedy performance at the cost of some diversity.At ‘6 of 8’, removing simpler samples from Dexp concentrates exploration on more challenging problems while the KL term supports performance.

7 CONCLUSION … E.2 DYNAMIC SAMPLING POLICY OPTIMIZATION (DAPO)

The paper presents DPH-RL as a divergence-based approach that preserves solution diversity and mitigates catastrophic forgetting, while connecting its forward-KL and JS variants to rehearsal mechanisms. It also contrasts these methods with GRPO and DAPO and documents how reverse-KL training collapses multiple solution styles.

  • 7 CONCLUSION: DPH-RL mitigates diversity collapse and catastrophic forgetting through f-divergence, with Generator and Divergence Definition implementations showing similar performance at computational costs comparable to GRPO.The Generator form requires no reference model during training, improving adaptability.
  • B MULTIPLE STYLE CAPABILITY EXPERIMENT: After reverse-KL-constrained GRPO, outputs degenerate into one style, whereas forward KL significantly mitigates degradation in the model’s ability to produce diverse SQL solution styles.The base model initially outputs all five styles.
  • C PRELIMINARIES ON f-DIVERGENCE: The paper defines f-divergences through strictly convex, lower-semicontinuous generator functions satisfying f(1) = 0, with forward and reverse KL as special cases of α-divergence.Table 5 summarizes representative divergences and their generator functions.
  • D THEORETICAL CONNECTION TO EXPERIENCE REPLAY AND REGULARIZED REHEARSAL; D.1 DPH-F AS DIVERGENCE-BASED EXPERIENCE REPLAY: Forward-KL rehearsal maximizes likelihood on reference-policy trajectories, using curated perfect solutions to preserve probability mass and prevent catastrophic forgetting.DPH-F therefore functions as an Experience Replay mechanism.
  • D.2 DPH-JS AS REGULARIZED REHEARSAL VIA DUAL-SIDED SAMPLING: JS divergence provides regularized rehearsal by comparing reference and current-policy samples, anchoring diversity while limiting drift without rigidly memorizing the reference buffer.Its symmetric mixture distribution requires dual-sided sampling.
  • E.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO): GRPO uses group-relative advantages, clipped policy updates, and a KL penalty to stabilize training and keep the fine-tuned policy near a reference policy.Its loss averages within sequences before averaging across samples.
  • E RLVR ALGORITHMS; E.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO): Reverse KL restricts exploration by heavily penalizing new-policy actions assigned low probability by the base policy, encouraging adherence to the base policy’s high-probability modes.This mode-seeking behavior narrows the exploration space.
  • E.2 DYNAMIC SAMPLING POLICY OPTIMIZATION (DAPO): DAPO removes the KL penalty, sets its upper clipping bound to 1 + εupper with εupper = 0.28, and adds dynamic sampling, token-level loss, and overlong reward shaping.Its dynamic sampling moves beyond all-or-nothing sample selection.

F METHOD FOR DIVERGENCE DEFINITION · G TRAINING DETAILS

The method implements f-divergences through divergence definitions or generators, with special handling for JS divergence because it requires samples from both the reference and new policies. DPH-JS separately computes the two JS components and balances them using β1 determined by dataset size and η.

  • F METHOD FOR DIVERGENCE DEFINITION: JS divergence requires sampling from both the reference policy πref and the new policy πθ, unlike equivalent generator and divergence-definition implementations for forward KL.
  • F METHOD FOR DIVERGENCE DEFINITION: The first JS component, DKL, is computed from data used by the Dpef loss.
  • F METHOD FOR DIVERGENCE DEFINITION: Training the Dexp loss introduces a reference model to calculate the second JS component, DKL.
  • F METHOD FOR DIVERGENCE DEFINITION: The DPH-JS loss on Dexp is defined separately to incorporate the second JS component during training.
  • F METHOD FOR DIVERGENCE DEFINITION: The DPH-JS loss on Dpef is expressed separately from its Dexp loss to account for the first JS component.

G.1 TASK SETTINGS … H ADDITIONAL EXPERIMENTS

The experiments evaluate SQL and mathematical reasoning across in-domain and cross-domain datasets, using specified RL implementations, training configurations, and offline data-construction procedures.

  • G.1 TASK SETTINGS: SQL experiments train Llama-3.1-8B-Instruct and OmniSQL32B on BIRD, then evaluate on BIRD and cross-domain Spider.The study also tests mathematical reasoning as an out-of-domain evaluation.
  • G.1 TASK SETTINGS: Mathematical experiments train Llama-3.1-8B-Instruct and Qwen2.5-Math-7B on filtered DAPO-17K and test seven datasets.The evaluation datasets are AIME24, AIME25, AMC23, Math500, Olympiad, Minerva, and College Math.
  • G.1 TASK SETTINGS: SQL evaluation executes generated statements and compares prediction-ground-truth Cartesian products, while math evaluation uses the official Qwen2.5 tool.The SQL executor returns structured results such as tuples for comparison.
  • G.2 METHOD SETTINGS: RKL follows standard GRPO, DPH-RL defaults to the Generator method, and all RL algorithms use token-level loss.The default DPH-RL ϵ_high is 0.28, reduced to 0.2 for all methods in unstable Llama math experiments.
  • G.3 HYPERPARAMETERS: Experiments use 32 NVIDIA A800-80G GPUs with VeRL, 16 rollouts, batch sizes of 128 and 256, and an effective D_exp:D_pef ratio of 8:1.The setup uses 2048 D_exp samples and 256 D_pef samples per learning step.
  • G.3 HYPERPARAMETERS: Offline data construction uses eight current-model rollouts, retaining imperfect SQL examples for D_exp and discarding entirely incorrect math responses.For SQL, perfectly correct examples are separated from the rollouts; for math, filtering applies to DAPO-17K.

H.1 32B EXPERIMENTS FOR SQL … I.1 SQL TASK

Across larger-scale SQL experiments and training-progress analyses, DPH-F and DPH-JS preserve diversity more effectively than baseline methods, while the case study specifies the SQL task prompts and example solution format. The merged material also documents error-bar evaluation procedures and mean@k validation for mathematical reasoning.

  • H.1 32B EXPERIMENTS FOR SQL: On the larger 32B model, DPH-F and DPH-JS are reported as equally effective; Bird’s greedy score of 70.5 is near the open-source single-model performance limit.These results are presented for Omnisql-32B SQL tasks in Table 8.
  • H.2 ANALYSIS OF TRAINING PROGRESS: DPH-F and DPH-JS consistently maintain Llama’s Pass@8 above its initial level, whereas GRPO collapses on Bird and DAPO performs poorly on Spider.DAPO’s performance also oscillates on Bird and sacrifices out-of-domain generalization for improved in-domain performance.
  • H.4 MEAN@K METRIC FOR MATHEMATICAL REASONING TASK: Mean@k was used to verify that reinforcement-learning training was effective on the mathematical reasoning tasks.The comparison is reported in Table 9, whose training used filtered DAPO-17k data and 5.8k Llama or 10.2k Qwen samples.
  • H.3 ERROR BAR: Each Llama SQL method was evaluated with three independent reinforcement-learning runs, selecting the model with the highest Bird Pass@16 score.The procedure is summarized in Figure 6’s error-bar description.
  • I CASE STUDY: The case study provides detailed use cases and prompts for the two tasks to support future reproduction.This establishes the reproducibility purpose of the subsequent prompt and response examples.
  • I.1 SQL TASK: The SQL task asks a data-science expert to generate a valid SQLite query from a database schema and natural-language question.The example instructs the model to place the generated query in a SQL code block and reason step by step.
  • I.1 SQL TASK: The example SQL response sums two enrollment columns, filters schools whose total exceeds 500, and returns their CDSCode values.The demonstrated output lists three school codes.

I.2 MATH TASK

The response solves Aya’s walking-time problem by determining s and t from two scenarios, then computes the total time at the intermediate speed. It concludes that the walk takes 204 minutes, including the coffee-shop stop.

  • I.2 MATH TASK: The solution determines that Aya’s walking speed parameter is s = 2.5 km/h.The response selects s = 2.5 km/h from the quadratic-equation solutions.
  • I.2 MATH TASK: The coffee-shop duration is found to be t = 24 minutes before computing the target travel time.The response substitutes s into the original equations to obtain t, then evaluates the requested intermediate-speed case.
  • I.2 MATH TASK: 204 minutes is the final total time for Aya’s walk, including the coffee-shop stop.The response explicitly gives 204 as the final answer after converting 3.4 hours to minutes.

J PROOF OF THEOREM 1

The proof establishes divergence-based bounds on total variation, then combines policy-performance lemmas with coupling conditions to derive Theorem 1’s lower bound on performance improvement.

  • Divergence bounds: Lemma 1 bounds squared total variation by KL, α-divergences, and JS-divergence, with a positive constant depending on the chosen divergence.The proof invokes Pinsker-type inequalities and establishes that a suitable constant exists for each considered f-divergence.
  • Auxiliary policy lemmas: The proof uses Schulman’s policy comparison lemma and an α-coupled-policy lemma to control the performance terms needed for Theorem 1.These lemmas relate policy expectations and coupled-policy advantages under trajectory sampling.
  • Theorem 1 proof: Assuming (π, ˜π) and (π_pef, ˜π) are α- and β-coupled through total-variation bounds, the proof derives a lower bound for J(˜π) − L_π(˜π).The derivation combines equalities from Lemma 2 and an inequality from Lemma 3, followed by denominator simplification using x ∈ (0, 1).
  • Theorem 1 proof: Lemma 1 completes the argument by converting KL-divergence bounds into total-variation bounds that establish the required α- and β-coupling conditions.Thus, the divergence inequalities supply the conditions used in the theorem’s policy-performance bound.
Loading 2509.07430v4…