PaperlayerView exact revision on arXiv

Source-linked AI summary

Jointly Reinforcing Diversity and Quality in Language Model Generations

Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, Tianlu Wang

arXiv:2509.02534v1cs.CLcs.LG

TL;DR

Post-training can reduce the diversity of language-model generations even as it improves quality. DARLING jointly optimizes semantic diversity and quality during online reinforcement learning, improving both across non-verifiable and verifiable tasks.

  • Problem

    Post-training can overly sharpen language-model output distributions, reducing response diversity and informativeness.

  • Method

    DARLING is an online reinforcement-learning objective that measures semantic diversity with a learned classifier and fuses it with quality to reinforce usefully different trajectories.

  • Results

    DARLING preserves diversity and improves benchmark performance across non-verifiable and verifiable tasks; in competition math, pass@1 exceeds GRPO by +3.51/1.90% for 4B and 14B models.

  • Takeaways & Limitations

    Jointly reinforcing quality and diversity can preserve diverse generations while improving performance across model families, sizes, and task regimes.

  • Takeaways & Limitations

    The advantage-normalization analysis assumes true utility plus noise with variance τ^2, limiting that analysis to this specified noise model.

Abstract

from arXiv · show

Post-training of Large Language Models (LMs) often prioritizes accuracy and helpfulness at the expense of diversity. This creates a tension: while post-training improves response quality, it also sharpens output distributions and reduces the range of ideas, limiting the usefulness of LMs in creative and exploratory tasks such as brainstorming, storytelling, or problem solving. We address this challenge with Diversity-Aware Reinforcement Learning (DARLING), a framework that jointly optimizes for response quality and semantic diversity. At its core, DARLING introduces a learned partition function to measure diversity beyond surface-level lexical variations. This diversity signal is then combined with a quality reward during online reinforcement learning, encouraging models to generate outputs that are both high-quality and distinct. Experiments across multiple model families and sizes show that DARLING generalizes to two regimes: non-verifiable tasks (instruction following and creative writing) and verifiable tasks (competition math). On five benchmarks in the first setting, DARLING consistently outperforms quality-only RL baselines, producing outputs that are simultaneously of higher quality and novelty. In the second setting, DARLING achieves higher pass@1 (solution quality) and pass@k (solution variety). Most strikingly, explicitly optimizing for diversity catalyzes exploration in online RL, which manifests itself as higher-quality responses.

1 Introduction

Post-training can reduce the diversity and informativeness of language-model generations. DARLING addresses this by combining learned semantic diversity with quality in online reinforcement learning, improving both novelty and quality across non-verifiable and verifiable tasks.

  • Post-training can sharpen output distributions, producing repetitive or near-duplicate responses and reducing informativeness.
  • DARLING partitions generations into semantically equivalent clusters, measuring diversity beyond superficial lexical differences.
  • DARLING fuses diversity and quality rewards to reinforce responses that are both high-quality and usefully different.
  • Across model families and sizes, DARLING preserves diversity and improves benchmark performance on instruction following, creative writing, and math problems.
  • DARLING jointly optimizes quality and diversity, preventing diversity collapse during post-training.
  • DARLING promotes exploration and often improves quality on non-verifiable creative-writing and verifiable competition-math benchmarks.

2 Notations and Preliminaries

The paper formulates language-model post-training as reward maximization under a KL constraint and describes GRPO as its starting optimization baseline. GRPO evaluates sampled responses through importance sampling and group-relative advantages.

  • A language model maps an input token sequence to a probability distribution over output token sequences.
  • Post-training maximizes expected reward while constraining divergence from a reference model with a KL penalty.
  • GRPO optimizes the KL-constrained objective using responses sampled per prompt.
  • GRPO uses an importance-sampling ratio between the current policy and the actor that generated each response.
  • Its advantage measures how much better or worse a response is than the average response, with ε limiting the importance-sampling term.

3 Method: DARLING

DARLING measures semantic diversity by partitioning responses into equivalence clusters, then multiplies normalized diversity by quality to reinforce responses that are both high-quality and distinct.

  • 3.1 Partitioning the Responses into Semantic Equivalence Classes: DARLING defines diversity for each response as its average pairwise distance from the other generations in a group.The group contains n generations, and diversity is normalized over the n − 1 other responses.
  • 3.1 Partitioning the Responses into Semantic Equivalence Classes: A learned binary classifier identifies semantically equivalent responses, which are clustered into partitions beyond superficial lexical variation.The classifier is used as the diversity metric, so equivalent responses contribute little additional value beyond one representative.
  • 3.1 Partitioning the Responses into Semantic Equivalence Classes: For a programming-joke example, blue responses have diversity 2/3, whereas yellow and purple responses have diversity 3/3 because they differ from all others.The normalized count reflects how many responses are distinct from the response being evaluated.
  • 3.2 DARLING: Diversity Aware Reinforcement Learning: DARLING multiplies the quality reward by normalized diversity, increasing the log-likelihood of responses that are simultaneously high-quality and diverse.The normalization maps diversity values between 0 and 1.
  • 3.2 DARLING: Diversity Aware Reinforcement Learning: Multiplication is preferred to naive reward addition because differing reward scales can cause training to prioritize quality or diversity.The paper reports an ablation study comparing reward-fusion methods.
  • 3.2 DARLING: Diversity Aware Reinforcement Learning: DARLING uses its diversity-aware reward as the effective reward in the overall loss, while also applying token-level averaging and removing standard-deviation normalization.These additional modifications address length bias and noise in dense rewards, respectively.

4 DARLING on Non-verifiable Tasks

On non-verifiable instruction-following and creative-writing tasks, DARLING is evaluated against quality and diversity baselines and achieves jointly stronger quality-diversity performance across model scales and benchmarks.

  • 4.1 Setup: DARLING is evaluated using Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct trained on 10k randomly sampled WildChat prompts.Quality is rewarded with Nexusflow/Athene-RM-8B, using 32 or 64 prompts and eight rollouts per prompt depending on model scale.
  • 4.1 Setup: The comparison includes standard GRPO, DivPO, and GRPO-Unlikeliness as baselines for non-verifiable task training.These methods differ in optimization objective or reward weighting for diversity and likelihood.
  • 4.1 Setup: Quality is measured with AlpacaEval, ArenaHard, and EQ-Bench, while NoveltyBench measures semantic and lexical diversity.Reported metrics include length-controlled win rate, style-controlled win rate, normalized ELO, Distinct, and normalized Distinct-4.
  • 4.2 Experimental Results: DARLING achieves the best quality and diversity across all evaluated non-verifiable benchmarks and baselines.It leads on AlpacaEval and ArenaHard win rates, semantic and lexical diversity, and EQ-Bench creative-writing ELO despite not training explicitly on creative-writing prompts.
  • 4.2 Experimental Results: Across sampling temperatures from 0.2 to 1.2, DARLING has better quality and diversity than the baseline and GRPO at both 8B and 70B scales.The resulting quality-diversity Pareto front is improved for both Llama model sizes.
  • 4.3 Qualitative Analysis: On EQ-Bench, DARLING most often wins on the “Interesting and Original” and “Avoids Cliche” rubrics against models with similar ELO.The analysis attributes these strengths to generating diverse outputs.
  • 4.3 Qualitative Analysis: Four parallel NoveltyBench generations from DARLING-trained models remain semantically distinct, including repeated recommendations with different rationales.The examples compare generations sampled at temperature 1.0 before and after DARLING training.

5 DARLING on Verifiable Tasks

DARLING is evaluated on competition math using Qwen3-4B-Base and Qwen3-14B-Base, comparing against GRPO across four benchmarks. It improves both solution quality and solution diversity.

  • 5.1 Setup: The experiments train Qwen3-4B-Base and Qwen3-14B-Base on 10,000 filtered DeepscaleR examples and compare DARLING with GRPO.
  • 5.1 Setup: The competition-math evaluation uses AIME25, OlympiadBench, HMMT 2025, and Brumo 2025.
  • 5.1 Setup: The pass@k metric counts correct generations among k sampled responses, with c denoting the number of correct generations.
  • DARLING outperforms GRPO on pass@1 by +3.51% and +1.90% averaged across four benchmarks for 4B and 14B models, respectively.
  • DARLING outperforms GRPO on pass@128 by +7.62% and +10.16% averaged across four benchmarks for 4B and 14B models, respectively.

6 Ablations

The ablations test reward aggregation, diversity measurement, and GRPO normalization. They support multiplicative semantic-diversity rewards and show that normalization and lexical rewards can be problematic in noisy or math settings.

  • The ablations compare additive versus multiplicative reward aggregation, semantic versus lexical diversity, and the GRPO standard-deviation term.
  • 6.1 Ablations on Multiplicative v.s. Additive Aggregation: Multiplicative quality–diversity aggregation outperforms additive aggregation on AlpacaEval 2.0 and performs similarly on ArenaHard v2.0 and NoveltyBench.
  • 6.2 Ablations on Lexical Metrics for Diversity: DARLING’s semantic diversity assessment outperforms 4-gram diversity on NoveltyBench, although 4-gram diversity matches its quality on LM-as-a-Judge evaluations.
  • 6.2 Ablations on Lexical Metrics for Diversity: Using 4-gram diversity as a reward underperforms GRPO on competition-math pass@1, with generations sometimes exploiting the reward through different-language text or self-reflections.
  • 6.3 Ablations on Advantage Normalization in GRPO: Removing reward normalization improves both quality and diversity in dense, noisy non-verifiable settings.
  • 6.3 Ablations on Advantage Normalization in GRPO: Normalization has little effect when rewards are sparse, binary, and deterministic, as in the reported math setting.

7 Related Work

Related work addresses language-model diversity through training-time objective changes and inference-time decoding strategies. DARLING is situated among approaches that jointly consider quality and diversity during online RL.

  • The related-work discussion distinguishes training-time diversity strategies from inference-time decoding approaches.
  • Training-time strategies for diversity: Training-time methods modify likelihood objectives, target distributions, preference optimization, reward weighting, entropy regularization, or KL regularization to encourage diversity.
  • Inference-time strategies for diversity: Inference-time research develops diverse beam-search variants and compares them with temperature-based sampling strategies.

8 Conclusion

The paper introduces DARLING as online RL that jointly optimizes quality and diversity, preserving diversity across model families, sizes, and verifiable and non-verifiable tasks.

  • DARLING jointly optimizes quality and diversity during online reinforcement learning.
  • Experiments across model families and sizes demonstrate effectiveness on both verifiable and non-verifiable tasks.
  • Unlike prior RL approaches described as often causing diversity collapse, DARLING preserves diversity in model generations.

A.1 Classifier for Non-verifiable Tasks

The paper builds semantic-diversity classifiers for both non-verifiable responses and math solution traces, evaluating them against human or model judgments. These classifiers support diversity measurement beyond lexical variation in training and evaluation.

  • Non-verifiable task classifier: 1,100 prompts and 2,200 responses supply human annotations for training a semantic-similarity classifier.The two responses are concatenated and classified as semantically similar or not.
  • Non-verifiable task classifier: 78% held-out accuracy for the trained ModernBERT classifier closely matches the NoveltyBench classifier’s 79%.Evaluation uses 100 held-out prompts, corresponding to 200 responses.
  • Non-verifiable task classifier: Classifier-based approaches outperform proprietary models on 100 human-annotated examples when judging semantic similarity.Figure 7 compares different classifier approaches with proprietary models.
  • Verifiable task classifier: 89% held-out accuracy is achieved by the classifier trained to distinguish different strategies in competition-math solution traces.Llama-3.3-70B-Instruct provides the similarity judgments used as ground truth.

C.2 Example generations in NoveltyBench

NoveltyBench examples show that DARLING broadens parallel generations beyond repeated answers, especially by varying the chosen entity or adding distinct informative framing. The examples illustrate diversity qualitatively across animal and number prompts.

  • Number prompt: Four baseline generations for the number prompt repeat 53, whereas DARLING produces 53, 73, and 67 with distinct explanations.DARLING’s examples frame number selection through randomness, bias avoidance, or generation criteria.
  • Animal prompt: Four baseline generations for “Pick a random animal” all choose the quokka with near-identical wording.The outputs repeat the same animal and closely related phrasing.
  • Animal prompt: DARLING generates an axolotl, quokkas with different factual elaborations, and a narwhal across the same four animal generations.The outputs differ in both selected animals and the information used to describe them.
  • Benchmark context: NoveltyBench measures diversity with a partition classifier and response quality with a reward model, while the paper primarily uses the human-trained distinct classifier.The reward model is described as vulnerable to reward hacking.

E Full Results on Math

The paper reports full competition-math results for Qwen3-4B-Base and Qwen3-14B-Base. The associated tables present pass@k performance through pass@128.

  • Qwen3-4B-Base: Table 9 reports full math results for Qwen3-4B-Base.Its values represent pass@k performance up to pass@128.
  • Qwen3-14B-Base: Table 10 reports full math results for Qwen3-14B-Base.Its values represent pass@k performance up to pass@128.

F Removing Normalization in Math Tasks

In math tasks, normalization has little effect when rewards are binary, sparse, and noise-free, while n-gram diversity can be exploited by generating irrelevant reflections after the final answer.

  • Normalization ablation: Normalization has little effect on GRPO across four competition-math benchmarks in the reported setting.Table 11 compares GRPO with and without dividing by standard deviation.
  • Normalization ablation: Binary, sparse, and noise-free rewards explain why removing normalization has little to no effect in this math setting.This contrasts with settings where reward normalization may matter more.
  • Reward hacking: The model hacks the n-gram diversity reward by adding irrelevant reflections after the final answer.The added text reflects on its performance and the question’s difficulty rather than solving the problem.
  • Reward-design context: Online diversity rewards must account for semantic equivalence because lexical metrics can reward superficial variation.Prior diversity measures include distinct n-grams, Self-BLEU, embedding similarity, and LM judgments; LM judges add online-training overhead.
Loading 2509.02534v1…