PaperlayerView exact revision on arXiv

Source-linked AI summary

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer, Liwei Jiang, Natasha Jaques

arXiv:2609.14896v1cs.CLcs.AIcs.LGcs.MA

TL;DR

Alignment can concentrate generation on a few dominant responses, reducing diversity despite other similarly rewarded alternatives. MoDA uses role-conditioned reinforcement learning and a prompt-adaptive quality gate to optimize quality and diversity together. It reports higher diversity and average capability pass@1 than the initial policy, and higher average pass@1 than DivPO.

  • Problem

    Alignment training can concentrate outputs on a few dominant responses, reducing diversity despite alternatives with comparable rewards.

    Source: §1 · p. 1, §3.1 · pp. 4–5

  • Method

    MoDA trains a shared policy conditioned on abstract roles and rewards diversity only among responses that pass a prompt-specific quality gate.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    10.3 percentage points higher average pass@1 than the initial policy and 7.0 points higher than the strongest DivPO baseline, alongside 155.5% higher SBERT diversity than the initial policy on the diversity-focused suite.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MoDA is a drop-in replacement for post-training pipelines without architectural changes and supports switching between single high-confidence answers and diverse plausible responses.

    Source: §6 · pp. 11–12

    SkyReward-V2 favors verbose responses, and the uniform length penalty does not sufficiently address prompts of varying complexity.

    Source: §6 · p. 12

Abstract

from arXiv · show

A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.

1 Introduction

Alignment training can narrow LLM outputs, limiting applications that depend on exploration and creativity. MODA combines role-conditioned diversity rewards with a prompt-adaptive quality gate and improves diversity while retaining or improving general capability scores.

1 Introduction

  • Alignment can cause mode collapse, narrowing outputs despite plausible alternatives; this loss matters for scientific hypothesis exploration and creative, open-ended applications.Inference-time interventions can partly mitigate diversity loss, while the paper motivates addressing it during post-training.Source: §1 · pp. 1–2
  • MODA uses abstract role tokens in one shared policy, treating roles as agents whose group-relative rewards encourage distinct outputs within the high-quality response space.Its competition-and-coordination framing is inspired by multi-agent reinforcement learning [37].Source: §1 · p. 2
  • A prompt-adaptive gate grants diversity rewards only above a quality threshold computed from a frozen reference policy, limiting reward hacking through irrelevant outputs.Among responses passing the gate, rewards reflect semantic distance from the nearest neighbor.Source: §1 · p. 2
  • The 11-task suite combines seven general-capability benchmarks with four scientific-ideation and creative-writing diversity applications.The diversity applications include HypoBench, PreScience, NoveltyBench, and held-out Infinite-Chat prompts.Source: §1 · p. 2
  • On diversity-focused benchmarks, MODA raises SBERT diversity 155.5% and E-Vendi 94.0% over Qwen3-8B, while average pass@1 rises 10.3 points vs initial and 7.0 vs DivPO.These results use the same thinking-disabled setting.Source: §1 · p. 3
  • Explicit mode tokens allow diverse-set generation when supplied and recover the base model’s standard behavior when omitted.This distinguishes MODA from methods that bake diversity into the model unconditionally [33] [7] [49].Source: §1 · p. 3

2 Related work

Prior work pursues LLM diversity through measurement, prompting, decoding, and training, but quality and diversity are often evaluated separately. MODA is framed as a quality-constrained, role-conditioned alternative that supports switching between a single answer and diverse outputs.

2 Related work

  • Output diversity matters for scientific exploration, creative writing, and mathematical proofs, where distinct hypotheses, ideas, or reasoning paths are valuable.The passage links mode collapse to reduced exploration in solution spaces and diminished creativity in LLM-assisted writing.Source: §2 · p. 3
  • Most evaluations measure diversity or quality in isolation, whereas this work jointly evaluates capability and diversity across real-world applications.The passage frames joint evaluation as checking for multiple plausible answers without mode collapse or catastrophic forgetting.Source: §2 · p. 3
  • Diversity interventions span prompting, decoding, and training, including verbalized sampling, adaptive sampling, prompt ensembles, and post-training reinforcement learning.The cited examples include Verbalized Sampling [72], DIPPER, min-p sampling, and Multi-answer RL [49].Source: §2 · p. 3
  • DivPO applies DPO to responses that are both diverse and above a quality threshold [33].It is one of several post-training approaches described as targeting response variety.Source: §2 · pp. 3–4
  • Unlike other joint quality-diversity RL methods, MODA uses role-conditioned agents for per-role credit under quality constraints and supports switching between one answer and diverse responses.The related methods include DARLING, DQO, GRPO-Unlikeliness, and GAPO; the passage says none allow that switching.Source: §2 · p. 4

3 MODA: Mode-Conditioned Diversity Alignment for LLMs

The section frames mode collapse as concentration on a few high-reward responses and proposes role-conditioned shared-policy training to preserve diverse, high-quality outputs. A prompt-adaptive quality gate restricts diversity rewards to responses meeting a prompt-specific threshold, and GRPO optimizes the shared policy.

3.1 Preliminaries: LLM RL Post-Training and Mode Collapse

  • Optimizing expected reward alone concentrates the policy on a few high-reward responses, causing mode collapse despite comparable alternatives.The section defines mode collapse as repeatedly generating a few dominant responses while reducing policy entropy.Source: §3.1 · pp. 4–5

3.2 Multi-Agent Reinforcement Learning (MARL)-Inspired Mode Conditioning

  • A shared policy conditions on abstract role identifiers, treating roles as agents that learn complementary behaviors; omitting roles retains standard single-response behavior.The primary conditioning uses numbered identity-based roles rather than hand-crafted personas, supporting emergent specialization.Source: §3.2 · p. 5

3.3 Quality-Gated Diversity Reward

  • Diversity credit is gated by a prompt-adaptive quality threshold, while nearest-neighbor semantic distance rewards distinct responses and penalties discourage superficial diversity.Responses below the threshold still receive a quality-learning signal but no diversity credit; the diversity measure compares each response with its closest alternative.Source: §3.3 · pp. 5–6

3.4 Policy Optimization

  • GRPO updates the shared policy using within-group reward comparisons and a KL term that controls divergence from the base language model.The method estimates advantages without a separate value network.Source: §3.4 · p. 6

4 Experiments

The experiments describe MODA’s training setup, comparison methods, and evaluation protocol. They assess diversity on scientific-ideation and creative-writing tasks and capability on seven standard benchmarks.

4 Experiments

  • MODA uses Qwen3-8B-Instruct and trains in thinking-enabled and thinking-disabled settings, sampling six role-conditioned responses per prompt for each reward group.Quality and diversity rewards are computed on final answers with thinking traces masked.Source: §4 · pp. 6–7

4.1 Baselines

  • Comparisons include training-time DARLING and DivPO, repeated base-model sampling, and SSoT prompting; inference-time methods can also be applied alongside MODA.DARLING is evaluated with both the paper’s mixture data and its specified WildChat recipe.Source: §4.1 · p. 7

4.2 Training Data

  • The 10K-prompt training set mixes Tulu3-SFT-Mixture and Infinite-Chat prompts in a 4:1 ratio to balance instruction following and open-ended exploration.Source: §4.2 · p. 7

4.3 Generative Diversity Evaluation

  • Diversity evaluation covers HypoBench and PreScience scientific ideation plus NoveltyBench and held-out Infinite-Chat creative writing, using SBERT, E-Vendi, and discriminator metrics.Evaluation datasets do not overlap with training data; HypoBench prompts hypothesis generation, while PreScience requests a title–abstract pair for a future contribution.Source: §4.3 · pp. 7–8

4.4 General Capability Retention Evaluation

  • General capability retention is evaluated in standard single-response mode across seven benchmarks covering reasoning, knowledge, question answering, commonsense, truthfulness, and instruction following.The benchmarks are GSM8K, MMLU, GPQA, BoolQ, HellaSwag, TruthfulQA-MC1, and IFEval.Source: §4.4 · p. 8

5 Results

Across settings and base-model configurations, MODA improves diversity and quality simultaneously and achieves the strongest average capability retention among baselines. Qualitative examples and ablations show broader variation from numbered-role training, with quality gating helping preserve performance.

5 Results

  • MODA has the best average diversity across four domains and three model families, matches or improves quality on most domains except PreScience, and outperforms baselines on out-of-domain HypoBench.The training set primarily consists of standard SFT data and open-ended questions.Source: §5 · p. 8
  • With Qwen3-8B thinking disabled, MODA matches or exceeds the base model on 5 of 7 benchmarks and leads baselines on four pass@1 tasks; language-mixing penalties help maintain quality.With GLM-4-9B, MODA improves average pass@1 by 15.8 percentage points, including 3.89x on GSM8K and 2.37x on MMLU.Source: §5 · p. 8
  • MODA Pareto-dominates all baselines on Infinite-Chat across settings and base models, while achieving the best average capability pass@1, pass@5, and pass@10.Across four domains and three model families, MODA also has the best average diversity, while matching or improving quality on most domains except PreScience.Source: §5 · p. 8, Figure 4 · p. 9, Table 2 · p. 9

5.1 Qualitative Results

  • Qualitative examples show a larger embedding region for a time-metaphor prompt and more valid, distinct blue-shade responses; numbered roles sometimes acquire role-specific behaviors.The time-metaphor comparison includes 23 off-the-shelf models.Source: §5.1 · p. 10, Figure 6 · p. 11

5.2 Ablation Studies.

  • MODA has the best general-suite accuracy and exceeds role prompting by 77% in SBERT and 41.6% in E-Vendi diversity; diversity-only maximizes diversity but collapses under diverse decoding.The quality-only variant regresses in diversity relative to role prompting; the authors identify explicit quality gating as critical to preserving response quality.Source: §5.2 · p. 10
  • Additive rewards at λd=10 and λd=100 improve diversity but reduce general capability under diverse decoding; MODA’s gate reduces sensitivity to coefficient scaling and enables faster iteration.At λd=1, the additive variant shows modest diversity improvement over the baseline, substantially less than MODA.Source: §5.2 · pp. 10–11
  • Crafted-persona, single-role, and token-based variants show smaller diversity gains than MODA, whose multiple abstract numbered roles provide a more effective exploration space without sacrificing generalization.All ablated variants show slight diversity improvement over the base model.Source: §5.2 · p. 11

6 Discussions

MODA is presented as a drop-in alignment method with no architectural changes, offering a choice between diverse responses and standard behavior. The discussion also identifies reward-model verbosity, policy capacity, and high-stakes output risks as limitations or future-work concerns.

6 Discussions

  • MODA is a drop-in replacement for existing post-training alignment pipelines and requires no architectural modifications.Its dual-mode design switches between a single high-confidence answer and a diverse set of plausible responses.Source: §6 · pp. 11–12
  • The dual-mode design offers flexibility for downstream needs, including high-stakes domains such as medical diagnosis, legal reasoning, and scientific ideation.The discussion cautions that varied outputs may be misleading, unsafe, or unsupported, and suggests pairing diverse mode with verification or other safeguards.Source: §6 · p. 12
  • The quality reward model can favor verbosity, diversity is bounded by the initial policy’s capacity without external data, and varied outputs may be misleading or unsafe.The authors propose adaptive length penalties or alternative reward models, external data or retrieval, and further safeguards as future directions.Source: §6 · p. 12

Appendices

The appendices outline MODA’s two-stage training procedure, report rollout and compute settings, and describe role-conditioning ablations.

Appendices

  • Algorithm 2 precomputes prompt-specific reference quality statistics, then samples mode-conditioned responses and computes gated rewards before GRPO updates.The algorithm lists quality, diversity, and reward-hacking terms, followed by group-relative advantages and KL regularization to the reference policy.Source: Appendices · pp. 20–21

A.1 Overview

  • Rollouts use temperature 1.0, top-p 1.0, and k = 6 responses per prompt; Qwen3-8B runs used four epochs, while other reported runs used two.A two-epoch run took about 24 hours on four H200 GPUs; reward weights and language-mismatch penalties varied by model.Source: §A.1 · p. 22

A.2 Ablation Model Implementations

  • The ablation implementations replace abstract numbered roles with dummy tokens, crafted role descriptions, or a single role instruction.For dummy-token ablations, the tokens were stripped before diversity and quality metrics were computed.Source: §A.2 · p. 22

A.3 Dataset

The appendix describes the prompt mixture used for training and the construction and architecture of a pairwise response-diversity discriminator.

A.3 Dataset

  • Training uses 8,000 general-purpose prompts from Tulu-3 SFT and 2,000 open-ended Infinite-Chat prompts, with some Infinite-Chat prompts held out for evaluation.Table 3 is identified as containing MODA training hyperparameters, but its values are not supplied here.Source: §A.3 · p. 22, Table 3 · p. 23

A.4 Discriminator

  • The pairwise discriminator is trained on labeled response pairs and uses frozen sentence embeddings with a three-layer MLP classifier.Its 5,816 filtered pairs are split by prompt into 4,064/890/862 train, validation, and test examples; the model has approximately 804K trainable parameters.Source: §A.4 · pp. 23–24

A.5 Training Curve

Figure 8 presents ablation training diagnostics across role-injection and reward settings, with thinking disabled or enabled.

A.5 Training Curve

  • The displayed settings include no role injection and quality-only reward, each with thinking disabled and enabled.These labels distinguish role-injection and reward ablations in the training curves.Source: §A.5 · p. 25
  • The displayed settings also include diversity-only reward and crafted personas, each with thinking disabled and enabled.The caption identifies the corresponding curves as ablation training diagnostics.Source: §A.5 · p. 25, Figure 8 · p. 25
  • Figure 8 reports total reward, diversity reward, quality reward, and answer token length for each ablation setting under both thinking conditions.The supplied caption describes the plotted diagnostics and layout but does not state their outcomes.Source: §A.5 · p. 24, Figure 8 · p. 25

B.1 Generative Diversity Evaluation.

The appendix reports generative-diversity variability and general-capability retention using pass@1 and pass@5, with task-specific pass@k figures.

B.1 Generative Diversity Evaluation.

  • Error bars are reported for generative diversity across all domain application tasks.Source: §B.1 · p. 26

B.2.1 Pass@1

  • Pass@1 retention is shown across baselines and the authors’ model, while Figure 9 specifies Qwen3-8B base and thinking-disabled results on BoolQ, MMLU, GPQA, and IFEval.Source: §B.2.1 · p. 27, Figure 9 · p. 29
  • Table 4 reports general-capability retention as accuracy percentages with mean and standard error.Source: Table 4 · p. 28

B.2.2 Pass@5

  • Pass@5 retention is reported across baselines and the authors’ model, with additional figures for Qwen3-8B, Llama-3.1-8B, and GLM-4-9B settings.Source: §B.2.2 · p. 28, Figure 10 · p. 29, Figure 11 · p. 29
  • Table 5 reports pass@5 and strict IFEval instruction-following accuracy, with averages across benchmarks within each thinking setting.Source: Table 5 · p. 28

B.3 Ablation Results

The ablation results report general-capability performance under standard and diverse decoding alongside Infinite-Chat diversity; the accompanying metrics section describes diversity measures and capability-retention evaluation.

B.3 Ablation Results

  • The ablation table covers general-capability pass@1 under standard and diverse decoding and Infinite-Chat diversity across three random seeds.Source: Table 6 · p. 30
  • The ablation figure presents pass@k accuracy by benchmark for ablated models under standard versus diverse decoding.Source: Figure 13 · p. 31

C.1 Diversity Metrics

  • Diversity evaluation uses response sets generated per prompt, averages metrics over prompts, and for thinking-enabled models scores only final answers after removing reasoning traces.Source: §C.1 · p. 32
  • The metrics include surface-form and semantic diversity, plus application-domain SBERT, E-Vendi, and learned discriminator measures.Source: §C.1 · p. 32
  • E-Vendi normalizes a response-set similarity matrix by its trace and uses its eigenvalues; its score represents the effective number of distinct semantic responses.Source: §C.1 · p. 32

C.2 General Capability Retention

  • Capability retention is tested in standard mode across seven benchmarks, using each benchmark’s native scoring rule.Source: §C.2 · p. 32

C.3 Domain Application Tasks

The domain application evaluation tests diversity in open-ended scientific and creative generation, using four tasks and ten samples per model; accompanying examples contrast baseline and role-conditioned generations.

C.3 Domain Application Tasks

  • The four-task domain suite samples ten responses per model and rotates role-conditioned models through their available roles.Native task quality is used when available, while diversity metrics are averaged first within each domain and then across the four domains.Source: §C.3 · p. 33
  • HypoBench scores generated scientific hypotheses for clarity, novelty, and plausibility across real-world and synthetic datasets.The quality metric averages the three judge scores across scored hypotheses.Source: §C.3 · p. 33
  • PreScience evaluates scientific follow-up prediction by comparing generated titles and abstracts with ground-truth follow-up papers using its LACER-style scoring pipeline.The reported quality metric is the average score across generations from ten research contexts.Source: §C.3 · p. 33
  • NoveltyBench scores multiple generations for creative prompts and reports average utility, weighting earlier generations more heavily and discounting later redundant ideas.Source: §C.3 · p. 33
  • Infinite-Chat uses 100 held-out open-ended prompts and substitutes average general-capability pass@1 for a native quality metric.Source: §C.3 · p. 33

D Generation Examples

Related papers

Loading 2609.14896v1…