Source-linked AI summary
Where does output diversity collapse in post-training?
Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras
TL;DR
Output diversity collapse is poorly separated between training-data composition, post-training method, generation format, and learned weights. The paper traces Olmo 3 lineages across 15 tasks and four metrics, finding that data composition determines collapse trajectories, while CoT suppression reduces accuracy without restoring answer diversity. The task-dependent split between error removal and narrowing among correct outputs shows why diversity effects differ by task.
Problem
Prior work does not isolate training-data composition from post-training method or generation format from model weights when explaining output diversity collapse.
Method
The study compares Think, Instruct, and RL-Zero Olmo 3 lineages across 15 tasks and four diversity metrics, including CoT-suppressed Think evaluations and quality-filtered diversity.
Results
Diversity collapse co-varies with data composition: Think drops most at SFT, Instruct drops more at DPO, and CoT suppression lowers accuracy without recovering diversity.
Takeaways & Limitations
Diversity collapse is determined during training by data composition and cannot be addressed through inference-time generation-format changes alone.
Takeaways & Limitations
The study measures distributional diversity but cannot determine which perspectives are lost, limiting conclusions about representational diversity.
Abstract
from arXiv · showhide
Post-trained language models produce less varied outputs than their base counterparts. This output diversity collapse undermines inference-time scaling methods that rely on varied samples, and risks homogenizing model outputs on creative and value-laden tasks. Prior work attributes collapse to specific post-training methods, without separating the role of training data composition from the method, or the generation format from the model weights. We trace output diversity through three parallel post-training lineages of Olmo 3, Think (chain-of-thought distillation), Instruct (broad multi-source data), and RL-Zero, across 15 tasks and four text diversity metrics. We find that the location of collapse co-varies with data composition: the Think lineage loses most semantic diversity at supervised fine-tuning, and the effect of DPO is larger in Instruct than in Think. Suppressing chain-of-thought reasoning at inference in Think models drops accuracy on hard tasks, yet leaves answer-level diversity unchanged, showing that the collapse is embedded in the model weights by training data, not imposed by the generation format. Decomposing diversity loss on six verifiable tasks into a quality-control component (removal of incorrect outputs) and a residual component (genuine narrowing among correct outputs) reveals that the split is task-dependent, and Think models retain more correct-answer diversity than Instruct despite collapsing more in aggregate. Our results indicate that diversity collapse is determined during training by data composition and cannot be addressed at inference time alone.
1 Introduction
Post-training improves helpfulness, safety, and instruction compliance but is associated with output diversity collapse. Prior work links collapse to particular algorithms without isolating the role of training-data composition.
- Post-training combines supervised fine-tuning with preference optimization or reinforcement learning to improve helpfulness, safety, and instruction compliance.
- Output diversity collapse produces more uniform outputs than base models across summarization, reasoning, and open-ended generation.
- This collapse limits self-consistency, pass@k sampling, and test-time compute scaling because these methods rely on varied outputs.
- Prior studies attribute collapse to specific algorithms, including DPO in narrative generation, reward steps in creative tasks, and SFT in reasoning, without examining data composition.
KEY EXPERIMENTS LINEAGE & DATA FINDINGS
The paper separates training method from data composition and generation format from learned weights using parallel Olmo 3 lineages. It finds that collapse location depends on data, while suppressing CoT at inference reduces quality without restoring diversity.
- Study design: The study traces output diversity through three parallel Olmo 3 post-training lineages to locate, explain, and quantify diversity loss.
- Research questions: The paper asks whether collapse follows post-training method or data composition, and whether CoT format or model weights constrain diversity.
- Controlled comparison: Think and Instruct share SFT→DPO→RL, differ in data, and are compared with RL-Zero, which bypasses SFT and DPO, across 13 models, 15 tasks, and four metrics.
- Lineage findings: Think loses more diversity at SFT, whereas Instruct experiences a larger DPO drop, associating collapse location with upstream data composition.
- Generation format: Suppressing CoT at inference does not recover diversity on any task–stage combination, although quality drops, indicating collapse resides in model weights rather than generation format.
2 Related work
Related work connects diversity collapse to alignment methods, optimization mechanisms, data recursion, and formatting choices. Proposed mitigations target optimization, regularization, preference learning, training data, and decoding.
- Aligned models show output homogeneity across model families and scales, while RLHF reduces diversity within and across inputs.
- In reasoning, SFT improves pass@1 but degrades pass@k, illustrating a reliability–diversity tradeoff in post-training.
- Proposed mechanisms include DPO gradient imbalance, likelihood displacement, KL amplification, chat-template effects, and recursive synthetic-data training.
- Suggested mitigations include forward-KL optimization, entropy-constrained RL, decoupled regularization, game-theoretic SFT, diversity-aware preference optimization, and conformative decoding.
3 Experimental setup
The experiments compare Olmo 3 checkpoints across parallel post-training lineages, diverse task families, four diversity axes, quality measures, and quality-filtered diversity under controlled generation settings.
- Models and lineages: The study evaluates 13 Olmo 3 checkpoints at 7B scale, starting from the same base model and applying up to SFT, DPO, and RL stages.
- Models and lineages: Think uses about 2.3M synthetic CoT traces from two teachers, followed by about 200K DPO pairs and RL with verifiable rewards.
- Models and lineages: Instruct initializes from Think-SFT, then uses about 2.2M multi-source examples and about 260K DPO pairs before the same RL stage.
- Models and lineages: RL-Zero applies RL directly to Base, bypassing SFT and DPO, across math, code, instruction-following, and general reward domains.
- Tasks: Evaluation covers summarization, code, reasoning, instruction following, creative writing, and value pluralism across 15 tasks.
- Data checks: Training–evaluation overlap is negligible for nine datasets and elevated to 7–30% for six flagged benchmarks, while contaminated-task findings remain consistent with clean-task patterns.
- Metrics: Diversity uses lexical EAD and semantic SBERT measures, with additional code-task semantic metrics and quality-filtered all-output versus correct-output scores.
- Generation and scoring: Each model–task pair generates 16 outputs at T=0.6 and top-p=0.95, with Think reasoning traces stripped before diversity and quality scoring.
4 Results
Diversity collapse co-varies with training data composition: Think loses most diversity at SFT, while Instruct loses more at DPO, and both converge toward similar final floors. Inference-time CoT suppression does not restore answer diversity, while quality-filtered analyses show that genuine narrowing among correct outputs varies by task.
- Lineage-dependent collapse: Think-SFT loses 62% of Base diversity on average, compared with 38% for Instruct-SFT across 15 tasks.The asymmetry is consistent with the narrower two-teacher SFT data used for Think.
- Lineage-dependent collapse: DPO removes more diversity in Instruct than Think, while Think’s RL stage modestly recovers semantic diversity on most tasks.On GSM8K, Instruct RL erases 37% of Base diversity, whereas Think enters RL near its diversity floor.
- Semantic versus lexical diversity: Final semantic diversity falls from 0.32 SBERT for Base to 0.12 for Think and 0.11 for Instruct, while EAD remains stable or increases.Vendi Score also drops from approximately 3.4 effective modes to approximately 1.8, with near-total collapse on math tasks.
- CoT suppression: Suppressing CoT leaves answer-level diversity unchanged but reduces accuracy on harder tasks, including HumanEval and MATH-Geometry by 32%.WritingPrompts is an exception, with a small SBERT increase of +0.046; suppression does not recover diversity overall.
- CoT suppression: Diversity collapse is embedded in learned model weights rather than imposed by output format, so switching Think models to direct-answer inference does not recover it.The comparison also indicates that differences between Think and Instruct reflect data composition rather than generation format.
- Quality-filtered diversity: Quality-control and genuine narrowing contribute task-dependently: genuine narrowing accounts for 83.4% of the IFEval Da drop, 38% on MBPP, and less than 10% on HumanEval.Math reasoning falls between 57–64% genuine narrowing.
- Correct-answer diversity: Think and Instruct converge to 1.3–1.6 effective Vendi modes among correct GSM8K answers, limiting majority-voting gains from repeated sampling.At matched accuracy on MATH-Algebra, RL-Zero-Math has twice Think-not-thinking’s correct-answer diversity and gains +15% versus +7% from majority voting.
5 Discussion
Diversity loss follows task-dependent paths and consequences: data composition affects when models collapse, while the final floor may remain similar. The practical significance ranges from useful error filtering to harmful homogenization, and distributional metrics do not establish which perspectives remain.
- Training dynamics: Data composition determines when and how sharply models approach the diversity floor, but not the floor itself.Think collapses mainly during SFT, whereas Instruct narrows more sharply during DPO; both converge to similar Vendi modes among correct answers.
- Mitigations: Data-level interventions can slow diversity loss but may not raise the final diversity level.Broader sources and more teachers may delay collapse, while algorithmic changes such as entropy constraints or removing KL penalties may be needed to shift the floor.
- Task dependence: Diversity loss has different meanings across tasks, ranging from removal of incorrect paths to genuine narrowing among correct outputs.The paper characterizes code correctness filtering as relatively helpful, while identifying creative writing and value-pluralism as settings where homogenization is especially concerning.
- Representational limits: Distributional diversity metrics measure output spread but cannot determine which perspectives or stances are lost.This limitation is especially important for value-pluralism tasks, where narrow training data may risk imposing a single perspective.
6 Conclusion
Across Olmo 3 post-training lineages, diversity collapse is shaped by training data composition rather than method alone and is embedded in learned weights rather than inference format. Its split between error removal and homogenization varies by task, motivating broader SFT sources and careful use of KL-free RL.
- 6 Conclusion: Diversity collapse follows training data composition: narrow distillation causes an SFT cliff, while broader data shifts the sharpest drop to DPO.Suppressing CoT reduces accuracy without recovering diversity, indicating that collapse resides in learned weights; quality-filtered decomposition shows task-dependent error removal versus genuine narrowing.
A Implementation details
The implementation evaluates sampled outputs with four complementary diversity measures across a broad task suite. Metrics cover lexical, semantic, inferential, effective-mode, and code-structural variation, alongside judged quality for non-verifiable tasks.
- Evaluation setup: Each model–task pair uses K=16 sampled outputs per prompt, with N=500 prompts except tasks evaluated on their full datasets.Outputs use a 32,768-token generation limit, and all diversity metrics operate on the same post-stripping text.
- Evaluation setup: The evaluation spans 15 tasks grouped into reasoning, code, instruction following, summarization, creative writing, and other quality settings.Tables report task groupings and corresponding quality or diversity results across the benchmark suite.
- Diversity metrics: EAD measures lexical diversity through unique n-grams normalized by expected uniqueness under a uniform vocabulary draw.The score averages n-grams for n ∈ {1, . . . , 5}, with tokenizer vocabulary size supplying the vocabulary scale and length bias corrected by normalization.
- Diversity metrics: SBERT diversity is the mean pairwise cosine distance between L2-normalized all-mpnet-base-v2 output embeddings.Values near 0 indicate semantic collapse, while values near 1 indicate highly dissimilar outputs; UniXcoder is additionally used for code.
- Diversity metrics: NLI diversity converts pairwise entailment and contradiction probabilities into a symmetrized disagreement score, excluding code tasks.Scores near 0 indicate mutual entailment, values near 1 neutrality, and values above 1 net contradiction.
- Diversity metrics: Vendi Score estimates the effective number of dissimilar outputs from eigenvalue entropy of the SBERT similarity kernel.It ranges from 1 for identical outputs to K for orthogonal outputs, but agreement with SBERT is expected because both reuse the same kernel.
- Code metrics: Code diversity uses mean pairwise AST-subtree Jaccard distance, computed from parsed subtree multisets on correct outputs only.Subtrees are extracted up to height 4; unparseable outputs are excluded.
- Quality evaluation: Non-verifiable tasks use gpt-4.1-mini LLM-as-judge comparisons against references or Base-model outputs.The evaluation covers summarization, instruction following, value pluralism, and creative writing with task-specific judging prompts.
D Quality results
Quality results are reported with task-specific accuracy, pass, consistency, win-rate, and judged-score measures, while diversity results cover four general metrics and quality-filtered correct-output analyses. The latter separate overall spread from diversity retained among correct solutions.
- Non-verifiable-task quality: Summarization and open-ended quality use pairwise win rates, while WildBench uses checklist-guided WB-Score.Judgments are produced with gpt-4.1-mini against references or the Base model, depending on task.
- Verifiable-task quality: Reasoning quality is reported with accuracy@1, majority vote@16, and pass@16 across four reasoning tasks.Code quality uses pass@k for k ∈ {1, 5, 10, 16}, while IFEval reports strict and loose accuracy@1, pass@16, and consistency@16.
- Quality-filtered diversity: Quality-filtered diversity reports accuracy, SBERT on all outputs, SBERT on correct outputs, and Vendi Score on correct outputs.For six verifiable tasks, K=16 generations are labeled correct or incorrect using answer matching, test execution, or constraint satisfaction.
- Quality-filtered diversity: Code-specific diversity is measured on correct outputs with UniXcoder SBERT and AST-subtree Jaccard distance.Missing entries indicate models with no parseable correct outputs.
G Output length analysis
Output length does not systematically predict diversity: tasks with similar lengths can have very different diversity, while highly diverse tasks span varied output lengths.
- GSM8K and TruthfulQA average 137 and 142 words but have SBERT diversities of 0.128 and 0.262, respectively.The comparison shows that similar output lengths can coincide with substantially different diversity levels.
- WritingPrompts and HumanEval exhibit high mean diversity despite spanning a wide range of output lengths.
- Base-model SBERT diversity decreases 11% on average under the matched temperature, compared with a 62% drop from Base to Think-SFT.EAD decreases 18% and NLI 3% under the temperature change, indicating the temperature difference is modest relative to the documented SBERT gap.
I Stage attribution per task
Diversity loss varies sharply by post-training lineage and stage. Think loses most diversity during SFT, whereas Instruct experiences substantial additional loss during DPO, while RL-Zero generally preserves or exceeds Base diversity.
- Stage attribution per task: Think loses 45–80% of Base SBERT diversity at SFT, with DPO contributing minimally across tasks.The largest Think loss occurs on XSum and the smallest on IFEval.
- Stage attribution per task: Instruct’s SFT losses range from 8–73%, while DPO adds a further 2–63% diversity loss.This stage pattern contrasts with Think, where most collapse occurs during SFT.
- Stage attribution per task: RL-Zero retains 71–105% of Base diversity across tasks at the final checkpoint.Table 18 defines Retain as the fraction of Base diversity preserved at the final checkpoint.
- Stage attribution per task: Training–evaluation overlap is measured with C13 13-gram matching as the fraction of test tokens covered by matching training phrases.The overlap results compare four Dolci post-training datasets with fifteen evaluation benchmarks.
- Stage attribution per task: Table 18 attributes per-task diversity changes as percentages of Base SBERT diversity lost or recovered at each post-training stage.The table also reports final-checkpoint retention, enabling stage-level comparison across all 15 tasks.