PaperlayerView exact revision on arXiv

Source-linked AI summary

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R. Tomz, Christopher D. Manning, Weiyan Shi

arXiv:2510.01171v4cs.CLcs.AI

TL;DR

The paper asks why post-training alignment reduces language-model diversity and identifies typicality bias in preference data as a pervasive data-level driver. It formalizes and empirically validates this account, then introduces training-free Verbalized Sampling, which prompts models to verbalize response distributions. Across tasks, VS improves diversity and performance while preserving factual accuracy and safety.

  • Problem

    Post-training alignment can cause mode collapse, and the paper addresses limited explanations that focus mainly on algorithmic causes by examining typicality bias in preference data.

  • Method

    The paper formalizes typicality bias, verifies it on preference datasets, and introduces Verbalized Sampling, which asks models to generate candidate responses with corresponding probabilities.

  • Results

    VS significantly improves diversity and the quality-diversity trade-off across tasks and model families without compromising factual accuracy or safety; creative-writing diversity increases by 1.6-2.1× over direct prompting.

  • Takeaways & Limitations

    Mode collapse can be mitigated at inference time through prompting, and aligned models retain significant inherent diversity that VS can recover.

  • Takeaways & Limitations

    Base-model log-probability is an imperfect proxy for psychological typicality, although its estimated typicality coefficient remains positive and statistically significant after surface-form controls.

Abstract

from arXiv · show

Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling, a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., "Generate 5 jokes about coffee and their corresponding probabilities"). Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

1. Introduction

The paper identifies typicality bias in preference data as a data-level driver of mode collapse and introduces Verbalized Sampling, which prompts models to verbalize response distributions to recover diversity. Experiments across multiple tasks show improved diversity and quality without compromising factual accuracy or safety.

  • Problem: Mode collapse narrows models toward a small set of responses, reducing diversity and limiting applications including creative writing, social simulation, pluralistic alignment, and synthetic data generation.
  • Problem: Typicality bias—the preference for familiar, fluent, and predictable text—is identified as a fundamental data-level cause of mode collapse.The paper formalizes this mechanism and verifies it empirically on preference datasets.
  • Method: Verbalized Sampling asks models to generate a distribution of responses with corresponding probabilities instead of a single instance.The method is training-free and is motivated by typicality-like preferences across multiple preference datasets.
  • Method: VS’s distribution-level prompting tends to recover diversity associated with the underlying base model rather than the stereotypical mode elicited by instance-level prompts.
  • Empirical gains: VS significantly improves diversity across creative writing, dialogue simulation, synthetic data generation, and open-ended QA, while maintaining factual accuracy and safety.The experiments also report more human-like dialogue behaviors and broader practical possibilities such as richer exploration and hypothesis generation.
  • Empirical gains: 1.6-2.1×: VS boosts creative-writing diversity over direct prompting while improving the quality-diversity trade-off across tasks and model families.Creative-writing human evaluation scores also improve by 25.7%, and VS recovers 66.8% of the base model’s diversity.

2. Related Work

Related work attributes mode collapse to alignment and decoding mechanisms, while this paper adds a data-driven perspective centered on typicality bias and theoretically grounded verbalization.

  • Mode collapse and alignment: Prior studies report that alignment reduces output diversity and creative capacity relative to base models.
  • Mode collapse and alignment: Existing explanations emphasize inadequate reward models and KL-regularized optimization that amplifies common, majority-style responses.Supervised fine-tuning overfitting and rigid chat templates are also described as restricting diversity.
  • Methods to improve diversity: Earlier diversity interventions include training methods, decoding strategies, and prompting approaches that regulate or broaden generation.
  • Methods to improve diversity: Other work uses language models to generate response lists or verbalize knowledge, but the paper distinguishes its approach through theoretical grounding for distribution-level queries.

3. Typicality Bias Causes Mode Collapse

The paper identifies typicality bias in preference data as a data-level driver of mode collapse: human preferences favor familiar, fluent, and predictable responses, sharpening aligned-model distributions toward typical completions.

  • Typicality bias describes human preference for familiar, fluent, and predictable text, motivated by cognitive tendencies toward frequent, easily recalled, fluent, and schema-congruent content.
  • Base-model log-probability remains a positive significant predictor after surface-form controls, but is an imperfect proxy rather than a pure measure of psychological typicality.
  • 51.6–60.8% of HelpSteer comparisons favored the response with higher base-model log-probability, exceeding chance across human-annotated preference data.Across 6,874 correctness-matched pairs, estimated typicality coefficients were 0.57 ± 0.07 and 0.65 ± 0.07, both with p < 10^-14.
  • The reward model represents preference as true task utility plus α log πref(y | x) and noise, where α > 0 rewards higher-typicality responses at equal utility.
  • When true task quality is flat or approximately flat, positive typicality bias sharpens πref toward typical completions, compressing probability mass and producing mode collapse.Within an equal-quality response set, the optimum is π*(· | x) ∝ πref(· | x)^γ with γ > 1, behaving like temperature scaling.

4. Method: Verbalized Sampling

Verbalized Sampling is a training-free prompting method that asks models for representative response distributions with probabilities rather than single instances. Across diverse tasks and models, it restores output diversity while retaining quality, factual accuracy, and safety.

  • Verbalized Sampling asks the model to generate representative samples from a distribution and verbalize corresponding probabilities instead of answering with a single instance.
  • Distribution prompts create a reward gap favoring high-entropy distributions, which sharpening amplifies to recover diversity, unlike instance and list prompts that favor top modes.
  • KL divergence of the verbalized distribution from a pre-training-corpus proxy is 0.12, while experiments report substantial recovery of base-model diversity over direct prompting.
  • The method compares prompting strategies under the same budget of N generated responses; VS variants include VS-CoT and VS-Multi, with k candidates per call.
  • VS is training-free, model-agnostic, and requires no logit access, and is evaluated on closed, open, and reasoning models.

5. Creative Writing

Across poem, story, and joke generation, Verbalized Sampling consistently improves diversity over baseline prompting while preserving quality, with larger models benefiting more and diversity tunable through probability thresholds.

  • Diversity Results: VS-Standard consistently and significantly outperforms baseline methods on semantic diversity across poem, story, and joke generation.VS-CoT and VS-Multi further improve generation diversity.
  • Quality Trade-off: VS-CoT achieves the highest poem diversity while maintaining a high-quality score, approaching the Pareto front of the diversity-quality trade-off.VS-Standard quality remains comparable to other methods.
  • Scaling Trend: Larger models achieve diversity gains 1.5 to 2 times greater than smaller models across VS variants.The comparison uses GPT-4.1 and Gemini-2.5-Pro versus GPT-4.1-Mini and Gemini-2.5-Flash.
  • Diversity Tuning: VS increases diversity as the probability threshold decreases, enabling direct diversity tuning without changing decoding parameters.This behavior is shown across tasks on Gemini-2.5-Flash.
  • Post-training Effects: VS maintains higher diversity than baseline prompting across Tulu-70B post-training stages, mitigating drops after SFT and DPO.The base model diversity reference is 45.4%.
  • Human Study: In a human study, VS-Standard consistently receives higher diversity scores than Direct and Sequence while maintaining comparable quality win-rates.Ninety annotators evaluated poem, story, and joke outputs.

6. Dialogue Simulation

In persuasive dialogue simulation, Verbalized Sampling produces donation distributions and linguistic behavior more aligned with humans than direct prompting, especially for larger and reasoning-focused models, while readability remains a limitation.

  • Sampling Design: Model-decided random sampling and human-decided probability-weighted sampling best balance response quality and diversity.These choices are evaluated at each multi-turn interaction step.
  • Donation Alignment: VS simulates donation distributions more aligned with human behavior than direct prompting across models.GPT-4.1 with VS matches a fine-tuned Llama-3.1-8B simulator, while DeepSeek-R1 surpasses it on median donation simulation.
  • Donation Alignment: Larger and reasoning-focused models benefit more from VS in donation-amount alignment.The comparison includes GPT-4.1 versus GPT-4.1-mini and DeepSeek-R1.
  • Linguistic Alignment: VS outperforms direct prompting on Distinct-1/2/3 and semantic diversity, approaching fine-tuned-model and human distributions.Qualitative analysis finds more substantive responses instead of repetitive fillers.
  • Linguistic Alignment: VS readability remains higher than fine-tuned-model and human levels, indicating room for improvement.The figure treats closer values to human levels as better stylistic match.

7. Synthetic Data Generation

The synthetic-data experiments test whether Verbalized Sampling improves downstream mathematical reasoning after generating and using a 1,000-question dataset; VS improves averaged accuracy across the evaluated datasets.

  • Generation Setup: The experiment generates N = 1,000 synthetic competition math questions with k = 5 candidates per call.GPT-4.1 and Gemini-2.5-Flash generate questions, while Qwen3-32B produces reasoning trajectories and answers.
  • Fine-tuning Setup: The generated 1K dataset is used to fine-tune Qwen2.5-7B, Qwen3-1.7B-Base, and Qwen3-4B-Base.Fine-tuning uses five epochs and a learning rate of 5e-6.
  • Evaluation: Downstream evaluation uses MATH500, OlympiadBench, and Minerva Math with Math-Verify4.The reported aggregate accuracy averages results across the three datasets.
  • Results: VS and its variants improve downstream performance across the synthetic-data generation experiments.Average accuracy is reported across MATH500, OlympiadBench, and Minerva Math.

8. Conclusion

The paper identifies typicality bias in human preference data as a fundamental driver of mode collapse and introduces Verbalized Sampling as a training-free inference-time mitigation validated across diverse tasks.

  • Conclusion: Typicality bias—the tendency to prefer conventional responses—is identified as a fundamental property of human preference data underlying mode collapse.The paper formalizes this bias theoretically and validates it across multiple preference datasets.
  • Conclusion: Verbalized Sampling asks models to generate a probability distribution over candidate responses, restoring diversity learned during pretraining.The method is described as simple, principled, and training-free.
  • Conclusion: Extensive experiments report improved performance across creative writing, dialogue simulation, synthetic data generation, and open-ended QA without compromising safety or factual accuracy.The paper also reports that stronger models benefit more from VS.

Software and Data

The paper releases its code and related data with supporting documentation and online resources. It also references an external Math-Verify GitHub URL.

  • Reproducibility: The authors released all code and related data to support reproducibility.Documentation and resources are available at verbalized-sampling.com.
  • Resources: A GitHub link for Hugging Face Math-Verify is listed.

Impact Statement

The work aims to provide more varied outputs and support diverse exploration while preserving safety standards. Its evaluation and dissemination included human-subject safeguards, detailed experimental documentation, and contributions from multiple researchers.

  • Impact: Verbalized Sampling provides more varied and inspiring creative outputs without compromising safety standards.The authors also identify possible applications in hypothesis generation and reinforcement learning.
  • Broader applications: The paper identifies hypothesis generation and reinforcement learning as areas where diverse exploration may be valuable.
  • Reproducibility: The authors provide comprehensive experimental details, including inference parameters, prompts, and computing resources.Training and open-source inference experiments used an 8×H100 GPU cluster, while proprietary-model queries used official APIs or OpenRouter.
  • Human study: The human study received institutional review board approval, obtained informed consent, and collected no personally identifiable information.Data handling followed institutional and ethical standards for privacy and confidentiality.

B. Limitations

The paper identifies inference cost and model capability as important boundaries for Verbalized Sampling. It also situates VS among inference-time scaling methods and discusses bias mitigation in reward models.

  • Computational Cost and Latency: Generating N candidates increases inference latency and token usage compared with generating a single response.This may constrain VS in latency-sensitive or resource-constrained environments, despite controlled total computing budgets in the experiments.
  • Dependence on Model Scale and Capability: VS performance gains are positively correlated with model scale and capability.Less capable models may not fully handle probability estimation and structured output, occasionally degrading output quality.
  • Dependence on Model Scale and Capability: VS effectiveness is contingent on a sufficient level of underlying model capability.
  • Mitigating Bias in Reward Models: Reward-model biases can favor stereotypical outputs or preferences for properties such as length and style.The paper discusses calibration techniques as a way to produce more balanced reward models, but the supplied passage is truncated before specific details.
  • Inference-time Scaling: Conventional inference-time scaling based on repeated sampling from one prompt can remain vulnerable to mode collapse and limited diversity.VS is presented as an alternative that elicits a broader response distribution.

D.1. Empirical Insights: Typicality Bias in Preference Datasets

The paper finds that typicality bias is present in preference data and can sharpen models toward familiar responses, contributing to mode collapse. Verbalized Sampling instead asks for response distributions, recovering diversity while maintaining quality across evaluated tasks.

  • Typicality bias in preference data: 4–12 percentage points above the 50% chance baseline, base models favored human-preferred responses across preference datasets.Larger models generally showed stronger preference alignment, while smaller models were more variable.
  • Typicality bias in preference data: 1.42–1.47× higher odds of a more typical response being judged more helpful, even after accounting for correctness.Across 28,283 within-prompt pairs, controlling for correctness produced similar but slightly smaller effects.
  • Typicality bias in preference data: A positive typicality coefficient remained statistically significant after controlling for length, readability, lexical diversity, and sentence structure.Surface form explains part, but not all, of the typicality-like preference signal.
  • Theoretical account of mode collapse: As γ increases, typicality bias sharpens the reference distribution, amplifying modes and suppressing tails.The sharpening effect strengthens when typicality bias α increases or KL penalty β decreases.
  • Theoretical account of mode collapse: Under flat rewards, instance prompts converge exponentially toward the single reference-distribution mode, while list prompts concentrate on the top-k modes.The paper characterizes this as a “bestseller list” with at most uniform diversity over k items.
  • Diversity recovery with Verbalized Sampling: 91.7–100% of trials selected diverse sequences for framings asking about typical or representative samples.Representative sequences were also rated substantially more typical across model families and scales, supporting the mechanism behind Verbalized Sampling.

E.9. Safety Evaluation

The section evaluates whether Verbalized Sampling preserves safety while increasing diversity. Across safety tests and broader ablations, VS maintains strong safety, improves diversity-quality trade-offs, and outperforms baselines in several settings.

  • Safety performance: Refusal rates consistently exceeded 97% across methods, while VS remained within 0.3–0.8 percentage points of the Direct baseline.The highest baseline refusal rate was 98.60% for Multi-turn.
  • Safety performance: VS generated diverse refusal statements for harmful prompts while preserving the model’s core safety principles.Examples included direct refusals and informative warnings.
  • Post-training diversity: VS-Standard recovered about 66.8% of the base model’s original diversity after DPO and exceeded Direct prompting by 182.6%.Direct diversity dropped from 20.8% after SFT to 10.8% after DPO, whereas VS remained around 30% across stages.
  • Decoding ablations: VS-Standard maintains a better diversity-quality Pareto front as the candidate count k increases from 1 to 20.Increasing k generally raises diversity while slightly reducing quality, but VS-Standard preserves higher quality at a given diversity level.
  • Decoding ablations: VS-Standard consistently achieves a superior diversity-quality balance across temperatures on GPT-4.1 and Gemini-2.5-Flash.Higher temperatures generally increase diversity but may reduce quality.
  • Decoding ablations: VS-Standard operates on a substantially better Pareto front than baselines under min-p sampling on Qwen3-235B and Llama-3.1-70B-Instruct.It maintains high quality at diversity levels where Direct and Sequence exhibit significant quality collapse.

F.6. Ablation on Probability Tuning in VS on Open-Ended QA

Probability tuning in Open-Ended QA controls the diversity–precision trade-off of Verbalized Sampling. Moderate thresholds improve coverage and distributional uniformity, while very small thresholds can destabilize outputs; VS-Standard also offers favorable cost efficiency.

  • Probability tuning: Moderate probability thresholds improve Coverage-N for VS-Standard and VS-Multi, with performance peaking near p = 0.1 before declining at p = 0.01.Both methods consistently outperform the Sequence baseline across GPT-4.1 and Gemini-2.5-Flash.
  • Probability tuning: Lower probability thresholds generally reduce KL Divergence from the uniform distribution, indicating broader access to low-frequency knowledge.VS-Standard and VS-Multi achieve lower divergence than the Sequence baseline, although GPT-4.1 spikes at p = 0.01.
  • Probability tuning: Precision generally peaks at p = 0.9 and decreases as probability thresholds become smaller.The preferred threshold therefore depends on the required balance between response diversity and precision.
  • Efficiency: VS-Standard incurs a 1.12× cost ratio while achieving an 86% diversity gain under cost-normalized comparison.The cost ratio reflects probability annotations and longer prompts rather than issuing k separate API calls.
  • Efficiency: VS-Standard and Sequence have nearly identical costs of 1.12× and 1.11×, but diversity gains of 1.86× and 1.56×, respectively.This comparison attributes the advantage to probabilistic guidance rather than token overhead.
  • Efficiency: VS-Standard exchanges a 12% cost increase and 23% latency increase for an 86% diversity gain.The paper describes this trade-off as economically feasible for applications requiring high diversity.

G.4. Evaluation Details

The evaluation uses task-specific automated judging procedures for creative writing, jokes, commonsense reasoning, and safety. Each task applies explicit scoring criteria and constrained output formats.

  • Creative writing evaluation: Poem and story quality are evaluated by Claude-3.7 using ten writing criteria, including creativity, imagery, emotional engagement, and overall reader engagement.Scores are requested on a 0–20 scale with a designated output format.
  • Joke evaluation: Jokes are scored on relevance, comedic device, and humor quality, each using a 0–5 scale.The evaluator checks topic relevance when a prompt specifies a topic and recognizes devices such as puns, irony, and reversal.
  • Commonsense reasoning evaluation: Commonsense reasoning predictions receive one of three grades: CORRECT, INCORRECT, or NOT_ATTEMPTED.The grading prompt includes examples and rules for numerical answers, partial information, and minor name typos.
  • Safety evaluation: Safety responses are judged for refusal, convincingness, and specificity using explicit numeric rubrics.Convincingness and specificity use 1–5 scales, and the evaluator always supplies numeric scores.

H.3. Qualitative Example on Synthetic Data Generation

The qualitative synthetic-data example presents math problems generated with Verbalized Sampling across multiple mathematical domains. The examples illustrate breadth in the generated problem set.

  • Qualitative example: Table 44 contains example math problems generated through Verbalized Sampling prompting.
  • Qualitative example: The generated problems span algebra, geometry, number theory, and combinatorics.The table is presented as a cross-domain example of synthetic data generation.
  • Qualitative example: The examples cover various mathematical domains rather than a single problem type.
Loading 2510.01171v4…