PaperlayerView exact revision on arXiv

Source-linked AI summary

Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs

Minh Nhat Nguyen, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, Ravid Shwartz-Ziv

arXiv:2407.01082v8cs.CL

TL;DR

LLM decoding must balance creative diversity with coherent text, a tension that becomes more difficult at higher temperatures. The paper proposes min-p sampling, which scales a truncation threshold by the model’s top-token probability, and reports improved quality and diversity with human preference for its outputs.

  • Problem

    LLM sampling methods struggle to balance creativity and coherence, with higher temperatures often reducing coherence.

  • Method

    Min-p sampling dynamically adjusts the truncation threshold using the model’s top-token probability as a confidence scaling factor.

  • Results

    Across diverse benchmarks and model settings, min-p consistently improves generated-text quality and diversity, while human evaluations show a clear preference for its outputs.

  • Takeaways & Limitations

    Min-p offers a practical alternative for high-temperature generation and has been integrated into widely used open-source LLM frameworks.

  • Takeaways & Limitations

    Human-evaluation diversity scores are point estimates based on three samples per configuration, providing directional rather than broad estimates.

Abstract

from arXiv · show

Large Language Models (LLMs) generate text by sampling the next token from a probability distribution over the vocabulary at each decoding step. Popular sampling methods like top-p (nucleus sampling) often struggle to balance quality and diversity, especially at higher temperatures which lead to incoherent or repetitive outputs. We propose min-p sampling, a dynamic truncation method that adjusts the sampling threshold based on the model's confidence by using the top token's probability as a scaling factor. Our experiments on benchmarks including GPQA, GSM8K, and AlpacaEval Creative Writing show that min-p sampling improves both the quality and diversity of generated text across different model families (Mistral and Llama 3) and model sizes (1B to 123B parameters), especially at higher temperatures. Human evaluations further show a clear preference for min-p sampling, in both text quality and creativity. Min-p sampling has been adopted by popular open-source LLM frameworks, including Hugging Face Transformers, VLLM, and many others, highlighting its considerable impact on improving text generation quality.

1 INTRODUCTION

Min-p sampling is introduced to address the creativity–coherence tradeoff in LLM generation, especially when higher temperatures increase diversity but reduce coherence. Experiments and human evaluations report improved quality and diversity, with preference for min-p outputs.

  • Contribution: Min-p sampling dynamically truncates tokens using model confidence to balance creativity and coherence, particularly at high temperatures.It keeps high-confidence tokens when the model is confident and allows more diverse options when uncertain.
  • Evaluation: Min-p sampling is evaluated on GPQA, GSM8K, and AlpacaEval Creative Writing against top-p, top-k, and η-sampling.The evaluation covers benchmark performance and human judgments of generated quality and diversity.
  • Validation: Human evaluators prefer min-p outputs over outputs from other sampling methods in both quality and diversity.The paper also reports practical guidelines and adoption in open-source LLM frameworks.
  • Implication: Min-p enables high-temperature creative generation without sacrificing coherence, according to the paper’s reported results.The authors characterize it as a viable alternative to existing sampling methods.

2 RELATED WORK

Prior decoding methods expose a quality–diversity tradeoff: conservative strategies can limit creativity, while higher-temperature or broad stochastic sampling can reduce coherence. Min-p is motivated by limitations in fixed-threshold and entropy-based approaches.

  • Existing Methods: Greedy decoding and beam search can produce repetitive, generic text, while beam search also incurs a runtime penalty.These deterministic methods lack the diversity of stochastic sampling.
  • Existing Methods: Temperature scaling increases diversity but often produces incoherent outputs at higher temperatures, and top-k does not adapt to varying confidence across contexts.Both methods use relatively fixed controls over the sampling pool.
  • Top-p Sampling: Top-p sampling can include low-probability tokens at high temperatures, creating incoherent outputs despite preserving a high-probability nucleus.This high-temperature limitation is the specific tradeoff addressed by min-p.
  • Entropy-Based Sampling: Entropy-based methods such as η-sampling and mirostat dynamically adjust the sampling pool but can require complex tuning and substantial computation.The paper reports additional experimental challenges with η-sampling.
  • Dynamic Threshold Methods: An adaptive plausibility constraint also uses a threshold proportional to the maximum token probability, making it similar to min-p in its confidence-relative design.The supplied passage begins describing a distinction between that prior method and min-p.

3 MIN-p SAMPLING

Min-p sampling makes token truncation context-sensitive by adjusting its threshold according to the model’s confidence at each decoding step. This design aims to improve the balance between creativity and coherence, particularly at high temperatures.

  • Overview: The method is designed to improve sensitivity to context and uncertainty while balancing creativity and coherence at high temperatures.Its core idea is to use confidence-dependent rather than absolute truncation.
  • Overview: Min-p sampling varies its truncation threshold according to model confidence during autoregressive next-token generation.The method operates on the probability distribution conditioned on the existing sequence.
  • Overview: When model confidence is high, min-p restricts sampling to high-probability candidates; when confidence is low, it relaxes the pool for more creative and diverse generation.This conditional behavior is the method’s central intuition.

top-p=0.9

Min-p computes a confidence-scaled threshold and samples from tokens above it, adapting the pool to certainty. The method is presented as balancing coherence and diversity, with simple implementation, practical parameter guidance, and broad framework integration.

  • Comparison: Figure 1 compares top-p, top-k, and min-p across high- and low-certainty token distributions.Min-p focuses on likely tokens when confident and permits more diverse options when uncertain.
  • Threshold Construction: At each decoding step, min-p identifies the maximum token probability and scales a base threshold by that confidence value.The resulting threshold is context-dependent rather than absolute.
  • Sampling Pool: The sampling pool contains tokens whose probabilities meet or exceed the scaled threshold, after which the next token is sampled from the normalized pool.This filters less probable tokens while retaining sufficiently probable alternatives.
  • Adaptive Behavior: Min-p dynamically adjusts within a sequence, restricting high-confidence cases and relaxing low-confidence cases to balance coherence and diversity.This contrasts with more static top-p and top-k thresholds.
  • High-Temperature Robustness: At higher temperatures, min-p scales truncation with model confidence to preserve coherence while retaining creative options.The paper identifies high-temperature robustness as a key advantage over existing methods.
  • Implementation: Min-p requires minimal calculations and integrates into existing inference pipelines with negligible overhead through vectorized operations.The method is described as simpler than auxiliary-model and entropy-based approaches.
  • Parameter Selection: A base threshold of 0.05 to 0.1 is reported as an effective balance across tasks and models, while higher values can favor coherence over diversity.The supplied guidance also describes higher temperatures as increasing diversity with lower coherence loss.
  • Adoption: Min-p is integrated into Hugging Face Transformers, vLLM, SGLang, and other open-source frameworks.The paper presents this community adoption as evidence of availability and potential practical impact.

4 CASE STUDIES: ILLUSTRATIVE EXAMPLES

The case studies show min-p adapting its truncation threshold to model confidence: it permits plausible alternatives when uncertain and prioritizes likely tokens when confident. This behavior is intended to preserve creativity and coherence across contrasting contexts.

  • Low-certainty context: In a high-temperature creative-writing prompt with multiple plausible continuations, min-p behaves similarly to top-p and preserves a range of diverse options.The low-certainty case uses a relatively flat next-token distribution.
  • High-certainty context: In a factual prompt where the model expects “light,” min-p prioritizes relevant tokens and maintains factual accuracy and coherence at τ = 3.The example represents a high-confidence next-token context.
  • Practical interpretation: The contrast between naming or setting generation and later factual recall illustrates how one sampler can support both exploratory and consistency-sensitive continuations.The paper connects these use cases to diverse options early and accurate narrative details later.
  • Confidence-adaptive truncation: Min-p adjusts its sampling threshold according to model confidence, allowing more options for uncertain prompts and filtering low-probability tokens for confident prompts.The paper illustrates this behavior with creative and factual continuation prompts.

5 EXPERIMENTS

The experiments compare min-p with established sampling methods across reasoning, mathematics, and creative writing benchmarks, model families, scales, temperatures, and hyperparameters. Across these evaluations, min-p generally performs best or maintains its advantage particularly as temperature increases.

  • Experimental design: Min-p is evaluated against top-p, temperature, ϵ, η, top-k, and mirostat sampling across GPQA, GSM8K CoT, and AlpacaEval Creative Writing.The study also varies temperatures and uses benchmark-specific metrics.
  • Graduate-level reasoning: Min-p consistently achieves higher GPQA accuracy than other methods across temperature settings, with the performance gap widening at higher temperatures.The result is reported for Mistral 7B and is described as maintaining correctness while increasing diversity.
  • Model scaling and generalization: Min-p’s advantages persist on Mistral Large with 123B parameters and generalize across Llama 3 models from 1B to 70B parameters.These results indicate that the reported benefits extend across model scales and families.
  • Grade-school mathematics: At lower temperatures τ < 1.5, min-p and top-p perform similarly on GSM8K CoT, while min-p’s advantage becomes more pronounced at higher temperatures.The paper attributes the low-temperature similarity to a smaller token candidate pool.
  • Grade-school mathematics: η and ϵ sampling show exponential runtime increases with temperature and fail completely at τ > 1.5, whereas min-p remains computationally efficient.This runtime comparison is reported in the GSM8K CoT evaluation.
  • Accuracy-diversity trade-off: Across GSM8K self-consistency configurations, min-p lies closer to the Pareto frontier and offers a better accuracy-creativity trade-off than top-p.Its broader configuration spread also provides finer control over diversity and coherence, while top-p configurations cluster more strongly.

6 HUMAN EVALUATION

The human evaluation compares perceived quality and diversity of sampled stories across methods, temperatures, and diversity settings. Min-p receives comparatively higher evaluations, especially at higher temperatures, while the study acknowledges constraints related to sample size and evaluation setup.

  • Evaluation boundary: The human-evaluation evidence is presented as directional because each diversity score reflects the three specific samples rated for a configuration.The authors limited sample size to manage attention, standardized scoring, and participant costs.
  • Response quality: The study filtered responses with attention checks and adversarial prompts, retaining 54 valid responses from 70 initial submissions.Small bonuses were also offered for detailed written feedback.
  • Experimental setup: The study evaluates Llama 3 70B with top-p and min-p across τ = 1.0, 2.0, and 3.0, using low- and high-diversity settings for each method.The design yields 12 total configurations.
  • Evaluation criteria: Participants rated sets of three outputs from 1 to 10 for quality and diversity, where diversity measured how creative or distinct the stories appeared.Quality covered prompt fulfillment, coherence, relevance, and overall quality.
  • Human-evaluation results: Min-p maintains comparatively higher quality and diversity scores than top-p at higher temperatures, where top-p’s scores often decline more sharply.Table 4 reports mean scores with standard errors and identifies significant differences where available.
  • Follow-up evaluation: An initial evaluation produced varied qualitative feedback, while a refined study with an improved inference engine and setup refinements reinforced the findings.Both sets of results are reported as available in supplementary materials or the project repository.

7 CONCLUSION

The paper introduces min-p as a confidence-adaptive truncation method for balancing creativity and coherence, especially at high temperatures. Experiments and human evaluations report improved quality and diversity, while the method’s simplicity, efficiency, and framework adoption support practical use.

  • Contribution: Min-p dynamically adjusts sampling thresholds based on model confidence to balance creativity and coherence, particularly at higher temperatures.This is the paper’s central methodological contribution.
  • Empirical conclusion: Across diverse benchmarks, min-p consistently outperforms existing methods in both generated-text quality and diversity, with human evaluations confirming preference for its outputs.The conclusion frames this result across quantitative experiments and human judgments.
  • Practical significance: Min-p’s simplicity, computational efficiency, and integration into existing pipelines are associated with rapid adoption in leading open-source frameworks.The paper presents this adoption as evidence of practical usefulness for applications requiring high-quality and diverse text.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility through public implementation resources, detailed experimental documentation, and publicly accessible datasets. Independent evaluations and replications broadly confirm the reported advantages of min-p.

  • Implementation code, experimental configurations, hyperparameters, evaluation protocols, and publicly accessible datasets support reproduction.The implementation is provided in Appendix A.1 and the project repository, with further setup details in Section 5 and Appendix A.2.
  • Independent replications reproduced reported improvements on medium-to-high-temperature GPQA and GSM8K evaluations.
  • Independent AlpacaEval Creative Writing and EQ-Bench evaluations broadly confirmed min-p’s advantages.

ETHICS STATEMENT

The paper identifies misuse and safety risks associated with more fluent or higher-temperature generation, while emphasizing responsible implementation, filtering, transparency, and safety research. It also presents uncertainty- and entropy-based methods as relevant to robustness and truthfulness research.

  • Min-p could enhance the fluency of misleading or harmful content, motivating responsible implementation and content filtering.
  • Higher-temperature generation may increase risks of circumventing safety finetuning, although the authors report no such instances in practice.
  • The paper supports transparency by open-sourcing implementation, detailing experiments, and removing human survey respondents’ identifying information.
  • The authors are exploring min-p for mechanistic interpretability, uncertainty-aware generation, neuron activation filtering, and structured latent selection.
  • The authors argue that entropy- and uncertainty-based methods may benefit robustness, truthfulness, and reduced hallucinations, while encouraging safety and alignment research.

CONTRIBUTION STATEMENT

The contribution combines the original min-p idea, extensive evaluation and supervision, implementation resources, methodological analysis, and adoption evidence. The section also documents computational challenges, parameter behavior, and a conservative approach to measuring community integration.

  • Contributors: Andrew Baker conceived min-p, tested it with open-source hobbyists, and helped drive its external adoption.
  • Contributors: The authors contributed research planning, drafting, benchmark evaluations, supervision, visualization, human-evaluation coordination, and methodological guidance.
  • Resources: Reference implementations and automated GPQA, GSM8K Chain-of-Thought, and AlpacaEval Creative Writing evaluations are available in the project repository.
  • Implementation: The implementation defines a min-p logits warper with configurable min_p, filtering behavior, and minimum tokens to keep.The code validates min_p and min_tokens_to_keep, converts logits to probabilities, scales the threshold using the top-token probability, masks low-probability tokens, and preserves a minimum token count.
  • Implementation: The implementation converts logits to probabilities, obtains the top-token probability, computes a scaled threshold, masks tokens below it, and returns processed scores.
  • Evaluation methodology: For Mixtral-7b at Risk Level 15, the reported table compares stability and diversity across truncation methods, risk levels, and models.
  • Evaluation methodology: Top-k settings varied substantially by temperature, complicating fair comparison, while η and ϵ sampling became computationally impractical at higher temperatures.Min-p, top-p, and top-k generally took 2–5 minutes per temperature on Mistral 7B, whereas η and ϵ runtime increased sharply and neither functioned at τ >= 2.
  • Method analysis: Min-p and top-p thresholds are not equivalent percentage thresholds; in high-certainty distributions, min-p can truncate disproportionately more probability mass.For example, top-p = 0.9 preserves the top three tokens in the cited distribution, whereas min-p = 0.1 preserves only the top token under its scaled threshold.

C.2 ADDITIONAL HUMAN EVALUATION WITH VLLM INFERENCE ENGINE

A refined human evaluation used VLLM and improved study procedures to assess sampling methods more accurately, especially at high temperatures. Min-p maintained substantially better quality and diversity than baseline methods under these conditions.

  • Inference engine: Switching to VLLM ensured temperature scaling occurred before truncation, properly evaluating min-p and top-p at high temperatures.The authors attribute weaker differences in the initial evaluation partly to Hugging Face applying temperature scaling after truncation.
  • Methodological improvements: The refined evaluation used vetted AI-experienced participants, full-length stories, standardized hyperparameters, longer reading time, and stronger attention checks.These changes were intended to improve comparability and detect incoherence more reliably.
  • Evaluation refinement: Initial participants sometimes rated clearly incoherent standard-sampling text highly, motivating clearer instructions to penalize narrative incoherence.A cited example received a 7–8/10 rating despite grammatical and narrative problems.
  • Human evaluation results: At temperature 3.0, min-p achieved quality and diversity scores of 5.80 and 5.90, compared with 1.23 and 1.27 for the comparison condition.The refined evaluation reported a pronounced high-temperature gap between min-p and baseline sampling.
  • Implications: The evaluations indicate that min-p supports creative writing and diverse reasoning paths while preserving coherence at higher temperatures.The paper presents these as practical domains where high-temperature incoherence had limited generation quality.

D.2 LIMITATIONS AND FUTURE RESEARCH DIRECTIONS

The paper identifies theoretical, generalization, hyperparameter, and evaluation boundaries for min-p, while outlining extensions to other models, domains, and decoding approaches. Future work also considers related dynamic truncation methods and combinations with other samplers.

  • Related methods: Min-z is presented as a related variation using median-centered Z-score normalization, intended to improve robustness for skewed or heavy-tailed logit distributions.The paper also proposes extending this dynamic selection idea beyond token probabilities to ranked or continuous selection processes.
  • Generalization: The experiments focus on Mistral and Llama 3, so larger models and different architectures require further evaluation.The paper notes promising recommendations for DeepSeek-R1 but does not present those models as part of its experiments.
  • Hyperparameter sensitivity: The base probability threshold p_base is critical, and dynamically adapting it across contexts or tasks remains unresolved.The authors note limited prior experimental guidance for selecting min-p hyperparameters with temperature scaling.
  • Combined sampling: Combining min-p with other truncation methods is theoretically plausible but difficult to test because relative contributions and hyperparameters must be isolated.The authors report that their own tests did not find benefits from such combinations.
  • Open questions: A deeper theoretical account of min-p’s high-temperature advantage remains an open research direction.Such analysis could clarify language-model behavior and guide development of other sampling strategies.
  • Generalization: Applying min-p to code generation and multimodal models could test whether its benefits extend beyond the evaluated text-generation settings.The paper explicitly identifies these domains as opportunities for future work.
  • Evaluation scope: The human evaluation used participant judgments of pre-generated outputs, leaving interactive evaluation as an important future direction.The paper specifically points to interactive platforms such as Chatbot Arena.
Loading 2407.01082v8…