Source-linked AI summary
Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
Lujun Li, Lama Sleem, Niccolo' Gentile, Geoffrey Nichil, Radu State
TL;DR
This study examines how temperature affects six LLM capabilities across model sizes, addressing limited systematic evidence and the lack of a universally optimal setting. It evaluates temperature-selection methods and precision settings, finding skill- and size-specific effects, improved SuperGLUE performance with prompt-specific selection, and higher mutation temperatures in larger models.
Problem
Prior studies provide limited systematic evidence on temperature effects across capabilities and models, making it unclear whether effects are uniform or whether each prompt has an optimal temperature.
Method
The study analyzes temperature effects across six capabilities and model sizes, compares FP16 with 4-bit inference, and uses BERT- or GPT-based selectors to predict prompt abilities and choose temperatures.
Results
Temperature effects vary substantially by capability and model size; prompt-specific selection improves three SuperGLUE tasks for small and medium models, while larger models generally have higher mutation temperatures and FP16 preserves similar optimal temperatures.
Takeaways & Limitations
No single temperature is optimal across tasks, so selecting temperature according to the prompt's required ability can improve performance for small and medium models on SuperGLUE.
Takeaways & Limitations
The study covers only six text-based abilities, and the accuracy of GPT-based evaluations is difficult to verify without manually checking each case.
Abstract
from arXiv · showhide
The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output tokens. Recent studies have challenged the Stochastic Parrots analogy by demonstrating that LLMs are capable of understanding semantics rather than merely memorizing data and that randomness, modulated by sampling temperature, plays a crucial role in model inference. In this study, we systematically evaluated the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities, conducting statistical analyses on open source models of three different sizes: small (1B--4B), medium (6B--13B), and large (40B--80B). Our findings reveal distinct skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection in practical applications. To address this challenge, we propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt. We demonstrate that this approach can significantly improve the performance of small and medium models in the SuperGLUE datasets. Furthermore, our study extends to FP16 precision inference, revealing that temperature effects are consistent with those observed in 4-bit quantized models. By evaluating temperature effects up to 4.0 in three quantized models, we find that the Mutation Temperature -- the point at which significant performance changes occur -- increases with model size.
1. Introduction
The paper studies how sampling temperature reshapes LLM output probabilities and affects performance across capabilities. It asks whether effects differ by ability and model, and whether prompt-specific optimal temperatures can be identified.
- Temperature and LLM inference: Temperature scales output logits before sampling, controlling the randomness of model predictions.As temperature increases, the token probability distribution becomes more uniform; as it approaches zero, generation becomes greedy.
- Temperature and LLM inference: At T approaching zero, the probability distribution collapses toward the most likely token, whereas higher T produces more varied token choices.Each generation step samples from the updated probability distribution.
- Research questions: The study evaluates whether temperature affects different LLM abilities and model sizes uniformly.The research questions cover the magnitude of effects, cross-ability and cross-model differences, and optimal temperature selection.
- Research questions: The paper investigates whether an optimal temperature exists for each capability and can be selected for a specific prompt.This question motivates prompt-level temperature selection based on the required ability.
2. Related Work
Prior research provides limited and inconsistent evidence about temperature effects across tasks. The paper frames this gap as the Temperature Paradox, where opposing ability-specific effects may obscure overall performance changes.
- Limits of prior work: Temperature investigations have generally used narrow settings, limiting systematic evidence about its effects across language-model tasks.Most studies report only one temperature value, with limited exceptions testing multiple settings.
- Ability-specific effects: Higher temperatures are associated with greater creativity, whereas lower temperatures are associated with improved logical reasoning.These task-specific tendencies motivate examining abilities separately rather than relying on broad multitask evaluations.
- Ability-specific effects: The Temperature Paradox describes cases where opposing temperature effects on logic and creativity may cancel in tasks requiring both.This makes the overall impact of temperature difficult to observe in general-purpose capability datasets.
- Prior findings: Some prior studies found weak or nonsignificant temperature effects, while adaptive strategies assigned temperatures dynamically during generation.Reported examples include multitask prompting, creative-writing evaluation, and token-level code-generation strategies.
3. Approaches
The study evaluates six intrinsic abilities with capability-specific datasets and metrics, using different evaluation procedures for complex and reference-answer tasks. It also addresses the challenges of judging stochastic, semantically equivalent outputs.
- Evaluated abilities: Six abilities are evaluated: causal reasoning, creativity, in-context learning, instruction following, machine translation, and summarization.The selected tasks use capability-focused datasets intended to reduce bias from mixed-ability evaluation.
- Evaluated abilities: Causal reasoning uses CRASS with Top-1 Accuracy, while in-context learning uses LongBench-TREC with classification accuracy.CRASS requires selecting the correct answer, and LongBench-TREC requires classifying a final question using prior context.
- Evaluated abilities: Creativity is assessed across fluency, flexibility, originality, and elaboration using customized Torrance Test of Creative Writing questions.The framework uses expert judgments to determine true-or-false outcomes for evaluation questions.
- Evaluated abilities: Instruction following uses InfoBench’s Decomposed Requirements Following Ratio, while translation uses normalized spBLEU on FLORES-101.The DRFR metric decomposes complex instructions, and spBLEU is divided by 100 to place scores in [0, 1].
- Evaluated abilities: Summarization uses expert-written references and Rouge-L F1 to measure longest-common-subsequence overlap.Rouge-L F1 balances precision and recall and is reported as correlating with human judgments.
- Evaluation procedures: LLM-as-a-Judge evaluates causal reasoning, creativity, and instruction following, while standard reference-based methods evaluate the other abilities.The distinction reflects the difficulty of reliably matching stochastic outputs with flexible wording.
4. Experiments
Experiments test quantized open-source models across temperature settings and use ability prediction to select temperatures for SuperGLUE prompts. Additional experiments compare extended temperatures and FP16 inference.
- General settings: Each question is tested three times across 12 models quantized to 4 bits with AWQ, using vLLM for inference acceleration.The experiments use question-answer or matching-based evaluation with dataset-specific metrics.
- General settings: Temperatures range from 0.1 to 1.9 in seven configurations, while values above 2.0 are excluded from the main experiments.Nucleus sampling is used, and each model receives only one question per test.
- Best temperature selection: The BERT-based Selector predicts a prompt’s required ability and uses prior performance distributions to choose its optimal temperature.A GPT-based Selector provides an alternative ability-prediction route.
- Best temperature selection: SuperGLUE evaluation covers three model sizes: Llama-3.2-1B-Instruct, Llama-3-8B-Instruct, and Mixtral-8x7B-Instruct-v0.1.Each benchmark question is generated three times with successive random seeds.
5. Results and Analysis
Temperature affects LLM abilities differently: correlations are strong for some tasks, while model size changes robustness and performance patterns. Capability-specific temperature settings and dynamic selection are evaluated across models, precisions, and extended temperature ranges.
- Statistical analysis: Table 3 summarizes temperature-performance correlations, relative performance ranges, coefficient variation, and average accuracy across six abilities and three model categories.“P. Coef.” and “S. Coef.” denote Pearson and Spearman correlations; the reported accuracy values are accompanied by standard deviations.
- Statistical analysis: Larger models are more robust to temperature variation and achieve higher average accuracy across all six abilities.Range Max and CV Max decrease with model size, while size differences are more pronounced for CR, ICL, and MT than for CT, IF, and SUMM.
- Ability-specific effects: Creativity generally rises and then falls with temperature, while In-Context Learning generally declines as temperature increases.Temperature 1.3 is recommended for creativity in medium and large models; large ICL models remain stable from 0 to 2.
- Ability-specific effects: Instruction Following remains stable through T = 1 but undergoes abrupt negative changes above 1, with later mutation temperatures in larger models.Mutation intervals are 1.0–1.3 for small, 1.3–1.6 for medium, and 1.6–1.9 for large models.
- Ability-specific effects: Machine Translation shows the greatest temperature sensitivity, with the optimal temperature near zero and comparable declines across model sizes.Slight low-temperature increases marginally help only small and medium models; MT has the highest performance range and CV.
- Temperature selection: Dynamically selecting temperature improves SuperGLUE performance for small and medium models, while large models show no significant performance difference.The selector avoids drops from fixed, suboptimal settings but does not inherently boost performance beyond supervised fine-tuning.
- Extended settings: Extending temperature to 4.0 identifies model-specific mutation temperatures, which generally increase with model size, while FP16 preserves optimal-temperature patterns seen under 4-bit quantization.FP16 and 4-bit inference have no substantial optimal-temperature difference despite a 10%–20% performance difference.
6. Conclusion
The study finds that temperature affects LLM capabilities differently across model sizes, with no universally optimal setting. Temperature selectors can improve SuperGLUE performance for smaller models, while extended-range and precision analyses show size-dependent mutation thresholds and similar temperature effects across precision levels.
- 6. Conclusion: Comparisons extended to temperatures up to 4.0 show that larger models generally have higher mutation temperatures, the thresholds where performance drops sharply.The study also reports that large-model mutation temperatures can exceed 2.0.
- 6. Conclusion: Temperature effects vary substantially by capability and model size, with generally negative performance correlations but stronger impacts on machine translation and creativity in small models.The reported maximum relative effects reach 192.32% for machine translation and 186.81% for creativity in small models.
- 6. Conclusion: Large models are more resilient to temperature changes, while higher temperatures can initially improve causal reasoning, in-context learning, and instruction following before performance declines.Summarization and machine translation generally worsen as temperature increases, especially in smaller models.
- 6. Conclusion: No single temperature is optimal across tasks; a BERT-based selector can identify prompt-specific settings and improves performance for small and medium models on three SuperGLUE tasks.The approach selects temperatures by identifying the required ability for each prompt and consulting the experimental results.
- 6. Conclusion: The study is limited to six text-based abilities, relies partly on GPT evaluations that are difficult to verify manually, and needs testing across additional models.A mathematical explanation of temperature effects also remains unresolved.