Source-linked AI summary
The Effect of Sampling Temperature on Problem Solving in Large Language Models
Matthew Renze, Erhan Guven
TL;DR
The paper asks whether sampling temperature has a measurable effect on LLM problem-solving performance, addressing limited systematic evidence behind prevailing guidance. It evaluates benchmark-derived MCQA exams across models, prompts, and temperatures using accuracy and response-similarity measures. Performance from 0.0 to 1.0 did not differ significantly across the tested models, prompt techniques, and problem domains, although the study’s scope is limited to selected models and prompt settings.
Problem
Systematic empirical evidence is limited regarding the optimal sampling temperature for LLM problem-solving tasks.
Method
The study evaluates benchmark-derived MCQA exams across multiple LLMs and prompt-engineering techniques using accuracy and response-similarity measures.
Results
Changes in temperature from 0.0 to 1.0 did not produce statistically significant differences in MCQA problem-solving performance across multiple LLMs, prompt-engineering techniques, and problem domains.
Takeaways & Limitations
For the tested MCQA problem-solving settings, temperature changes within 0.0 to 1.0 appear not to significantly affect performance.
Takeaways & Limitations
The findings may not hold for excluded LLMs, more complex prompts, or agent architectures because the study used selected models and limited prompt settings.
Abstract
from arXiv · showhide
In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature
1 Introduction
The paper examines how sampling temperature affects LLM problem-solving and addresses the lack of systematic evidence behind conflicting recommendations. It frames temperature as a runtime control over output randomness, with lower values favoring exploitation and higher values favoring exploration.
- Motivation: Optimal sampling-temperature guidance for problem-solving remains uncertain because existing recommendations are often anecdotal or based on guesswork.The paper notes that systematic studies and empirical evidence supporting many community claims are limited.
- Motivation: The study aims to determine the optimal LLM sampling temperature for problem-solving tasks through systematic empirical analysis.Its stated goal is to contribute evidence for LLM and prompt-engineering best practices.
- Sampling Temperature: Sampling temperature controls the randomness of an LLM’s output during inference.It modifies the softmax probability distribution over possible next tokens.
- Sampling Temperature: Lower temperatures favor more deterministic, repetitive, and focused outputs, whereas higher temperatures favor more diverse and unconventional outputs.The paper describes temperature as a trade-off between exploiting probable solutions and exploring the solution space more broadly.
- Prior Guidance: Community guidance commonly recommends low temperatures for precision tasks and higher temperatures for creative tasks, while warning that higher values may increase hallucination probability.These recommendations are presented as prevailing opinions rather than systematically established rules.
2 Methods
The study evaluates temperature across multiple LLMs, prompts, exams, domains, and repeated attempts using standardized MCQA benchmarks. It combines correct-answer accuracy with response-similarity measures and extends an initial temperature sweep until outputs become incoherent.
- Models and Prompts: The experiment used nine popular LLMs and five prompt-engineering techniques, including baseline, domain expertise, self-recitation, CoT, and composite prompts.Each prompt used a single request-response cycle with one-shot in-context learning.
- Data: The dataset comprised standardized MCQA exams sampled from widely used benchmarks across multiple domains and difficulty levels.The researchers selected MCQA benchmarks so correct-answer accuracy could serve as the primary metric.
- Data: The large exam contained 1,000 questions, while the small exam contained 100 questions and supported analysis across all nine models, five prompts, and temperatures from 0.0 to 1.6.The smaller exam was used for broad analysis because of cost and runtime considerations.
- Experimental Procedure: For each model, prompt, exam, and temperature combination, the study collected ten answers per question and computed mean correct-answer accuracy across problems.Other sampling parameters were fixed during the initial temperature sweep to isolate temperature’s effect.
- Experimental Procedure: The initial sweep found that performance dropped rapidly after temperature 1.0 and generated text became incoherent at 1.6, motivating later sweeps from 0.0 to 1.0.GPT-3.5 first completed the small exam across temperatures extending to 2.0 in 0.1 increments.
- Metrics: The analysis used correct-answer accuracy as the primary metric and response text-similarity metrics as additional evidence about output variability.The similarity analysis included measures such as Jaccard, Bag-of-Words, TF-IDF, Levenshtein, BLEU, and SBERT similarity.
3 Results
Across models, prompts, and problem domains, accuracy remained stable from temperatures 0.0 to 1.0, while performance deteriorated above 1.0. Text variability increased with temperature, paralleling the accuracy decline at higher temperatures.
- Accuracy remained stable from temperatures 0.0 to 1.0 across LLMs, prompt-engineering techniques, and problem domains.
- For GPT-3.5 with CoT, accuracy stayed stable through 1.0 but dropped rapidly afterward, reaching zero around 1.6.
- Most LLMs showed a gradual, non-significant performance decrease with temperature; Llama 2 7B was the exception.
- Llama 2 7B failed to outperform statistically random guesses, partly because 39% of answers were incorrectly formatted and 36% were correctly formatted but incorrect.
- Accuracy remained stable across all five prompts, while CoT outperformed the other four prompts.
- Across ten exams, visual analysis found stable accuracy across problem domains, although LSAT-AR and SAT-Math showed statistically significant Kruskal-Wallis differences.
- Individual-problem accuracy varied significantly pairwise across temperatures in LSAT-AR and SAT-Math, but average accuracy remained similar.
- Text similarity decreased as temperature increased, especially after 1.0, matching the rapid accuracy decline above 1.0.
4 Discussion
Across the tested settings, temperatures from 0.0 to 1.0 generally did not significantly affect problem-solving performance, while performance and text similarity declined rapidly above 1.0. The authors recommend 0.0 for reproducibility, but identify several scope and experimental limitations.
- Text similarity decreased as temperature increased, with a rapid decline after 1.0 that corresponded to the observed accuracy drop.
- Temperatures from 0.0 to 1.0 did not significantly affect problem-solving performance across tested LLMs, prompting techniques, and domains.
- The authors recommend temperature 0.0 for problem-solving because it maximizes reproducibility without compromising accuracy.
- Limitations: The study was limited to subsets of LLMs, prompting techniques, problems, domains, tasks, and model-prompt-exam-temperature combinations.
- Limitations: The experiments used two randomly sampled MCQA test sets containing 1,000 and 100 questions, potentially limiting statistical reliability.
- Limitations: All other sampling parameters were fixed, so different parameter combinations may produce different outcomes.
- Future work: The authors propose testing additional models, open-ended tasks, larger and more creative problem sets, wider temperature ranges, and problem-level error patterns.
5 Conclusion
The study empirically examined sampling temperature across multiple LLM problem-solving settings. It found no statistically significant performance differences from 0.0 to 1.0 on MCQA problems across models, prompting techniques, and domains, with practical and theoretical implications.
- The study empirically investigated sampling temperature effects on LLM problem-solving performance across multiple problem domains.
- No statistically significant performance differences occurred from 0.0 to 1.0 across multiple LLMs, prompting techniques, and problem domains on MCQA problems.
- The findings have practical implications for AI engineers and theoretical implications for researchers studying hallucination and solution-space search.
A Appendix
The appendix provides examples of the composite system prompt and the MCQA problem format used in the study.
- Figure 10 shows a composite system prompt containing a one-shot problem-and-solution example.
- Figure 11 shows an MCQA problem represented in JSON-L format with added whitespace for readability.