Source-linked AI summary
Is Temperature the Creativity Parameter of Large Language Models?
Max Peeperkorn, Tom Kouwenhoven, Dan Brown, Anna Jordanous
TL;DR
The paper asks whether temperature is the creativity parameter of large language models, a question complicated by creativity’s multidimensional nature. It combines computational and human evaluation of LLAMA 2-CHAT stories generated under a fixed model and prompt, finding weak novelty gains but no compelling overall support for the claim. The authors therefore argue for more controlled approaches to LLM creativity rather than reliance on temperature-driven chance.
Problem
The paper examines whether temperature, which increases generation randomness and diversity, can serve as a general creativity parameter for LLMs.
Method
The study combines computational analysis and human evaluation of LLAMA 2-CHAT stories across temperature values, using fixed context and four creativity conditions.
Results
Temperature is weakly positively correlated with novelty and moderately negatively correlated with coherence, while the overall evidence does not support it as a creativity parameter.
Takeaways & Limitations
LLM creativity likely requires evaluation and generation methods beyond relying on temperature to introduce randomness.
Takeaways & Limitations
Computational analysis cannot determine whether higher-temperature stories are creative in ways not captured by its diversity and overlap measures.
Abstract
from arXiv · showhide
Large language models (LLMs) are applied to all sorts of creative tasks, and their outputs vary from beautiful, to peculiar, to pastiche, into plain plagiarism. The temperature parameter of an LLM regulates the amount of randomness, leading to more diverse outputs; therefore, it is often claimed to be the creativity parameter. Here, we investigate this claim using a narrative generation task with a predetermined fixed context, model and prompt. Specifically, we present an empirical analysis of the LLM output for different temperature values using four necessary conditions for creativity in narrative generation: novelty, typicality, cohesion, and coherence. We find that temperature is weakly correlated with novelty, and unsurprisingly, moderately correlated with incoherence, but there is no relationship with either cohesion or typicality. However, the influence of temperature on creativity is far more nuanced and weak than suggested by the "creativity parameter" claim; overall results suggest that the LLM generates slightly more novel outputs as temperatures get higher. Finally, we discuss ideas to allow more controlled LLM creativity, rather than relying on chance via changing the temperature parameter.
Introduction
The paper tests whether temperature functions as an LLM creativity parameter by fixing the model and prompt and evaluating generated stories across four creativity conditions. It finds only weak, nuanced effects: higher temperature slightly increases novelty while reducing coherence.
- Research aim: The study investigates temperature’s effect on story creativity while controlling the model, prompt, and other parameters.This fixed-context design addresses reproducibility and comparison problems caused by prompt and model variation.
- Evaluation framework: Stories generated by LLAMA 2-CHAT are evaluated using novelty, typicality, cohesion, and coherence as necessary creativity conditions.The evaluation combines empirical analysis with human-participant creativity judgments.
- Primary findings: Temperature generally does not let the LLM access different embedding-space regions, although limited sampling can produce some novelty.The novelty effect is therefore constrained rather than evidence of broad exploration of the model’s possibilities.
- Primary findings: Temperature shows a weak positive correlation with novelty and a negative correlation with coherence, suggesting a novelty–coherence trade-off.The reported relationship is correlational rather than causal.
- Conclusion: Overall, temperature’s influence on creativity is weaker and more nuanced than the “creativity parameter” claim suggests.The paper motivates more controlled approaches than relying on temperature-driven randomness.
Background
The background frames creativity as multidimensional and narrative quality as requiring both distinctiveness and recognizable story form. It motivates fixed-context temperature experiments because randomness changes output distributions while complicating interpretation.
- Creativity conditions: Creative artifacts are characterized by novelty or originality together with usefulness or effectiveness.Typicality and surprise are discussed as additional properties related to novelty.
- Creativity conditions: Narrative creativity is evaluated through novelty, typicality, cohesion, and coherence because stories require both distinction and quality of narrative organization.The paper derives these four conditions from broader creative-writing criteria.
- Creativity conditions: A novel story differs from other stories, whereas a typical story conforms to patterns associated with its category.The two properties can conflict because high typicality often implies lower novelty.
- Probing large language models: Because prompt changes can cascade into different outputs, the study fixes the context when testing whether temperature reaches different regions of model knowledge.The expected comparison is between temperature variation and prompt-induced variation.
- The temperature parameter: Temperature regulates randomness by reshaping the softmax probability distribution over output tokens.Higher temperature flattens the distribution and redistributes probability mass; t = 0 practically corresponds to greedy sampling.
- The temperature parameter: Increasing randomness makes LLM outputs harder to probe and compare, motivating a consistent greedy-sample baseline.The baseline provides a point of departure for assessing temperature effects.
- Evaluation framework: Prototype and exemplar theories provide a comparative frame in which outputs can be related to representative category members.Exemplar theory compares new stimuli with multiple prior examples rather than one prototype.
The Exemplar Story
The exemplar-story setup establishes a baseline for comparing temperature-controlled generations. It uses a fixed prompt and transparent instruction-tuned model, while noting implementation choices and licensing constraints.
- Identifying the exemplar: The study uses a fixed model and prompt to establish a baseline for comparing generated stories.This controls lexical prompt changes that could otherwise cascade through the output.
- Identifying the exemplar: The greedy sample is considered as a possible prototype or exemplar for outputs generated from the same model and prompt.The paper presents prototype and exemplar perspectives for defining the comparison object.
- Generating the exemplar story: The generation uses the minimal prompt “[INST]Write a story.[/INST]” followed by a suffix intended to start the story immediately.The prompt is designed to introduce as little additional information as possible.
- Generating the exemplar story: Stories are generated with the instruction-tuned LLAMA 2-CHAT 70B model because its open weights provide access to architecture and parameters.That transparency is used to minimize influences other than temperature.
- Generating the exemplar story: The implementation applies 6-bit quantisation to fit the model on available hardware, with the authors describing minimal quality loss.The model also tends to add conversational material despite instructions to avoid it.
- Generating the exemplar story: The LLAMA 2 licence restricts commercial use, so the model is not strictly open source under the Open Source Initiative’s definition.This is a licensing qualification rather than a description of the model’s accessibility for the study.
Case Study – “Write a story.”
The case study combines computational and human evaluation to examine how temperature affects creativity in stories generated from a fixed narrative task. It compares output distributions with human judgments across four creativity conditions.
- Computational analysis: The study performs computational analysis across temperature values, examining story distributions in embedding space and distances from an exemplar.This measures output variation relative to the selected reference object.
- Human evaluation: A human evaluation study examines how temperature interacts with novelty, typicality, cohesion, and coherence in narrative generation.The two-fold design complements computational measurements with judgments about creativity.
Computational Analysis
Using a fixed prompt and model, the analysis tests whether temperature expands the diversity and exploratory reach of generated stories. Higher temperatures increase the chance of finding novelty in limited samples, but outputs still substantially overlap across temperatures and may move in a consistent direction relative to the exemplar.
- The analysis evaluates exploratory creative behaviour through diversity and similarity measures rather than directly judging story creativity.It compares lexical differences using normalised edit distance and semantic differences using cosine similarity between story embeddings and the exemplar.
- Higher temperatures do not necessarily produce more lexical or semantic diversity for this prompt and model.The distribution plots show overlap across temperature values, while higher temperatures do not extend semantic outputs much farther from the exemplar.
- Even lower temperatures immediately produce lexical diversity relative to the exemplar, while higher temperatures do not consistently extend that diversity.Normalised edit distance indicates an immediate effect at .334 < t < 1.0, whereas t > 1.0 does not necessarily lead to more diversity.
- With limited samples, higher temperatures increase the chance of finding novelty more quickly, although lower temperatures could eventually access the same generable embedding regions.The paper distinguishes finite real-world sampling from the theoretical case of unlimited time and space.
- Relative to the exemplar story, the generation process appears to move in a particular direction across outputs.The authors report similar behaviour for seemingly synonymous prompts as an observation warranting further investigation.
Experimental Analysis with Human Evaluation
The human evaluation used participant ratings of stories generated under a controlled setup, alongside mixed-effects analysis and reliability testing. Temperature showed weak positive associations with novelty and moderate negative effects on coherence, while computational metrics linked exemplar similarity to cohesion.
- Human Evaluation: 36 participants evaluated five stories each using four five-point ratings for novelty, typicality, cohesion, and coherence.Novelty and typicality were judged against an exemplar; cohesion and coherence were judged for each story.
- Analysis: Linear mixed-effects models controlled for participant-assignment effects, while Cronbach’s α assessed inter-rater reliability.The analysis tested significance, slope direction and magnitude using conditional R2.
- Human Evaluation: α = .639 indicated some inter-rater agreement, although it fell just below the acceptable .7 level.The authors attribute the lower agreement partly to the use of non-expert judges.
- Quantitative Results: β̂ = .240, SE = .122, R2c = .646 indicated a moderate temperature effect on coherence, with higher temperatures producing less coherent stories.Coherence was nevertheless rated relatively highly overall, with M = 2.13 on the study’s scale, and the shallow slope indicated a low rate of change.
- Computational Metrics: Cosine similarity showed a weak to moderate positive effect on cohesion, β̂ = 2.09, SE = .716, R2c = .431, suggesting exemplar proximity was associated with greater cohesion.The authors interpret larger semantic distance from the exemplar as accompanying loss of cohesion-related features.
Discussion and Further Work
The paper finds little support for temperature as a general creativity control and identifies evaluation, decoding, prompting, and benchmarking directions for more controlled LLM creativity.
- Discussion and Further Work: The study finds no compelling support for temperature as a general creativity parameter across the investigated creativity conditions.The authors characterize temperature’s influence as nuanced and weak.
- Discussion and Further Work: The exemplar-based evaluation methodology requires further testing across different domains, prompts, and models.The paper treats this as necessary to examine whether the observed exemplar phenomenon generalizes.
- Discussion and Further Work: Participants used different criteria for novelty and typicality and struggled to distinguish story details, complicating the robustness of the qualitative findings.The authors recommend repeating the experiment with experts, better stories, and creative-writing rubrics.
- Discussion and Further Work: The paper notes that strong LLM creativity benchmarks remain unavailable and that scaling reliable evaluation is challenging.Existing assessments often rely on psychology tests or creativity inferred from other-task benchmarks.
- Discussion and Further Work: Advanced decoding strategies, including top-k, nucleus sampling, mirostat, and locally typical sampling, may support creative writing when designed for specific purposes.The paper presents decoding strategy design as a direction beyond relying solely on temperature.
- Discussion and Further Work: Probing implicit information in LLMs could inform prompt design and help condition models toward higher-quality desired outputs.The paper also calls for methods that assess how information is preserved as prompt complexity increases.
- Discussion and Further Work: The authors argue that current creative output generation relies too heavily on chance and call for more informed, useful creative behavior.This motivates the paper’s recommendations for future research.
Related Work
Earlier narrative-generation systems used templates, rules, knowledge bases, or probabilistic transitions, while LLMs have largely displaced these approaches and appear to outperform them across many dimensions.
- Related Work: Early narrative-generation programs such as TALE-SPIN, TAILOR, and MINSTREL filled predetermined templates using rules and knowledge-base associations.These systems represent an earlier approach to automated story generation.
- Related Work: Some TALE-SPIN stories are described as surprisingly similar to outputs that LLMs might generate for simple prompts.The paper uses this similarity as a parallel between earlier narrative programs and LLM generation.
- Related Work: Markov chains provide a narrative-generation approach more akin to LLMs than template-based systems.Although simple low-order models generally perform poorly, variants have been applied successfully.
- Related Work: LLMs have largely displaced Markov approaches and appear to outperform them on most narrative-generation dimensions.The paper cites multiple examples of LLM-based narrative generation supporting this comparison.
Conclusion
The paper’s two-fold study finds that temperature increases variety among limited samples but does not broadly expand explored output space, and its creativity effects remain weak and nuanced. It recommends stronger benchmarks, purpose-built decoding, and methods for probing implicit model information and prompt effects.
- Conclusion: The study evaluates how temperature affects creativity in LLM-generated stories using an exemplar-based methodology.The investigation is explicitly two-fold and focuses on story generation.
- Conclusion: Limited sampling with higher temperature increases the chance of greater variety but does not generally access a larger slice of the probability distribution.This distinguishes sample-level variety from broader coverage of the model’s output space.
- Conclusion: A weak positive correlation links novelty with temperature, while a moderately negative correlation links coherence with temperature, implying a novelty–coherence trade-off.The paper concludes that temperature’s influence is weaker and more nuanced than the creativity-parameter claim suggests.
- Conclusion: The paper recommends developing scalable LLM-creativity benchmarks, advanced decoding strategies for creative purposes, and methods to probe implicit information and guide prompt design.These directions aim to support more informed evaluation and control of creative behavior.