Source-linked AI summary
Non-Determinism of "Deterministic" LLM Settings
Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, Breck Baldwin
TL;DR
This paper investigates how often supposedly deterministic LLMs vary across repeated runs and how that variation affects evaluation and downstream use. Across tasks, models rarely reproduce identical responses, while parsed answers are more stable but still show substantial instability.
Problem
The paper addresses the lack of systematic evidence about the prevalence and impact of non-determinism in LLM outputs under settings intended to be deterministic.
Source: §1 · p. 1Method
The authors analyze five LLMs across eight BBH and MMLU tasks in zero-shot and few-shot settings, using TARr@N and TARa@N to measure raw-output and parsed-answer agreement across runs.
Source: §1 · p. 2, §5.2 · p. 5Results
Takeaways & Limitations
Non-determinism can undermine single-number benchmark comparisons and cause downstream parser failures even when the underlying parsed answer remains stable.
Source: §8 · p. 8, §8.1 · p. 8
Abstract
from arXiv · showhide
LLM (large language model) practitioners commonly notice that outputs can vary for the same inputs under settings expected to be deterministic. Yet the questions of how pervasive this is, and with what impact on results, have not to our knowledge been systematically investigated. We investigate non-determinism in five LLMs configured to be deterministic when applied to eight common tasks in across 10 runs, in both zero-shot and few-shot settings. We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%. In fact, none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings. Sharing preliminary results with insiders has revealed that non-determinism perhaps essential to the efficient use of compute resources via co-mingled data in input buffers so this issue is not going away anytime soon. To better quantify our observations, we introduce metrics focused on quantifying determinism, TARr@N for the total agreement rate at N runs over raw output, and TARa@N for total agreement rate of parsed-out answers. Our code and data are publicly available at https://github.com/breckbaldwin/llm-stability.
1 Introduction
The paper investigates output instability in LLMs under settings expected to be deterministic, motivated by the risk that benchmark scores may not be reproducible. Across models, tasks, and prompting settings, it finds substantial variation and proposes metrics to quantify it.
1 Introduction
- The study finds that model performance is unstable across equivalent runs under presumed deterministic settings, varying by model, setting, and task.It quantifies instability across eight tasks from the BBH and MMLU benchmarks, comparing zero-shot and few-shot settings.Source: §1 · pp. 1–2
- Benchmark results may be unreliable when identical inputs to the same trained model produce different outputs at temperature=0.The paper frames this as an additional source of uncertainty beyond variation across training runs.Source: §1 · p. 2
- The analysis covers five models, eight tasks, and both zero-shot and few-shot settings across MMLU and BBH benchmarks.The paper compares settings including three-shot BBH and five-shot MMLU configurations.Source: §1 · p. 2
- TARr@N measures total agreement over raw outputs, while TARa@N measures agreement over parsed answers across N runs.These metrics distinguish variation in output strings from variation in extracted answers.Source: §1 · p. 2
2 Related Work
Prior work studies robustness to changed inputs or instability under variable sampling, but does not systematically examine identical inputs and maximally deterministic configurations across tasks and settings.
2 Related Work
- Existing studies examine input perturbations, temperature effects, or single-model code generation, leaving same-input instability under deterministic settings insufficiently investigated.The paper distinguishes its focus from work that introduces variability through temperature or studies only one model and task type.Source: §2 · p. 3
3 Controlling LLM Determinism
The paper treats temperature as the main control for determinism and evaluates the theoretical expectation that temperature=0 should reproduce the same output. It also checks whether top-p materially changes determinism.
3 Controlling LLM Determinism
- Temperature controls the probability distribution, with 0 intended as the most deterministic setting and 1 the least.Equation 1 describes word probabilities using temperature T and the model logit yi.Source: §3 · p. 3
- At T=0, the same prompt should theoretically produce the same output, while raising T diversifies outputs.The authors report that top-p had little effect on determinism in their checks.Source: §3 · p. 3
4 Datasets
The study uses two broad multiple-choice benchmarks and randomly selects four subtasks from each to balance task diversity against computational cost.
4 Datasets
- BBH contains 27 diverse tasks, while MMLU contains 57 tasks spanning humanities, social sciences, and STEM disciplines.The benchmarks include mathematics, commonsense reasoning, and other domains.Source: §4 · p. 3
- The experiments randomly select four subtasks from each benchmark, yielding eight tasks with listed example counts and answer-option counts.This sampling balances benchmark diversity against available computational resources.Source: §4 · p. 3
5 Methods
The study evaluates five LLMs under controlled zero-temperature prompting and introduces metrics for measuring instability in accuracy and generated responses.
5 Methods
- The experiments use fixed temperature, top-p, seed, infrastructure, inputs, and configurations in zero-shot and standard few-shot settings.Few-shot prompting uses 3 examples for BBH and 5 for MMLU, without fine-tuning or Chain-of-Thought.Source: §5 · p. 3
5.1 Models
- Five models from different families and sizes are evaluated: GPT-3.5 Turbo, GPT-4o, Llama-3-70B-Instruct, Llama-3-8B-Instruct, and Mixtral-8x7B-Instruct.Source: §5.1 · p. 4
5.2 Metrics
- TAR@N measures the percentage of questions with identical generated answers across N runs, using raw responses (TARr@N) or parsed answers (TARa@N).Raw-response agreement is stricter because any character variation counts as disagreement.Source: §5.2 · pp. 4–5
- BestAcc, WorstAcc, median accuracy, and maximum-minimum accuracy difference quantify complementary extremes and variation across runs.BestAcc counts a question as correct if any run answers it correctly, whereas WorstAcc counts it as incorrect if any run answers it incorrectly.Source: §5.2 · p. 5
- The study checks distributional assumptions with 20 GPT-4o and Llama-3-70B college-math runs, finding non-normal accuracy distributions and rejecting normality at p < 10^-9.Figure 2 reports median, mean, and 5% and 95% quantiles for identical runs.Source: Figure 2 · p. 4, §5.2 · p. 5
6 Experiments
The experiments cover 80 model–dataset–prompting conditions, with ten temperature-zero runs per condition to quantify within-condition instability.
6 Experiments
- Eighty conditions combine eight datasets, five base models, and zero-shot or few-shot settings, with ten temperature-zero runs per condition.Source: §6 · p. 5
7 Results
Across few-shot and zero-shot evaluations, LLM instability appears in both accuracy and exact outputs, varies by model and task, and is related to response length and accuracy.
7.1 Instability Results
- 5–15% accuracy differences occur across ten runs for some tasks, showing that identical conditions can yield materially different benchmark results.The reported difference is between minimum and maximum accuracies in ten runs for each condition.Source: §7.1 · pp. 5–6
- BestAcc, median accuracy, WorstAcc, TARa@10, and TARr@10 reveal substantial instability, with raw-output agreement typically lower than parsed-answer agreement.Zero-shot results show a similar degree of non-determinism, while TARr@10 varies widely across models and datasets.Source: §7.1 · p. 6
- GPT-3.5 Turbo has lower TARr and therefore less raw-output instability than other models, whereas Llama-3-70B often has very low TARr.This comparison is reported for the few-shot setting.Source: §7.1 · p. 6, Figure 3 · p. 6
- Parsed-answer agreement remains task-specific: navigation appears more reliable, while college math and professional accounting appear less robust.The comparison concerns few-shot TARa@10 results, which are less unstable than TARr@10 but still far from perfect.Source: Figure 4 · p. 6, §7.1 · p. 7
7.2 Correlation Analyses
- Longer outputs correlate with lower TARa@10 and TARr@10, while higher few-shot median accuracy correlates with higher TARa@10.Per-model analyses also find greater stability for shorter responses, with effects varying by model and prompting setting.Source: §7.2 · p. 7
8 Discussion
The discussion finds substantial non-determinism despite zero-temperature settings, with parsed answers more stable than raw strings but accuracy still varying enough to weaken single-score benchmarks. It also highlights engineering consequences, including parser failures, testing challenges, and compounding system-level degradation.
8 Discussion
- Parsed answers are more stable than raw output strings, but TARa@10 still shows instability of up to 15%.Raw format changes may matter little to human readers but can significantly affect downstream parsers.Source: §8 · p. 8
- 72% accuracy variation occurred for Mixtral-8x7b on college math, illustrating why reporting only one benchmark score can be unreliable.The difference was 75% maximum accuracy minus 3% minimum accuracy.Source: §8 · p. 8
8.1 Implications for Practical Engineering
- Non-determinism may arise from inference optimizations such as continuous batching, chunk prefilling, or prefix caching rather than models or GPUs alone.Llama3-8b was deterministic on local GPUs without such optimizations, while hosted API behavior could only be investigated speculatively.Source: §8.1 · p. 8
- Fine-tuning reduced instability in an experiment, but the authors hypothesize this may reflect the absence of co-mingled user tasks.The fine-tuned GPT-3.5 model was evaluated with two-fold cross validation.Source: §8.1 · p. 8
- Non-determinism limits unit testing, can cause parser failures, and may multiply performance losses across sequential classifiers.Four classifiers that are each 95% stable yield an expected stability of .95^4 = .814 before classifier accuracy is considered.Source: §8.1 · p. 8
- Zipfian input frequencies can make common-case testing appear reassuring while instability undermines confidence when rare mistakes are costly.The discussion defines frequency as inversely related to rank: frequency ∝1/rank.Source: §8.1 · p. 9
9 Conclusion
The conclusion reports that standard LLM setups can remain highly non-deterministic: raw responses rarely repeat, parsed answers are often more stable, and instability varies across tasks and models. It identifies further questions about reducing, managing, communicating, and diagnosing this instability.
9 Conclusion
- Standard LLM setups can be highly non-deterministic even with hyper-parameters intended to maximize determinism.The conclusion states that an LLM rarely produces the same response ten times for identical input.Source: §9 · p. 9
- Parsed answers are often more stable than raw responses, while instability is highly variable across tasks and models.The conclusion also reports that instability results are not normally distributed, complicating uncertainty measurement.Source: §9 · p. 9
- Open questions concern reducing instability during training or inference, incorporating it into products, communicating it to decision-makers, and relating it to error types.The authors specifically mention false positives and false negatives as possible error categories for further analysis.Source: §9 · p. 9
A.1 Correlation Matrices Per Model
This appendix presents metric-correlation matrices across few-shot and zero-shot settings for five models, alongside zero-shot agreement visualizations and a summary table of accuracy and agreement metrics. The supplied passages identify the visual encodings and scope but do not report cell-level comparisons.
A.1 Correlation Matrices Per Model
- Figures 6–10 compare Spearman correlations among metrics for GPT-4o, GPT-3.5-turbo, Llama-8b, Llama-70b, and Mixtral-8x7b across few-shot and zero-shot settings.Each matrix places few-shot results on the left and zero-shot results on the right.Source: Figure 6 · p. 13, Figure 7 · p. 13, Figure 8 · p. 13
- Figure 11 shows zero-shot TARa@10 for each task, using darker red hues for relatively lower-performing models and darker blue hues for relatively higher-performing models.Source: Figure 11 · p. 14
- Table 3 reports zero-shot BestAcc, median accuracy, WorstAcc, TARa@10, and TARr@10 as percentages.Source: Table 3 · p. 15
- Figure 12 shows zero-shot TARr@10 for each model, with darker red hues indicating relatively challenging datasets and darker blue hues indicating relatively easy datasets.Source: Figure 12 · p. 15