PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition

Tanmay Joshi, Shourya Aggarwal, Anusa Saha, Aadi Pandey, Shreyash Dhoot, Vighnesh Rai, Raxit Goswami, Aman Chadha, Vinija Jain, Amitava Das

arXiv:2601.07239v1cs.AI

TL;DR

Single-trajectory deterministic evaluation can misrepresent stochastic LLM behavior by obscuring capability, failure structure, and safety-relevant tail behavior. The paper proposes Stochastic CHAOS, treating output distributions as objects of measurement through stochastic exploration and multi-sample evaluation. Across tasks, greedy evaluation systematically hides latent competence and model failures, motivating distributional reproducibility and semantic stability over bitwise determinism.

  • Problem

    The paper addresses limited evidence from single-trajectory evaluation about how stochastic LLM behavior affects capability, failures, and safety assessment.

    Source: §1 · p. 2, §4.2.3 · p. 39

  • Method

    The paper models outputs as conditional distributions and evaluates decoding policies through stochastic exploration, multi-sample procedures, and hidden-tail-risk analysis.

    Source: §4.2.1 · p. 37, §6.2 · p. 60

  • Results

    Across classification, generation, reasoning, and safety tests, greedy single-trajectory evaluation systematically hides latent competence and the true shape of model failures.

    Source: §4.1.2 · p. 25, §7 · p. 78, §8 · p. 82

  • Takeaways & Limitations

    LLM evaluation and deployment should prioritize distributional reproducibility and semantic stability, while retaining deterministic modes for debugging and regression testing.

    Source: §8 · p. 82

    The study covers selected models and tasks, so its quantitative patterns may differ for larger mixture-of-experts, retrieval-augmented, or deeply multimodal systems.

    Source: §7.5 · p. 80

Abstract

from arXiv · show

Deterministic inference is a comforting ideal in classical software: the same program on the same input should always produce the same output. As large language models move into real-world deployment, this ideal has been imported wholesale into inference stacks. Recent work from the Thinking Machines Lab has presented a detailed analysis of nondeterminism in LLM inference, showing how batch-invariant kernels and deterministic attention can enforce bitwise-identical outputs, positioning deterministic inference as a prerequisite for reproducibility and enterprise reliability. In this paper, we take the opposite stance. We argue that, for LLMs, deterministic inference kills. It kills the ability to model uncertainty, suppresses emergent abilities, collapses reasoning into a single brittle path, and weakens safety alignment by hiding tail risks. LLMs implement conditional distributions over outputs, not fixed functions. Collapsing these distributions to a single canonical completion may appear reassuring, but it systematically conceals properties central to artificial cognition. We instead advocate Stochastic CHAOS, treating distributional variability as a signal to be measured and controlled. Empirically, we show that deterministic inference is systematically misleading. Single-sample deterministic evaluation underestimates both capability and fragility, masking failure probability under paraphrases and noise. Phase-like transitions associated with emergent abilities disappear under greedy decoding. Multi-path reasoning degrades when forced onto deterministic backbones, reducing accuracy and diagnostic insight. Finally, deterministic evaluation underestimates safety risk by hiding rare but dangerous behaviors that appear only under multi-sample evaluation.

1 What Do We Mean by “Determinism” in LLM Inference?

Determinism in LLM inference is layered rather than binary: intentional sampling, numerical and serving-system effects, and bitwise, distributional, and semantic stability define distinct behaviors. Although deterministic infrastructure can improve reproducibility, practical deployments often prioritize robust semantic and distributional stability over absolute bitwise identity.

1 What Do We Mean by “Determinism” in LLM Inference?

  • LLM determinism depends on the boundary and level of analysis: systems may have deterministic kernels yet produce variable user-facing outputs because external, internal, bitwise, distributional, and semantic notions differ (Chiang et al., 2013; Demmel et al., 2016; Atil et al., 2024; Kaikaus et al., 2024).The section separates algorithmic stochasticity, system-level nondeterminism, and user-facing stability.Source: §1 · p. 2

1.1 Idealized Determinism: Greedy Decoding at T=0

  • Even at T=0, floating-point perturbations, dynamic batching, scheduling, and kernel selection can change near-tied token choices and cause substantially divergent completions for the same prompt (Atil et al., 2024; He and Thinking Machines Lab, 2025).The user’s effective input may exclude concurrent requests and system state, creating a mismatch between server-level determinism and experienced behavior.Source: §1.1 · p. 2, §1.3 · pp. 3–4

1.2 Algorithmic Stochasticity: Sampling by Design

  • Sampling is intentional algorithmic stochasticity: temperature and top-k/top-p decoding generate varied outputs, while self-consistency and Tree-of-Thought use multiple reasoning trajectories to improve fluency, creativity, and correctness (Wang et al., 2023a; Yao et al., 2024).Unlike implementation randomness, this variability is an explicit decoding choice and can be removed by deterministic decoding.Source: §1.2 · p. 3

1.3 System-Level Nondeterminism

  • System-level nondeterminism arises from non-associative floating-point reductions, batch-dependent execution, hardware or software drift, and model updates, while deterministic kernels can enforce bitwise-identical outputs at performance and engineering cost (He and Thinking Machines Lab, 2025; Zhang et al., 2025).Deterministic modes require fixed reduction orders, controlled tiling, synchronization, and constrained environments, often reducing throughput.Source: §1.3 · pp. 3–5

1.4 Historical Perspectives

  • Inference inherits reproducibility trade-offs from HPC and training, but single-run serving makes stability especially important; deterministic infrastructure can remove variability without necessarily improving a probabilistic model’s distributional behavior (Demmel et al., 2016; He and Thinking Machines Lab, 2025).OpenAI’s seed and system_fingerprint controls provide visibility while promising only “mostly consistent” behavior (OpenAI, 2024).Source: §1.4 · p. 5

1.5 A Stability Taxonomy

  • The proposed taxonomy distinguishes costly bitwise determinism from distributional reproducibility and semantic stability, with most deployments targeting stable output distributions and task meaning rather than identical strings (Atil et al., 2024; Kaikaus et al., 2024).Distributional evaluation emphasizes repeated sampling and mean/variance, whereas semantic stability tolerates paraphrases when task outcomes remain consistent.Source: §1.5 · pp. 5–6

2 Letś Stress-Test “Deterministic Inference” in Practice

The section argues that deterministic inference is an engineering overlay on stochastic LLMs, and that treating bitwise reproducibility as the default can obscure generalization, reasoning, emergent abilities, and safety risks. It instead treats distributional variability as a signal to measure and control, while retaining deterministic modes for debugging and reproducibility.

Framing

  • Deterministic inference is an imposed engineering objective, although redesigned kernels and attention can produce bitwise-identical outputs under dynamic batching (He and Thinking Machines Lab, 2025).The deterministic ideal treats LLM inference as a pure function from prompts to strings.Source: §2 · p. 6

Central hypothesis

  • The paper treats nondeterminism as evidence about pθ(y | x), which supports uncertainty, multiple reasoning paths, and emergent behaviors (Wei et al., 2022a; Ganguli et al., 2022a).Its central question is what is lost when inference variability is defeated rather than whether it can be eliminated.Source: §2 · p. 6

Generalization

  • Single-output deterministic evaluation encourages canonical-answer optimization over robust semantic generalization, risking benchmark memorization and brittle progress.The section connects this concern to GLUE’s single-score saturation and argues that single deterministic runs can mask behavioral variability (Wang et al., 2018, 2019; Geirhos et al., 2020).Source: §2 · p. 6

Emergent abilities

  • Strict greedy decoding at T=0 suppresses exploration-dependent emergent abilities and systematically underestimates latent competence relative to multi-sample decoding.The affected behaviors include few-shot learning, chain-of-thought, and self-consistency gains on reasoning tasks (Brown et al., 2020; Wei et al., 2022b; Wang et al., 2023b; Yao et al., 2023; Wei et al., 2022a).Source: §2 · p. 7

Reasoning paths

  • Greedy decoding collapses diverse valid and near-miss reasoning paths into one brittle trace, concealing uncertainty and misclassifying nuanced item-level behavior.Multi-sample decoding exposes alternative strategies and failure modes, extending self-consistency-style analysis (Wang et al., 2023b).Source: §2 · p. 7

Safety and deployment

  • Deterministic safety evaluation can appear robust while missing rare harmful behaviors that emerge under modest stochasticity, paraphrased attacks, distribution shift, or prompt injection (Perez et al., 2022; Ganguli et al., 2022b; Greenblatt et al., 2024; Hubinger et al., 2024).Deterministic modes remain useful for debugging, regression testing, and exact on-policy RL, but the paper rejects determinism as a universal deployment and evaluation norm.Source: §2 · p. 7

3 Deterministic Inference Encourages Benchmark Memorization

This section examines how deterministic, single-output evaluation can turn generative-model assessment into benchmark-surface memorization rather than semantic robustness measurement. Across GLUE models and tasks, it shows that decoding policy materially affects observed robustness, with stochastic evaluation generally producing higher robustness ratios and tighter distributions, despite task-specific reversals.

3.1 GLUE as a Cautionary Tale

  • GLUE’s near-human or higher leaderboard scores were later linked to annotation artifacts, label biases, shortcut heuristics, and sharp drops under paraphrases, perturbations, and distribution shifts.The section presents finite-test-set, single-output evaluation as an evaluation surface that models can overfit while out-of-distribution behavior stagnates.Source: §3.1 · p. 8

3.2 From Label Determinism to Sequence Determinism

  • Deterministic inference reduces a conditional sequence distribution to one canonical completion and one correctness bit, replaying GLUE’s single-label evaluation error at the sequence level.Temperature sampling, nucleus sampling, self-consistency, and Tree-of-Thought probe different parts of the distribution and can reveal capabilities hidden by greedy decoding.Source: §3.2 · pp. 8–9

3.3 Experimental Setup: GLUE-Style Robustness Under Decoding Choices

  • The robustness protocol evaluates MNLI, QQP, QNLI, and SST-2 across original, paraphrased, perturbed, and adversarially paraphrased variants using deterministic decoding or K = 10 stochastic samples.It compares single-completion accuracy with stochastic majority-vote accuracy and uses a robustness ratio to separate absolute competence from generalization.Source: §3.3 · pp. 9–10

3.4 A GLUE Robustness Heatmap for Deterministic vs. Stochastic Inference

  • Stochastic rows are consistently greener than deterministic rows across tasks and models, indicating higher robustness to paraphrased, perturbed, and adversarial inputs.Figure 1’s robustness matrix shows that majority-vote evaluation better captures distributional generalization, whereas greedy decoding can hide both semantic diversity and failure tails.Source: Figure 1 · p. 11, §3.4 · pp. 11–12
  • Per-model analyses show stochastic decoding’s robustness advantage is cross-task and consistent, while its magnitude and variance profiles differ across architectures.The violin plots expose robustness variability across perturbation types and resampled examples rather than reducing each task-model-decoder combination to one heatmap cell.Source: §3.4 · p. 12
  • Decoding sensitivity varies by model and task: GPT-4o, GPT-3.5, and Mixtral show broad stochastic advantages, while Vicuna-7B and Mistral-7B exhibit task-specific reversals.Deterministic decoding is marginally better for Mistral-7B on SST-2 and Vicuna-7B on SST-2, despite stochastic gains elsewhere.Source: Figure 7 · p. 15, Figure 8 · p. 15, Figure 14 · p. 18
  • Inference-heavy tasks such as MNLI and QNLI are frequent decoding-sensitivity hotspots, with deterministic decoding producing lower medians and wider robustness distributions.This pattern is reported for LLaMA-2 7B, LLaMA-2 13B, and Mixtral-8×22B, while GPT-3.5 also shows large downward shifts on MNLI.Source: Figure 8 · p. 15, Figure 9 · p. 16, Figure 10 · p. 16
  • Scaling and architectural improvements raise robustness bands, but stochastic decoding retains its advantage for LLaMA and Mixtral models.LLaMA-2 13B improves over 7B, LLaMA-3 70B improves over 7B, and Mixtral-8×22B shows a particularly large QNLI margin.Source: Figure 10 · p. 16, Figure 13 · p. 18, Figure 16 · p. 19
  • Stochastic decoding generally outperforms deterministic decoding across GLUE models, with robustness gaps reaching about 0.18 absolute points.The advantage is strongest for LLaMA-3 8B on SST-2 and for Phi-2, while some tasks show reversals.Source: Figure 7 · p. 15, Figure 8 · p. 15, Figure 10 · p. 16
  • Small models can conceal substantial latent robustness under deterministic decoding, as shown by Phi-2’s roughly 0.10–0.17-point stochastic gains.Phi-2’s stochastic ratios reach 0.96–0.98 on QNLI, whereas deterministic decoding remains in the 0.74–0.80 band.Source: Figure 17 · p. 20

4 Deterministic Decoding Suppresses Exploration–Driven Abilities … qICL

The section argues that emergent abilities depend on the model–decoder system: greedy decoding can suppress successful trajectories that remain probable under pθ(τ | x). Across few-shot classification and constrained generation, exploratory multi-sample decoding can recover substantially higher performance, motivating a probability-based analysis of ICL success.

4 Deterministic Decoding Suppresses Exploration–Driven Abilities

  • Deterministic inference stifles emergence by collapsing a rich trajectory distribution into one greedy path, potentially hiding abilities that remain well represented under pθ(τ | x).If the success set excludes the unique maximizing trajectory but contains many nearby high-probability trajectories, the model can appear incapable despite substantial latent success mass.Source: §4 · p. 21
  • Greedy decoding induces a low-entropy kernel concentrated on one maximizing trajectory, whereas sampling with selection explores a richer subset of latent behaviors.The trajectory-space formalism represents successful outputs as a set S(x) and decoder behavior as a kernel Ke(τ | x, θ), allowing success to vary with policy.Source: §4 · p. 21
  • +10–30 absolute points of accuracy or constraint satisfaction can separate exploratory decoding from greedy decoding across standard ICL and controllable-generation benchmarks.The result supports treating emergence as a property of the model–decoder pair, because deterministic decoding never visits many competent trajectories.Source: §4 · p. 22

4.1 Few–Shot In–Context Learning Under Decoding Policies

  • The study tests few-shot sentiment and sarcasm classification on BESSTIE while varying only the decoding policy, using fixed prompts and model weights.BESSTIE covers three English varieties and is chosen as a post-2024 benchmark to reduce likely labeled-data contamination; evaluation uses 4- or 8-shot demonstrations without task-specific finetuning.Source: §4.1 · p. 22

4.1.1 Quantifying In–Context Ability and Exploration Gains

  • The experimental design quantifies each task–model–decoding combination through held-out predictions, treating ICL accuracy as an empirical estimator of success probability.The analysis indexes tasks, models, and decoding regimes and evaluates predictions on held-out examples under fixed prompt templates.Source: §4.1.1 · pp. 22–23

AccICL

  • AccICL is framed as the standard empirical ICL accuracy while explicitly accounting for decoder randomness such as sampling noise and random seeds.The resulting label is indexed by task, model, input, and decoding policy, with randomness represented separately by r.Source: AccICL · p. 23

qICL

  • qICL interprets empirical accuracy as an estimate of per-example success probability under a decoding policy rather than as a fixed model property.This framing separates the model’s latent distribution from the realized outcome of a particular stochastic or deterministic decoder.Source: qICL · p. 23, AccICL · p. 23

AccICL · qICL

The qICL section contrasts deterministic decoding’s degenerate outcomes with exploratory decoding’s distribution over trajectories and hidden success mass. It then measures exploration gains using best-of-k self-consistency decoding.

qICL

  • qICL averages input-wise success probabilities to quantify performance across inputs.Source: qICL · p. 23
  • Deterministic decoding, such as greedy decoding with T=0, is a degenerate regime with nearly constant outcomes across seeds.Source: qICL · p. 23
  • Exploratory decoding induces trajectory distributions in which qICL captures the hidden success mass available.Source: qICL · p. 23
  • The central quantity is the exploration gain: what the model could achieve through exploration versus what greedy decoding actually achieves.Source: qICL · p. 23
  • Best-of-k self-consistency decoding samples k completions under a stochastic base policy as its exploration budget.Source: qICL · p. 23
  • The best-of-k procedure maps sampled completions to discrete labels and returns the majority label, forming the composite regime ebest-k.Source: qICL · p. 23

AccICL

AccICL characterizes exploration at budget k and analyzes per-example success under stochastic sampling and best-of-k majority voting. It contrasts this stochastic treatment with greedy decoding, defined as the standard T=0 deterministic decoder.

AccICL

  • AccICL defines an exploration gain at sampling budget k.Source: AccICL · p. 23
  • Greedy decoding is the standard T=0 deterministic decoder.Source: AccICL · p. 23
  • At the per-example level, qICL_i,t,m(estoch) denotes the probability that one stochastic sample produces the correct label.Source: AccICL · p. 23
  • Best-of-k majority voting converts single-sample correctness probabilities into a success probability for each x_i.Source: AccICL · p. 23

qICL · (qICL

The section introduces an exploration-gain quantity based on the probability that at least half of k draws are correct, averaged over i.

(qICL

  • The exploration gain is defined through the probability that at least half of the k draws are correct and is averaged over i.Source: (qICL · p. 24

qICL

The qICL section examines how stochastic exploration and sampling reveal capabilities that greedy decoding can miss across open-model ICL benchmarks. It characterizes how these benefits vary by model, task, configuration, and intermediate uncertainty, including the effectiveness of small sampling budgets.

qICL

  • qICL increases when sampling recovers correct labels that greedy decoding misses, because best-of-k can approach the stochastic policy’s success probability without changing model parameters.A binary example gives q=0.6, qICL(greedy)=0, and best-of-9 success of approximately 0.73; positive EGICL measures this latent-capacity gap.Source: qICL · p. 24
  • Sampling distributions frequently have a correct-label majority despite an incorrect greedy label, so majority voting exploits structured hidden capacity that deterministic decoding discards.Low entropy indicates near-deterministic behavior, whereas intermediate entropy reveals multiple plausible labels; aggregating these hidden-majority cases predicts positive exploration gains.Source: qICL · p. 24

4.1.2 ICL Results: Exploration Recovers Suppressed Ability

  • Best-of-k exploration raises ICL accuracy by typically +10–20 points by k=16, with further +2–5 points at k=64; greedy decoding often remains at 40–65%.Figure 20 reports 8–22 absolute-point drops from best-of-16 to greedy decoding across both BESSTIE tasks and most models.Source: §4.1.2 · pp. 25–26, Figure 20 · p. 26
  • Exploration gains are common across tasks and models, with many fixed-budget k=16 gains of 0.08–0.20 and some exceeding 0.22, including smaller models.The same model–prompt pair can appear mediocre under greedy decoding yet competitive under modest stochastic exploration; LLaMA–2 7B/13B and Phi–2 often show larger relative gains.Source: §4.1.2 · pp. 25–26

4.1.3 Exploration–ICL Landscapes across Models

  • Strong models show nearly flat landscapes, while mid-size models have sharp 10–20-point ridges around T≈0.7 and k∈[8,32], and sarcasm often gains 15–17 points.LLaMA–3 70B peaks at approximately 5 points on sentiment and 10 on sarcasm; sarcasm gains are especially broad for Vicuna–7B, Gemma–2 9B, and Phi–2.Source: §4.1.3 · p. 27
  • Landscape gains depend on scale, task, architecture, and tuning: larger models flatten on easier tasks, Mixtral remains near 3–12 points, and narrow sweet spots can collapse under poor T or k choices.Phi–2 reaches approximately 11 points on sentiment in tuned pockets, while strong models still retain some exploration headroom.Source: §4.1.3 · pp. 27–28
  • Across models, exploration landscapes peak near T≈0.65–0.8 and k∈[8,24], but gains vary by task and backbone, from Mixtral’s ≈7 pp to Gemma-2’s ≈10–20 pp.Mixtral variants show task-agnostic moderate benefits, whereas Gemma-2 exhibits stronger sentiment than sarcasm dependence; other models also show marked task asymmetries.Source: §4.1.3 · p. 31, Figure 26 · p. 31, Figure 27 · p. 32

4.1.4 Entropy–Exploration Tradeoffs in Few–Shot ICL

  • For δ=0.05, many task–model pairs reach a ≥5 pp gain with best-of-4, while only a minority require k=64 to cross the 10 pp threshold.Even under δ=0.10, many pairs have k⋆ ∈{4, 16}, indicating that extreme sampling budgets are usually unnecessary.Source: §4.1.4 · p. 34
  • In hidden-majority cases, stochastic sampling exposes the correct dominant label while greedy decoding selects a locally probable but globally suboptimal verbalization.This mechanism explains the largest intermediate-entropy gains: best-of-k reweights trajectory space toward the sampled majority label.Source: §4.1.4 · pp. 34, 36
  • Intermediate normalized entropy (≈0.3–0.7) produces the largest gains, EGICL ≈0.10–0.20, whereas low (≲0.2) and high (≳0.8) entropy yield negligible gains.The relationship is consistently inverted-U shaped across models, with low entropy indicating certainty and high entropy indicating diffuse confusion.Source: §4.1.4 · pp. 34–35
  • Deployment should concentrate exploration on medium-entropy inputs, while handling very low- or high-entropy cases with cheap deterministic decoding.A modest best-of-k stack, often k∈{4,16}, can recover double-digit gains where exploration matters most.Source: §4.1.4 · pp. 35–36

4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search

This section frames InstruSum summarization as multi-objective search over semantic adequacy and instruction constraints, showing how decoding policies expose different regions of the same model distribution. It defines style exploration gain and evaluates whether small multi-sample budgets improve joint instruction satisfaction without changing model parameters.

4.2.1 Task Setup and Multi–Objective View

  • InstruSum pairs news articles with reference summaries and natural-language requirements covering length, content focus, and optional style or format.The benchmark explicitly tests instruction-following behavior in open-ended summarization.Source: §4.2.1 · pp. 36–37
  • Each candidate summary is scored jointly for semantic adequacy, length, inclusion, avoidance, and style or format satisfaction within a multi-objective trajectory space.Decoding policies induce different distributions over trajectories and therefore explore different regions of the same semantic–constraint landscape.Source: §4.2.1 · p. 37

4.2.2 Metrics, Success Sets, and Style Exploration Gain

  • The evaluation converts these component scores into thresholded success sets and defines style exploration gain as the improvement over greedy decoding achieved by alternative policies.This isolates benefits attributable to distributional exploration rather than parameter changes.Source: §4.2.2 · p. 38

4.2.3 Decoding Policies as Multi–Objective Search Strategies

  • Greedy decoding commits to one objective-space corner, whereas multi-sample lexicographic selection prioritizes joint constraint satisfaction before semantic adequacy and seeks a better local trade-off.The policy uses a small sampled neighborhood to move toward the instruction-satisfying Pareto frontier.Source: §4.2.3 · p. 39

4.2.4 Experimental Protocol on InstruSum

  • All policies share the same model distribution, so gains from multi-sample decoding indicate that deterministic evaluation can hide instruction-following ability already present in pθ(· | di, ri).The experiments compare greedy, single-sample, and multi-sample lexicographic decoding across budgets k ∈{4, 8, 16, 32}.Source: §4.2.3 · p. 39, §4.2.4 · p. 40

4.3 Results: Stochastic Search Unlocks Latent Instruction Following

On InstruSum, multi-sample search reveals instruction-following competence already present in fixed model distributions, with most gains arriving by k≈8 and varying substantially by model. Density landscapes explain these gains as movement from under-constrained probability ridges toward semantically and stylistically aligned regions.

4.3.1 How Much Exploration is Enough, and Where Do Gains Come From?

  • Style exploration gains rise sharply between k=2 and k=8, mostly saturate by k⋆=8, and reach EGstyle m (k⋆)≈0.10–0.13 for larger recent models.Smaller models such as Phi–2 saturate quickly at low gains, while k∈{4, 8} often captures most available improvement.Source: §4.3.1 · pp. 41–42, Figure 32 · p. 42
  • Models’ ICL and style exploration gains correlate positively at ρ≈0.80, but outliers such as Phi–2 show that explorability is architecture-dependent across tasks.Mixtral–8×22B is strong on both dimensions, whereas Phi–2 is ICL-strong but style-weak.Source: §4.3.1 · pp. 42–43, Figure 33 · p. 43
  • At k=8, clen and cstyle typically jump by ≈0.07–0.13, while ssem improves ≈0.02–0.03 and cavoid changes only slightly.Search primarily improves length and style compliance rather than raw content quality or hallucination avoidance.Source: Table 1 · p. 41, §4.3.1 · pp. 42–43
  • At k=8, P style succ rises from 0.48 →0.64 for LLaMA–3 and 0.49 →0.68 for Mixtral–8×22B, while Phi–2 improves 0.30 →0.35.These gains occur without changing model weights, indicating that stochastic search reaches instruction-compliant trajectories missed by greedy decoding.Source: §4.3 · p. 40, Table 1 · p. 41, §4.3.1 · p. 43

4.3.2 Semantic–Constraint Density Landscapes on InstruSum

  • Density landscapes show many models concentrating probability on semantically adequate but under-constrained ridges, with only thin tails reaching high semantics and high constraints.Greedy decoding anchors outputs to these ridges even when aligned candidates exist in the distribution’s tails.Source: §4.3.2 · pp. 44–45, Figure 35 · p. 46
  • LLaMA–3, Gemma–2, Mixtral–8×7B, and Mixtral–8×22B shift more mass toward the upper-right region, with peaks around ssem(τ)∈[0.65, 0.85] and cjoint(τ)∈[0.35, 0.60].This greater high-quality mass explains why modest search budgets routinely surface well-aligned summaries, unlike Phi–2’s tiny high-constraint island.Source: §4.3.2 · p. 44, Figure 35 · p. 46
  • Model-specific landscapes range from LLaMA–2’s ridge at ssem(τ)∈[0.50, 0.70], cjoint(τ)∈[0.15, 0.35] to LLaMA–3’s upper-right mass near (0.70−0.80, 0.45−0.60).Gemma–2 has a broader mixed landscape, while Mistral–7B prioritizes semantics despite retaining higher-constraint candidates.Source: §4.3.2 · p. 45

5 Deterministic inference collapses diverse reasoning paths into a single brittle trace

The section argues that greedy decoding collapses the model’s distribution over valid and near-miss reasoning chains into one brittle trajectory, making reasoning ability and failure modes difficult to assess. Multi-sample decoding instead exposes alternative strategies, including correct paths that deterministic inference prunes.

5 Deterministic inference collapses diverse reasoning paths into a single brittle trace

  • Multi-sample decoding reveals multiple qualitatively distinct correct strategies and a spectrum of near-miss failures that greedy evaluation makes invisible.The sampled chains are organized as reasoning graphs whose leaves are labeled correct, near-correct, or clearly incorrect.Source: §5 · p. 47
  • Greedy decoding collapses a model’s diverse chain-of-thought distribution into one low-support, brittle trace, causing solvable problems to appear as deterministic “collapsed failures.”A correct multi-step strategy may exist in the sampled reasoning graph but remain unexpressed under the deterministic policy.Source: §5 · p. 47

5.1 Tasks and decoding setup

  • The evaluation studies GSM8K, SVAMP, and StrategyQA using instruction-tuned open-weight models and includes a greedy decoding regime with temperature T=0 and no sampling.Source: §5.1 · pp. 47–48

5.2 From sampled chains to reasoning graphs

  • The method converts sampled chains into finite reasoning graphs by segmenting intermediate steps, embedding them, and merging similar prefixes with a cosine-similarity threshold δ∈[0.85, 0.90].This representation treats each chain as a path through shared partial reasoning states, with complete rationales and final answers at the leaves.Source: §5.2 · p. 48

2. Process each sampled chain τ (j)

Each sampled chain is processed by extending a path from v_root step by step. At each step, the procedure reuses an existing node for a ∼-equivalent stored prefix or creates a new node, then adds a directed edge.

Process each sampled chain τ (j)

  • The procedure extends each sampled chain from v_root by iterating over its path prefixes.Source: §2 · p. 49
  • When a node already stores a ∼-equivalent prefix to π(j) i,t, the procedure reuses that node.Source: §2 · p. 49
  • Otherwise, it creates a new node representing π(j) i,t and adds it to V(m).Source: §2 · p. 49
  • In either case, the procedure creates a directed edge from the node encoding the current prefix and adds it to E(m).Source: §2 · p. 49

3. Once the last step s(j)

This section examines how multi-sample decoding exposes diverse reasoning paths, alternative high-quality outputs, and instruction-following capacity that greedy decoding can hide, across reasoning benchmarks, models, and tasks.

3. Once the last step s(j)

  • The reasoning graph shares common prefixes, labels leaves as Correct, NearMiss, or Failure, and shows deterministic inference selecting one root-to-leaf path while discarding alternatives.NearMiss requires a largely overlapping correct strategy, with α=0.7 in the experiments; the illustrative graph includes correct, near-miss, and failure branches.Source: §3 · p. 49

5.3 Empirical evidence of path diversity and collapsed failures

  • Across GSM8K and SVAMP, most models produce 4–6 distinct reasoning paths with k=16 samples, while StrategyQA often yields several semantically valid fact chains.Stronger instruction-tuned models frequently admit multiple correct strategies, including alternative decompositions or factual arguments supporting the same answer.Source: §5.3 · p. 51
  • Greedy decoding often selects a low-support, nonrepresentative branch, even when alternative correct paths exist, thereby pruning valid solutions and creating policy-induced collapsed failures.On GSM8K, SVAMP, and StrategyQA, greedy traces can follow shortcuts or omit evidence while sampled branches contain more careful correct reasoning.Source: §5.3 · pp. 51–52

5.4 Results: exploration reveals hidden strategies and near–miss failures

  • Models with richer reasoning-path diversity gain more from multi-sample decoding, with LLaMA–3 and Mixtral–8×22B outperforming lower-diversity Phi–2 and Vicuna–7B in exploration gains.The relationship holds qualitatively across GSM8K, SVAMP, and StrategyQA: higher path entropy or more strategy clusters corresponds to larger multi-sample accuracy improvements.Source: §5.4 · p. 52
  • Collapsed failures are greedy errors with a correct sampled path, whereas brittle successes are greedy-correct instances with fewer than 30% correct samples; both reveal instability hidden by single traces.Collapsed failures are especially common for stronger models, and brittle successes dominate robust successes for many models across the three datasets.Source: §5.4 · p. 53
  • Multi-sample decoding recovers 5–15 absolute points of CoT accuracy beyond greedy, while greedy support decreases with model strength and collapsed failures comprise nearly a third of some greedy mistakes.These trends are summarized across GSM8K, SVAMP, and StrategyQA in Table 2, with larger gains and more collapse among models with higher reasoning-path entropy.Source: §5.4 · p. 53, Table 2 · p. 53
  • Multi-sample decoding reveals richer latent strategies and near-miss attempts, whereas deterministic decoding collapses multi-path reasoning into a single brittle trace.The underlying model distribution pθ(· | x) is unchanged; exploration changes which of its outputs become visible.Source: §5.4 · p. 54
  • LLaMA–3.1–8B shows Hm,d≈1.35–1.70 and ∆Accm,d≈0.03–0.12 under stochastic decoding, with strongest gains on StrategyQA.Arithmetic datasets show smaller but consistent improvements in diversity and accuracy, while greedy decoding remains near ∆Accm,d≈0.Source: Figure 38 · p. 55
  • Mixtral–8×7B–Instruct shows Hm,d≈1.30–1.55 and ∆Accm,d≈0.02–0.10, indicating moderate, consistent exploration gains, especially on GSM8K and SVAMP.Its smaller greedy-to-stochastic shifts suggest deterministic inference collapses less of its reasoning landscape than in DeepSeek or LLaMA–3.1.Source: Figure 39 · p. 56

6 Deterministic safety evaluation creates an illusion of robustness

This section examines how deterministic safety evaluation can conceal harmful probability mass and create an illusion of robustness. It formalizes concealed stochastic risk and presents metrics, visualizations, Monte Carlo estimation, and case studies showing how multi-sample decoding reveals prompt-level harm and oversight sensitivity missed by greedy evaluation.

6 Deterministic safety evaluation creates an illusion of robustness

  • Greedy safety checks can certify models as safe while realistic stochastic decoders expose substantial harmful mass in non-argmax tails, with the mismatch worsening for stronger models and larger sampling budgets.The section’s central claim is that deterministic-only safety assessment is fundamentally incomplete and can drastically underestimate deployment risk.Source: §6 · pp. 57–58

6.1 Threat model, benchmarks, and decoders

  • The threat model treats any harmful completion among repeated queries as a failure, covering jailbreaks, indirect roleplay prompts, oversight framing, and harmful outputs such as self-harm, weapons, biological, or harassment guidance.Evaluations compare greedy decoding, temperature sampling at T ∈{0.7, 1.0} with top-p=0.9, multi-sample budgets k ∈{8, 16, 32}, and optional adversarial search.Source: §6.1 · pp. 58–59

6.2 Formalizing deterministic vs. stochastic risk

  • For harmful mass qθ(x), k-sample risk is rk(x) = 1 −(1 −qθ(x))k, whereas a safe greedy argmax reports zero risk regardless of deployment sampling budget.With ε = 0.01 and k=16, stochastic risk is approximately 0.15–0.16, so a prompt deemed safe greedily can fail in roughly one of six stochastic runs.Source: §6.2 · pp. 60–61
  • The deterministic illusion index measures hidden stochastic risk and increases systematically with both model capability and search budget, indicating that stronger models can appear safest under greedy decoding while hiding fatter harmful tails.The formalism also defines per-prompt hidden tail mass and compares aggregate greedy risk with k-sample risk.Source: §6 · p. 57, §6.2 · p. 60

6.3 Metrics and categories for concealed risk

  • The evaluation reports decoder-level attack success rates and the stochastic risk gap ΔASRm(k), then classifies prompts as robust safe, concealed risk, or deterministic failure.These three events are disjoint and exhaust the prompt set, distinguishing harmful greedy outputs from harmful behavior revealed only by stochastic samples.Source: §6.3 · pp. 61–62
  • Stronger models show lower ASRgreedy but much higher ∆ASRm(8) and Im(8), indicating that most stochastic risk is concealed under deterministic evaluation.The stochastic risk gap is numerically close to ρconcealed when harmful tails are reliably sampled, while Im(8) measures the fraction of stochastic risk invisible under greedy evaluation.Source: §6.3 · pp. 63–64, Table 4 · p. 64
  • Across models, risk surfaces show broad low-risk plateaus but sharp high-risk tails where k=8 or k=16 raises risk to approximately 0.8–1.0.Scaling enlarges the safe plateau and lowers greedy risk, but leaves concentrated dangerous prompts; compact models more often show high risk under both greedy and stochastic decoding.Source: §6.3 · pp. 64–65, Figure 41 · p. 66, Figure 42 · p. 66

6.4 Experimental design and key visualizations

  • Greedy evaluation systematically underestimates risk, with underestimation growing alongside model capability and permitted distributional exploration across risk curves, analytic surfaces, and composition plots.For harmful mass q=0.01, analytic risk reaches approximately 0.16 at k=16 and exceeds 0.28 at k=32, while k=1 can report zero risk.Source: §6.4 · pp. 72–73, Figure 54 · p. 73

6.5 Case studies: concrete prompts and empirical harmful tails

  • Per-prompt Monte Carlo sampling estimates harmful probability mass qθ(x) and empirical tail risk, with variance qθ(x)(1−qθ(x))/k and sharp concentration at modest budgets.The estimator uses harmful indicators from independent stochastic samples; k=32 can provide a sharply concentrated estimate when qθ(x) is not vanishingly small.Source: §6.5 · p. 74
  • Stronger, more safety-tuned models exhibited little deterministic-failure mass but large concealed-risk bands, so most stochastic risk appeared safe under greedy evaluation.The figure partitions prompts into robustly safe, concealed risk, and deterministic failure, with concealed risk defined by a safe greedy completion but at least one harmful stochastic sample.Source: §6.5 · p. 75, Figure 56 · p. 75
  • With 3 harmful completions among 16, a prompt judged safe by greedy decoding had estimated tail risk above 0.95 and an illusion ratio of about 10^5.The greedy risk estimate was 0, whereas the Monte Carlo harmful probability was approximately 0.19, yielding r_16(x⋆) ≈ 1 − (1 − 0.19)^16 > 0.95.Source: §6.5 · p. 76, Figure 57 · p. 77
  • For some prompts, harmful mass of 1%–10% implies risk in the tens of percent for realistic sample counts, even when greedy evaluation reports zero risk.Source: §6.5 · p. 76
  • Oversight framing can substantially change the harmful tail, making models appear oversight-robust at the mode while sampling reveals strong oversight differentials.The difference between q_θ(x_watch) and q_θ(x_hidden) can be much larger in the tails than at the mode.Source: §6.5 · p. 76
  • Because harmful mass and tail risk are inherently distributional, greedy-only protocols cannot recover them and should be replaced by decoder-aware, tail-sensitive evaluation.The case studies are illustrative rather than representative, and the gap between greedy safety and sampled risk widens as models become more capable and responsive to oversight cues.Source: §6.5 · p. 76

7 Discussion and Limitations

The discussion argues that bitwise determinism is a design choice that can conceal LLM competence, fragility, uncertainty, and safety risks by reducing stochastic distributions to single trajectories. It recommends distributionally faithful evaluation and decoder-aware deployment while delimiting conclusions by model, decoding, metric, contamination, and infrastructure limitations.

7 Discussion and Limitations

  • Across classification, generation, reasoning, and safety tests, single-trajectory evaluation hides latent competence and the true shape of model failures.LLMs express pθ(y | x), while inference code determines which parts of that distribution become visible.Source: §7 · p. 78, §7.1 · p. 78

7.1 Determinism as a Design Choice, Not a Default Law

  • Deterministic decoding selects one fragile path, while higher-entropy policies can reveal behaviors supported by pθ(· | x), making emergence a property of model–decoder pairs.Greedy decoding can miss valid behaviors and narrow the observed trajectory space.Source: §7.1 · p. 78

7.2 Reproducibility vs. Distributional Faithfulness

  • Bitwise reproducibility is neither necessary nor sufficient: stochastic systems can yield stable distributional estimates, whereas greedy outputs can conceal conflicting or unsafe probability mass.For robustness, tail-risk, and extreme-event questions, distributional faithfulness matters more than identical strings.Source: §7.2 · p. 79

(1) From single–shot scores to stochastic profiles

(2) Robustness as a first–class metric

7.4 Implications for System Design and Deployment

  • Deployment should separate serving requirements from distributional monitoring: deterministic defaults can underestimate risk and capability while overstating robustness, despite remaining essential for exact replay and regression testing.Safety systems may appear extremely safe under greedy evaluation while retaining harmful mass under modest multi-sample decoding.Source: §7.4 · p. 80

7.5 Limitations and Threats to Validity

  • The conclusions are bounded by limited model and task coverage, decoding choices, possible data contamination, discrete or ambiguous metrics, and temporal infrastructure drift.The qualitative divergence between single-trajectory and stochastic behavior should be rechecked across new domains and future model families.Source: §7.5 · pp. 80–81

7.6 Open Directions

  • Future work should develop principled stochastic evaluators, distributional-robustness training, rare-event safety monitors, and extensions to multimodal, tool-using, and agentic systems.The overall aim is to measure, shape, and leverage stochasticity rather than suppress it.Source: §7.6 · p. 81

8 Conclusion

The conclusion argues that deterministic evaluation misrepresents stochastic LLM behavior by hiding emergent abilities, multi-path reasoning, and tail risks. It calls for controlled randomness, distributional metrics, ensemble reasoning, tail-focused safety audits, and diagnostic rather than default determinism.

Core message

  • Stochastic CHAOS shows that greedy, bitwise-reproducible outputs hide emergent abilities, collapse multi-path reasoning, and create robustness illusions that disappear under distributional analysis.The conclusion frames treating LLMs as deterministic programs as a category error because their important behavior is stochastic.Source: §8 · p. 82

Conceptual reframing

  • The study distinguishes bitwise determinism, distributional reproducibility, and semantic stability, arguing that optimizing the first can undermine the latter two.Even modest multi-sample budgets and lightweight perturbations expose gaps in instruction following, compositional generalization, multi-step reasoning, and safety.Source: §8 · p. 82

Benchmark design

  • Benchmarks should prioritize multi-sample success probabilities, exploration gains, and risk gaps, documenting emergent capabilities as changes in success probability rather than binary outcomes.The recommendation is to elevate distributional metrics to first-class status instead of celebrating single-run scores.Source: §8 · p. 82

Reasoning evaluation

  • Reasoning evaluation should analyze ensembles of trajectories for diversity, stability, and failure modes instead of treating one canonical chain-of-thought as the reasoning path.The conclusion characterizes reasoning as an ensemble phenomenon.Source: §8 · p. 82

Safety governance

  • Safety audits should explore low-probability regions through paraphrased attacks, mixed decoders, and small sampling budgets rather than certify models under one deterministic recipe.The proposed redesign targets distributional tails instead of modal behavior.Source: §8 · p. 82

Deployment and future work

  • Determinism should serve debugging, ablation, and regression testing, while engineering prioritizes robust distributional reproducibility and semantic stability under noisy serving conditions.The study is scoped to subsets of models, decoding policies, and stress tests, leaving richer samplers, adaptive risk-focused budgets, training regularizers, and operational risk guarantees open.Source: §8 · p. 82

blog.connectionism.ai/defeating-nondeterminism-in-llm-inference. Blog post.

The supplied passages are bibliographic entries covering reproducibility and nondeterminism, reasoning and emergent abilities, and evaluation datasets and benchmarks; they do not provide the blog post’s substantive arguments or results.

Datasets and benchmarks

  • The bibliography further lists datasets and benchmarks for reading comprehension, summarization, formality transfer, mathematics, sentiment, natural-language understanding, and inference.Named resources include InstruSum, SVAMP, GYAFC, BESSTIE, GLUE, and the broad-coverage inference corpus.Source: blog.connectionism.ai/defeating-nondete… · pp. 84–85

Model behavior and safety

  • Related references address emergent abilities, neural text degeneration, deceptive models, and model-written evaluations.These entries include work by Wei et al. and Sagawa et al. on emergent abilities, Holtzman et al. on text degeneration, Hubinger et al. on sleeper agents, and Perez et al. on model-written evaluations.Source: blog.connectionism.ai/defeating-nondete… · pp. 84–86

Reasoning methods

  • It also cites research on chain-of-thought, self-consistency, zero-shot reasoning, and tree-of-thoughts methods for language-model problem solving.The listed works include Kojima et al. on zero-shot reasoning, Wang et al. on self-consistency, and Yao et al. on tree-of-thoughts.Source: blog.connectionism.ai/defeating-nondete… · pp. 84, 86

Reproducibility and nondeterminism

  • The bibliography includes work on deterministic implementations, reproducible outputs, and framework-level sources of randomness or nondeterminism.Cited examples include deterministic reinforcement-learning implementations, OpenAI reproducible outputs, and PyTorch reproducibility guidance.Source: blog.connectionism.ai/defeating-nondete… · p. 85

Related papers

Loading 2601.07239v1…