PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition

Tanmay Joshi, Shourya Aggarwal, Anusa Saha, Aadi Pandey, Shreyash Dhoot, Vighnesh Rai, Raxit Goswami, Aman Chadha, Vinija Jain, Amitava Das

arXiv:2601.07239v1cs.AI

TL;DR

The paper asks whether importing deterministic inference into LLM deployment obscures important properties of stochastic models. It proposes Stochastic CHAOS, using distributional variability as an object of measurement and control, and reports that greedy evaluation hides competence, fragility, reasoning diversity, and safety risks. The authors conclude that distributional reproducibility and semantic stability should take precedence over global bitwise determinism, while deterministic modes remain useful for diagnostics.

  • Problem

    Single-sample deterministic evaluation provides limited evidence about LLM capability, failures, and safety because model behavior is represented by a conditional output distribution.

  • Method

    The paper disentangles bitwise, distributional, and semantic stability and stress-tests deterministic versus stochastic decoding across capability, reasoning, and safety settings.

  • Results

    Across the studied settings, greedy single-trajectory evaluation systematically hides latent competence and the true shape of model failures.

  • Takeaways & Limitations

    LLM evaluation and deployment should prioritize distributional reproducibility and semantic stability, while retaining deterministic modes for debugging and regression testing.

  • Takeaways & Limitations

    The quantitative patterns may differ for other architectures and tasks, and alternative decoding strategies may change precise numbers.

Abstract

from arXiv · show

Deterministic inference is a comforting ideal in classical software: the same program on the same input should always produce the same output. As large language models move into real-world deployment, this ideal has been imported wholesale into inference stacks. Recent work from the Thinking Machines Lab has presented a detailed analysis of nondeterminism in LLM inference, showing how batch-invariant kernels and deterministic attention can enforce bitwise-identical outputs, positioning deterministic inference as a prerequisite for reproducibility and enterprise reliability. In this paper, we take the opposite stance. We argue that, for LLMs, deterministic inference kills. It kills the ability to model uncertainty, suppresses emergent abilities, collapses reasoning into a single brittle path, and weakens safety alignment by hiding tail risks. LLMs implement conditional distributions over outputs, not fixed functions. Collapsing these distributions to a single canonical completion may appear reassuring, but it systematically conceals properties central to artificial cognition. We instead advocate Stochastic CHAOS, treating distributional variability as a signal to be measured and controlled. Empirically, we show that deterministic inference is systematically misleading. Single-sample deterministic evaluation underestimates both capability and fragility, masking failure probability under paraphrases and noise. Phase-like transitions associated with emergent abilities disappear under greedy decoding. Multi-path reasoning degrades when forced onto deterministic backbones, reducing accuracy and diagnostic insight. Finally, deterministic evaluation underestimates safety risk by hiding rare but dangerous behaviors that appear only under multi-sample evaluation.

1 What Do We Mean by “Determinism” in LLM Inference?

LLM determinism has distinct algorithmic, systems, and user-facing meanings. Although sampling is intentionally stochastic, even T=0 inference can vary because numerical execution, batching, and hardware conditions affect outputs; strict determinism also carries performance and engineering costs.

  • Stability goals: LLM inference separates intentional decoding stochasticity from systems nondeterminism and distinguishes bitwise, distributional, and semantic stability.These goals respectively concern identical bits, stable output distributions, and preservation of safety, constraints, and coarse meaning under sampling.
  • System-level nondeterminism: Even T=0 inference can vary because floating-point non-associativity, dynamic batching, scheduling, kernel selection, and hardware conditions alter computation.Concurrent requests are not part of a user’s prompt but can influence results through serving-system behavior.
  • Algorithmic stochasticity: At non-zero temperature, sampling deliberately produces varied outputs, while temperature approaching zero approaches greedy selection.Top-k and top-p restrict the sampling space while retaining variability.
  • Algorithmic stochasticity: Self-consistency and Tree-of-Thought use multiple stochastic reasoning trajectories to improve mathematical and complex problem solving.Self-consistency aggregates sampled chain-of-thought solutions; Tree-of-Thought explores branches and selects promising ones.
  • Costs and trade-offs: Greedy decoding can select one brittle continuation despite alternative trajectories, making failures ambiguous between inadequate competence and an unsuitable decoding policy.Strict determinism also requires synchronization and sacrificed optimizations, producing throughput penalties and engineering overhead.

2 Letś Stress-Test “Deterministic Inference” in Practice

The paper stress-tests deterministic inference as an evaluation and deployment norm. It argues that single-path decoding can hide generalization failures, suppress exploration-dependent abilities, collapse reasoning diversity, and create misleading safety confidence.

  • Framing: Deterministic inference is treated as an engineering objective imposed on a stochastic system rather than a natural LLM primitive.The paper contrasts bitwise reproducibility with the underlying distribution pθ(y | x).
  • Evaluation: Single-score, single-output evaluation can encourage benchmark memorization and obscure brittle generalization under paraphrastic or distributional variation.The paper revisits GLUE as a cautionary example of saturation and spurious progress.
  • Emergent abilities: Greedy decoding can suppress emergent abilities that depend on sampling multiple trajectories, including in-context learning and self-consistency gains.The paper reports that exploration can reveal capabilities that a forced single path fails to expose.
  • Reasoning: Strict greedy decoding collapses multiple valid reasoning paths into one trace, hiding uncertainty and making reasoning failures harder to diagnose.Multi-sample decoding instead exposes alternative strategies and near-miss failures.
  • Safety: Deterministic safety evaluation can create an illusion of robustness because low-probability dangerous behaviors emerge under stochasticity or paraphrased attack prompts.The paper therefore distinguishes deterministic diagnostic modes from determinism as a deployment norm.

3 Deterministic Inference Encourages Benchmark Memorization

The paper argues that single-output deterministic evaluation repeats GLUE’s benchmark-memorization problem by collapsing generative distributions into canonical answers. Its robustness protocol finds that aggregating stochastic trajectories exposes semantic competence and failure patterns that greedy decoding hides.

  • Deterministic inference reduces evaluation to one canonical output and scalar score, measuring surface-form reproduction rather than distributional semantic stability.
  • GLUE’s single-label protocol omitted prediction distributions, uncertainty, and robustness, enabling benchmark scores to mask shortcut learning and brittle heuristics.
  • LLMs generate conditional distributions over text, but greedy decoding collapses them into a single completion whose token-level choices determine the reported label.
  • The proposed GLUE-style protocol evaluates MNLI, QQP, QNLI, and SST-2 on paraphrased, perturbed, and adversarial variants with deterministic and stochastic decoding.
  • Across tasks and models, stochastic majority-vote evaluation generally improves robustness, while greedy decoding collapses the distribution into brittle behavior.
  • Multi-trajectory aggregation reveals latent diversity, uncertainty, and failure tails that single deterministic scores can conceal, producing a more faithful robustness picture.

4 Deterministic Decoding Suppresses Exploration–Driven Abilities

The paper treats emergent abilities as properties of the model-plus-decoder system rather than the parameters alone. Greedy decoding collapses trajectory exploration and can make available abilities appear absent, with exploratory policies yielding substantially higher task performance.

  • Emergent abilities are framed as properties of the base model combined with the decoding policy that probes its trajectory space.
  • A success set contains trajectories that realize a desired behavior, while each decoding policy induces its own distribution over trajectories.
  • Greedy decoding at T=0 collapses inference onto one maximizing trajectory, whereas stochastic decoding explores a richer subset of the model’s behavior space.
  • Deterministic decoding can hide an ability when the greedy trajectory fails but many nearby high-probability trajectories succeed.
  • The experiments manipulate decoding while holding model weights and prompts fixed across few-shot classification and style- or constraint-satisfying generation.
  • +10–30 absolute points of accuracy or constraint satisfaction can separate greedy and exploratory policies across benchmark tasks.

qICL

qICL represents input-wise success probability under decoding randomness, distinguishing deterministic outputs from hidden success mass available through exploration.

  • Deterministic decoding makes the predicted label constant across almost all random seeds, producing a degenerate success indicator.
  • qICL captures an input-wise probability of correct classification under exploratory decoding.
  • Exploratory decoding induces a trajectory distribution in which qICL measures how much hidden success mass is available.
  • The exploration-gain analysis compares performance recoverable under best-of-k self-consistency with performance under greedy decoding.
  • Best-of-k draws k independent stochastic completions, maps them to labels, and returns the majority label.

AccICL

The AccICL formulation connects exploration to classification accuracy by evaluating whether best-of-k decoding produces the correct label.

  • The exploration gain is defined by comparing best-of-k classification outcomes with the standard greedy decoder.
  • qICL under a stochastic policy denotes the probability that one sample yields the correct label for an input.
  • Under best-of-k majority voting, success depends on at least half of the k sampled labels being correct.

(qICL

The per-example best-of-k success probability is based on obtaining at least half correct among k stochastic draws, then averaging across inputs.

  • Best-of-k success is the probability that at least half of k draws are correct, averaged over inputs to estimate exploration gain.

qICL

Stochastic exploration reveals in-context learning ability that greedy decoding often suppresses. Across models and tasks, moderate best-of-k sampling exposes hidden correct-trajectory mass, with gains shaped by model size, task difficulty, and exploration settings.

  • Hidden majority: Greedy decoding can select an inferior local mode even when the correct label is the dominant mode under stochastic sampling.Majority-vote decoding exploits this structured distribution, whereas deterministic decoding discards it.
  • Cross-model pattern: Across open-model ICL benchmarks, greedy decoding substantially underestimates in-context capability revealed by modest stochastic exploration.The effect is observed across LLaMA, Gemma, Mistral, Mixtral, Vicuna, and Phi-2 families.
  • Exploration budget: Increasing the sampling budget from k=1 to k=4 and k=16 typically yields +10–20 absolute points, while best-of-64 adds only a further +2–5 points.This pattern indicates diminishing returns after moderate exploration budgets.
  • Magnitude of recovery: Best-of-16 commonly produces 8–22 absolute-point gains over greedy decoding, with some cells exceeding 22 percentage points.These gains are common across both BESSTIE tasks, and smaller or mid-size models often show larger relative improvements.
  • Interpretation: The results frame emergence as a property of the model–decoder pair rather than solely of the parameter vector.A substantial fraction of few-shot competence resides in trajectories that deterministic decoding never visits.
  • Exploration geometry: Exploration landscapes form task- and model-dependent ridges: mid-size models gain 10–20 pp near T ≈0.7 and k ∈[8, 32], while strong models show flatter surfaces.Sarcasm often retains taller ridges than sentiment, and overly conservative or overly hot settings reduce gains.

4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search

The section frames instruction-controllable summarization as multi-objective search over trajectories, where decoding policies explore different regions of the model’s conditional distribution. It defines operational semantic and constraint objectives, success sets, and exploration-based comparisons between decoding policies.

  • 4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search: InstruSum pairs news articles with natural-language requirements specifying content, length, style, and format.The benchmark extends controllable summarization toward richer, free-form instruction-following evaluation.
  • 4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search: Each candidate summary is evaluated jointly for semantic adequacy and satisfaction of length, inclusion, avoidance, and style/format constraints.The resulting objective vector contains five components, each scored in [0, 1].
  • 4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search: A decoding policy induces a trajectory distribution, so different policies explore different regions of the same underlying model distribution.This makes decoding a search strategy over a semantic–constraint landscape rather than merely a fixed generation recipe.
  • 4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search: Full instruction-following success requires exceeding thresholds on semantic adequacy and all four constraint dimensions.The success set is defined over trajectories that satisfy the article-summary objective and every instruction axis.
  • 4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search: Changing only the decoding policy can unlock instruction-following behavior already present in the model distribution.The style exploration gain compares policy-level success while holding model parameters fixed.
  • 4.2 InstruSum: Style–Constrained Generation as Multi–Objective Search: Greedy decoding is a degenerate search that commits to one point on the semantic–constraint surface, making failures ambiguous between limited competence and poor policy choice.A deterministic output may miss alternative trajectories with higher joint instruction satisfaction.

4.3 Results: Stochastic Search Unlocks Latent Instruction Following

Multi-sample search consistently improves instruction following, with most gains arriving by roughly eight samples and larger, newer models showing more exploitable probability mass. The gains primarily improve constraint satisfaction and expose better semantic–constraint trade-offs, although the amount of headroom is architecture-dependent.

  • 4.3 Results: Stochastic Search Unlocks Latent Instruction Following: Style exploration gains rise sharply from k=2 to k=8 and mostly saturate by k⋆≈8.The pattern holds across models, making small search budgets often sufficient for most gains.
  • 4.3.1 How Much Exploration is Enough, and Where Do Gains Come From?: Budgets k∈{4, 8} often capture most style and constraint improvements, with diminishing returns beyond that range.The attainable improvement remains strongly dependent on the model’s distribution of successful trajectories.
  • 4.3.1 How Much Exploration is Enough, and Where Do Gains Come From?: ICL exploration gains and style/constraint gains correlate positively at ρ≈0.80, but outliers show that explorability remains architecture-dependent.Mixtral–8×22B is strong on both axes, whereas Phi–2 is ICL-strong but style-weak.
  • 4.3.1 How Much Exploration is Enough, and Where Do Gains Come From?: At k=8, length and style scores typically improve by ≈0.07–0.13, whereas semantic adequacy improves more modestly by ≈0.02–0.03.Inclusion generally improves, while avoidance changes little because it is already high under greedy decoding.
  • 4.3.1 How Much Exploration is Enough, and Where Do Gains Come From?: At k=8, full success rises from 0.48 →0.64 for LLaMA–3 and 0.49 →0.68 for Mixtral–8×22B, but only from 0.30 →0.35 for Phi–2.These differences reflect model-specific headroom in the probability mass of successful trajectories.
  • 4.3.2 Semantic–Constraint Density Landscapes on InstruSum: Newer and larger models place more density near jointly high semantic adequacy and constraint satisfaction than older or smaller models.For LLaMA–3, Gemma–2, and Mixtral models, peak density lies around ssem(τ) ∈[0.65, 0.85] and cjoint(τ) ∈[0.35, 0.60].
  • 4.3.2 Semantic–Constraint Density Landscapes on InstruSum: When the high-semantic, high-constraint region has substantial mass, small-budget sampling moves outputs off an under-constrained ridge; thin regions yield smaller gains.This geometric pattern explains why search benefits LLaMA–3 and Mixtral–8×22B more than Vicuna–7B and Phi–2.

5 Deterministic inference collapses diverse reasoning paths into a single brittle trace

The section argues that greedy decoding collapses a distribution of reasoning paths into one brittle trace, obscuring alternative correct strategies. It studies this effect by sampling chains of thought, representing them as reasoning graphs, and comparing deterministic with multi-sample decoding.

  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: Greedy decoding collapses the model’s distribution over reasoning chains into a single potentially brittle trajectory.The section evaluates not only final answers but also the diversity and support of alternative solution paths.
  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: The paper claims that deterministic inference can misdiagnose reasoning ability when correct multi-step strategies exist but are never expressed by the greedy policy.These cases are described as collapsed failures rather than definitive evidence of absent competence.
  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: The reasoning study covers GSM8K, SVAMP, and StrategyQA, which provide arithmetic, perturbed arithmetic, and multi-hop commonsense reasoning tasks.The benchmarks include settings where multiple qualitatively different chains can reach the same answer.
  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: The experiments use eight instruction-tuned open-weight models and compare greedy decoding with temperature-0.7 nucleus sampling.Multi-sample decoding draws k∈{8, 16, 32} chains per instance, while greedy decoding returns one chain.
  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: The method represents sampled chains as paths through a tree- or DAG-like reasoning graph whose leaves contain complete rationales and final answers.Internal nodes encode partial reasoning states, allowing alternative branches to be compared.
  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: Sampled chains are segmented into discrete reasoning steps, producing 4–10 steps for GSM8K and SVAMP and 3–7 steps for StrategyQA.Each step receives a dense representation for constructing reasoning prefixes and graphs.
  • 5 Deterministic inference collapses diverse reasoning paths into a single brittle trace: Prefix states are merged when their cosine similarity exceeds a threshold δ∈[0.85, 0.90], allowing minor lexical variation while preserving high-level plans.This equivalence rule determines when sampled prefixes reuse an existing graph node.

2. Process each sampled chain τ (j)

The graph construction extends each sampled reasoning path from a distinguished empty-prefix root, reusing an existing node for an equivalent prefix or creating a new node otherwise.

  • 2. Process each sampled chain τ (j): Each sampled reasoning chain is processed from the distinguished root node representing an empty prefix.The path is extended step by step through the graph.
  • 2. Process each sampled chain τ (j): When a stored prefix is equivalent to the current prefix, the construction reuses the existing graph node.This merges reasoning paths that share the same represented state.
  • 2. Process each sampled chain τ (j): When no equivalent node exists, the method creates a new node representing the current prefix.The new state is then incorporated into the graph for subsequent path construction.
  • 2. Process each sampled chain τ (j): In both cases, the method adds a directed edge from the node encoding the preceding prefix to the current reasoning state.Edges preserve the sampled chain’s progression through the graph.

3. Once the last step s(j)

The paper shows that reasoning diversity is common and that greedy decoding collapses this diversity into a single, potentially unrepresentative path. Multi-sample exploration exposes alternative correct strategies, recovers collapsed failures, and reveals brittleness that deterministic evaluation hides.

  • Greedy decoding selects one reasoning path and discards alternative branches, even when those branches carry substantial probability mass.
  • Path diversity is the norm across GSM8K and SVAMP, where most models produce substantially more than one reasoning path for most instances.
  • On GSM8K, SVAMP, and StrategyQA, greedy rationales can be low-support or idiosyncratic compared with alternative correct or more careful sampled paths.
  • Collapsed failures occur when greedy decoding is wrong although multi-sample reasoning contains a correct path, with this pattern especially common on SVAMP and StrategyQA.
  • Path diversity and collapsed-failure rates tend to increase with model strength, while greedy paths represent a shrinking fraction of sampled behavior.
  • Multi-sample decoding yields 5–15 absolute points of CoT accuracy beyond greedy across the benchmark panel, with larger gains for more diverse models.
  • Deterministic decoding can hide alternative high-quality outputs, underrepresent instruction-following capacity, and collapse multi-path reasoning into a brittle trace.

6 Deterministic safety evaluation creates an illusion of robustness

Deterministic safety evaluation can make models appear robust by inspecting only the greedy completion while ignoring harmful probability mass in the stochastic tail. Across analytic and model-level analyses, concealed risk grows with sampling budget and often with model capability.

  • Hidden tail mass: Greedy evaluation is blind to harmful probabilities that do not affect the argmax continuation.A safe greedy completion can yield zero measured risk even when stochastic sampling exposes harmful outputs.
  • Model-level risk: The deterministic illusion index increases with model capability and sampling budget, indicating that stronger models can hide larger stochastic safety risks.The index approaches 1 when k-sample risk is substantial but greedy evaluation indicates near-perfect safety.
  • Hidden tail mass: A harmful tail probability of ε=0.01 produces approximately 0.15 risk across k=16 samples, despite a safe greedy completion.The bound 1−(1−ε)^k implies harmful output in roughly one out of six stochastic runs.
  • Cross-model patterns: Across twelve decoder-level risk surfaces, most prompts form a low-risk plateau while a smaller high-risk tail rises sharply under k=8 and k=16 sampling.The low-risk plateau typically covers the lowest 40–75% of prompts, while roughly the top 10–30% form a sharp stochastic-risk tail.
  • Cross-model patterns: Scaling within model families primarily enlarges the safe plateau and lowers greedy risk while leaving a narrow band of prompts with near-certain stochastic risk.Compact models such as Phi–2 instead show broader regions where both greedy and multi-sample risks are high.
  • Overall implication: Risk-versus-budget curves, analytic surfaces, and composition analyses consistently show that greedy evaluation underestimates risk more as capability and distributional exploration increase.For nonzero harmful mass qθ(x), multi-sample risk grows as 1−(1−qθ(x))^k while greedy evaluation remains blind to it.

7 Discussion and Limitations

The discussion argues that bitwise determinism collapses LLMs’ stochastic structure, obscuring capability, brittleness, reasoning diversity, and safety risk. It recommends decoder-aware, distributional evaluation while bounding conclusions by model, task, decoding, metric, contamination, and infrastructure coverage.

  • Core implications: Emergent behaviors depend partly on the decoder: low-entropy argmax policies can hide behaviors supported by pθ(· | x), whereas higher-entropy and multi-sample policies reveal them.This reframes emergence as a property of model–decoder pairs rather than weights alone.
  • Core implications: Bitwise determinism can be distributionally misleading: stable greedy outputs may conceal conflicting or unsafe trajectories, while stochastic sampling can remain reproducible in aggregate.The relevant quantities include success rates, variance estimates, robustness to perturbations, and extreme-event probabilities rather than exact strings alone.
  • Evaluation and deployment: The paper recommends reporting multi-sample curves and stability metrics, testing paraphrased and out-of-distribution variants, specifying decoding regimes, and publishing per-input samples.These practices expose distributional behavior and allow model comparisons under explicitly stated inference policies.
  • Core implications: Single-trajectory evaluation underestimates both capability and fragility, missing valid reasoning paths and sharp performance degradation under paraphrases and small input shifts.The paper identifies both high-success regions revealed by stochastic decoding and brittle canonical-prompt islands.
  • Evaluation and deployment: Safety evaluation can appear extremely safe under greedy decoding while retaining substantial harmful probability mass under modest multi-sample decoding.The associated risk is low-probability, high-impact behavior that deterministic evaluation fails to surface.
  • Limitations: The conclusions are bounded by the examined models and tasks, limited decoding families, possible data contamination, discrete and incomplete metrics, and temporal infrastructure drift.The authors state that quantitative patterns may differ for other architectures, search procedures, held-out distributions, semantic criteria, and future systems.
  • Limitations: Extending this framework to multimodal, tool-using, and agentic systems remains an open direction because those systems add stochastic perception, environments, tools, and users.The paper presents this extension as a natural next step rather than an evaluated result.

8 Conclusion

The paper argues that deterministic evaluation obscures important properties of LLM behavior, including emergent abilities, reasoning diversity, and safety tails. It recommends treating distributional variability as a first-class object while retaining deterministic modes for targeted diagnostics.

  • Deterministic evaluation can hide emergent abilities, collapse multi-path reasoning, and create misleading impressions of robustness.
  • Distributional metrics should report success probabilities, exploration gains, and risk gaps instead of relying on single-run scores.
  • Reasoning evaluation should analyze ensembles of trajectories for diversity, stability, and failure modes rather than treating one trace as canonical.
  • Safety audits should explore low-probability regions through paraphrased attacks, mixed decoders, and small sampling budgets.
  • Engineering should prioritize distributional reproducibility and semantic stability, while using deterministic modes for debugging, ablation, and regression testing.
  • The study is scoped to a subset of models, decoding policies, and stress tests, leaving broader samplers, adaptive evaluation, and governance questions open.

blog.connectionism.ai/defeating-nondeterminism-in-llm-inference. Blog post.

This section contains bibliographic entries and source metadata rather than substantive argument or findings.

  • The section lists cited research on language models, reasoning, evaluation, and reproducibility.
  • Other entries identify research on reasoning, emergent abilities, summarization, and benchmark construction.
  • Several entries identify work on deterministic implementations and reproducibility in machine learning systems.
Loading 2601.07239v1…