PaperlayerView exact revision on arXiv

Source-linked AI summary

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen, Lele Wang, Giuseppe Carenini, Peter West

arXiv:2606.06622v3cs.CL

TL;DR

LLMs are increasingly used to simulate stochastic systems, but it remains unclear whether they can generate samples calibrated to true underlying distributions rather than merely varied outputs. UnpredictaBench addresses this gap with 448 distribution-sampling problems and KS@N evaluation, finding broad capability differences and persistent difficulty even for simple sampling.

  • Problem

    LLMs may collapse to plausible outputs instead of capturing the unpredictability and calibrated distributions required for simulation.

  • Method

    UnpredictaBench evaluates independent samples from 448 statistical, stochastic-program, and natural-language tasks against black-box ground-truth distributions using KS@N.

  • Results

    No current model comes close to solving the benchmark, with the strongest model achieving only 32.64% at KS@100 and reasoning, temperature, and list prompting helping modestly.

  • Takeaways & Limitations

    Even simple distributional simulation remains challenging, making UnpredictaBench a first step toward assessing LLMs as stand-ins for complex systems.

  • Takeaways & Limitations

    The benchmark is English-only, 89% GPT-5.4-generated, and Python-only, limiting generalizability across languages, authorship settings, and programming paradigms.

Abstract

from arXiv · show

We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems. Project website and resources are available at https://unpredictabenchmark.github.io/.

1 Introduction

UNPREDICTABENCH addresses whether LLMs can generate samples calibrated to underlying stochastic processes, a capability important for simulation but insufficiently evaluated. It tests this problem with 448 distributions and KS@N, finding broad performance gaps and persistent difficulty.

  • Motivation: LLMs may reason about randomness without producing samples calibrated to the underlying process, limiting their use as stand-ins for stochastic systems.Relevant applications include economic and epidemiological simulation, where distributional mismatch can yield biased estimates, overconfident predictions, and misleading conclusions.
  • Motivation: Prior studies report stochastic-generation failures in behavioral simulation, real-world distribution modeling, mixed-strategy games, and simple random tasks.
  • Benchmark: UNPREDICTABENCH contains 448 known distributions, stochastic code problems, and word problems spanning unimodal, multimodal, real-world, and shuffling tasks.Models generate independent samples evaluated with KS@N against black-box ground-truth samples using a Kolmogorov-Smirnov test.
  • Findings: No model surpasses 40% at KS@100, with most models scoring between 0% and 20%, showing that plausible samples of size 100 remain difficult.Reasoning can help, but it does not solve the problem; code-inference and shuffling tasks are especially challenging.
  • Conclusion: The benchmark provides a first step toward evaluating and improving LLMs’ ability to capture complex randomness.Its scope is deliberately simple relative to complex distributions such as human behavior.

2 Related Work

Prior work separates probabilistic reasoning from faithful stochastic generation and identifies training, prompting, and calibration factors that affect model randomness. These findings motivate repeated-output evaluation against target distributions rather than diversity alone.

  • Probabilistic reasoning and randomness generation: LLMs can perform non-trivial probabilistic reasoning, but prior studies find that reasoning about distributions does not reliably translate into accurate sampling.Reported failures include identifying probabilistic structure, encoding real-world observational distributions, and inferring latent distributions.
  • Alignment, uncertainty, and behavioral factors: Post-training and fine-tuning can reduce output diversity, while temperature-shifted self-samples and prompt structure can partially alter stochastic behavior.Fine-tuning may improve alignment with human opinion distributions, but persistent diversity reduction remains.
  • Alignment, uncertainty, and behavioral factors: These findings motivate evaluating whether repeated model outputs are calibrated to a target distribution, not merely whether they are diverse.

3 UNPREDICTABENCH

UNPREDICTABENCH combines generated and human-curated stochastic tasks across textual, code, multimodal, shuffling, and real-world settings. It evaluates independent model samples against reference distributions using a repeated-generation protocol.

  • Benchmark Construction and Task Types: The benchmark evaluates whether language models generate outputs consistent with target probability distributions rather than merely recognize or describe them.
  • Benchmark Construction and Task Types: Forty well-known distributions were selected from 176 distributions crawled from Wikipedia, with GPT-5.4 generating templated tasks across varied parameter regimes.
  • Textual Tasks: Text tasks either name the distribution and parameters or require inferring an unstated stochastic process from a natural-language scenario.
  • Code Tasks: Code tasks require predicting outputs from stochastic Python programs, either by direct sampling or by reasoning through transformations of samples.
  • Multimodal and Shuffling Tasks: Multimodal tasks combine component distributions, while shuffling tasks test uniform permutation generation and encode outputs with Lehmer codes.
  • Real-World Scenario Tasks: Real-world scenarios cover six practical sources of nondeterminism, including concurrency, hashing, stochastic networks, and asynchronous distributed communication.
  • Dataset Statistics: The dataset contains 448 English instances, including 398 GPT-5.4-authored and 50 human-authored tasks across explicit, implicit, multimodal, and real-world categories.
  • Evaluation Strategy: Evaluation compares N=100 independent model samples with M=10,000 ground-truth samples to assess reproduction of a target distribution.

4 Experiments and Results

Experiments across diverse open and proprietary models show that distributional fidelity varies widely and deteriorates as sample size increases. Instruction tuning and reasoning provide limited or model-specific benefits, while qualitative analyses reveal failures rooted in incorrect or collapsed internal distributions.

  • Overall Model Performance: 0%–20% covers most models’ KS@100 scores, while no model exceeds 40%, making 100-sample distributional matching broadly difficult.The benchmark evaluates open-weight and proprietary systems under standardized generation and parsing procedures.
  • Overall Model Performance: 32.64% KS@100 is achieved by Nemotron-3 Super 120B, nearly doubling the third-ranked model and leading the evaluated systems.GPT-4o scores 23.90%, DeepSeek V3.2 21.73%, and smaller Nemotron-3 Nano 30B 20.83%.
  • Metric and Sample-Size Analysis: KS@100 aligns with JSD and WDZ across models and categories, supporting KS as a measure of genuine distributional alignment.JSD captures global overlap, whereas WDZ emphasizes tail behavior.
  • Metric and Sample-Size Analysis: KS@1 is perfect for all models, but performance generally declines with N, with the steepest drops between KS@20 and KS@100.Claude Sonnet 4.6 falls from 97.76% at KS@2 to 4.70% at KS@100, showing that single-sample plausibility poorly predicts distributional fidelity.
  • Category-Level Analysis: Shuffling and Code are the hardest categories, with no model above 40%, whereas RealWorld reaches 59.09% for Llama-3.2-1B.Nemotron-3 Super 120B drops to 3.33% on RealWorld despite leading overall, indicating limited transfer across stochastic settings.
  • Instruction Tuning and Reasoning: Instruction tuning slightly improves KS@100 but often reduces diversity, while reasoning improves final outputs inconsistently across models and extraction conditions.Qwen3.5-35B-a3b gains +35.18 at KS@20 when numbers are extracted from reasoning tokens, despite nearly zero final-output improvement.
  • Qualitative Analysis: Llama 3.2’s logits and samples remain closely aligned across Beta and Poisson-Binomial tasks, indicating that collapsed diversity reflects stable internal beliefs rather than decoding alone.Figure 1 also shows incorrect mass allocation on Skellam and concentration at small values for OLMo-3-7B on Poisson.
  • Alignment with Creativity Benchmarks: KS@100 correlates positively with creative-generation utility, while lexical diversity correlates negatively and does not substitute for distributional fidelity.CREATE Utility correlations are r = 0.75 at p=0.7 and r = 0.78∗ at p=0.9; NoveltyBench Utility10 is r = 0.65, while Distinct10 is r = −0.21.

5 Ablations

The ablations show that temperature, generation budget, and list prompting affect distributional fidelity in distinct, model-dependent ways. Higher diversity can improve short-horizon coverage, but stricter evaluation exposes persistent biases and some interventions trade global coverage against local coherence.

  • The Effect of Temperature: 39.57% KS@100 is Nemotron-3 Super 120B’s peak average at T=1.2, while performance drops to 5.23% at T=0.1.Near-greedy decoding is particularly harmful for stochastic tasks.
  • The Effect of Temperature: Higher temperatures generally improve KS@100, but weaker models may amplify tail deviations instead of improving distributional coverage.OLMo-3 7B’s WDZ remains persistently high across temperatures and increases slightly with temperature.
  • The Effect of Sampling Budget: KS@100 increases as the generation budget grows from 100 to 1000, whereas KS@500 and KS@1000 reveal deeper biases under stricter scrutiny.Models can appear well-calibrated over small samples while their biases become statistically detectable across larger generated sets.
  • The Effect of Asking for a List of Samples Instead of One: Requesting 10 outputs per call improves KS@100 by +17.12 for Nemotron-3 Super 120B and +14.32 for Ministral-3 3B.The gains slightly hurt KS@20, suggesting improved global coverage at the cost of local coherence.
  • The Effect of Asking for a List of Samples Instead of One: List prompting is model-sensitive: 35 outputs cause a −3.81 change at KS@100 for Llama-3.2-1B, while capable models can improve without additional training.The benefit does not hold uniformly across evaluation thresholds or models.

6 Conclusion

UNPREDICTABENCH evaluates whether LLMs can generate samples consistent with true statistical distributions. Across its test instances, current models remain far from solving the benchmark, while several interventions help only modestly.

  • UNPREDICTABENCH evaluates LLMs’ ability to generate samples consistent with true underlying statistical distributions.
  • Across 448 test instances spanning 40 distributions and four task categories, no current model comes close to solving the benchmark.
  • 32.64% at KS@100 is the strongest model’s result, while instruction tuning worsens collapse and reasoning, temperature, and list prompting help modestly.
  • Current models fail through either weak internal representations of target distributions or collapse toward a narrow set of outputs.

Limitations and Broader Impact

UNPREDICTABENCH targets faithful distributional generation for simulation, scientific modeling, and decision-making, while its benchmark design and scope impose important limits on interpretation.

  • Broader impact: Faithful distributional generation matters for simulations whose outputs must reflect true underlying distributions rather than collapse onto dominant modes.The benchmark is positioned as a concrete target for economic, epidemiological, and multi-agent simulations.
  • Limitations: All prompts are in English and 89% are GPT-5.4-generated, limiting generalizability to multilingual or human-authored settings.
  • Limitations: Code tasks are Python-only, so findings may not transfer to other programming languages or paradigms.
  • Limitations: Ground-truth distributions depend on constructed reference samples, and alternative task formulations could produce different targets.
  • Limitations: UNPREDICTABENCH is an evaluation benchmark rather than training data, and direct optimization risks overfitting to its tasks and metrics.Strong benchmark performance should not be interpreted as real-world deployment readiness.

D Extended Model Results

Extended results show that the main category-level patterns persist across the full evaluated model set, with RealWorld tasks easiest and Code and Text most demanding.

  • Per-category results: RealWorld tasks yield the highest individual scores, while Code and Text remain the most demanding across the reported metrics.Models with strong overall KS@100 generally also show lower JSD and WDZ values.
  • Per-category results: 59.09% on RealWorld is achieved by Llama-3.2-1B-instruct despite near-bottom performance elsewhere.
  • Per-category results: Qwen3.5 MoE variants underperform relative to their parameter counts across all categories.

E Per-Distribution KS@100 Breakdown

Performance varies sharply by target distribution and prompting condition: simple finite-support distributions are easiest, while heavy-tailed or multivariate targets and implicit prompts are harder overall.

  • Per-distribution results: 43.04% for Bernoulli, 34.78% for Categorical, and 16.52% for Discrete Uniform lead KS@100 across distributions.The passage attributes this pattern to small supports and likely pretraining familiarity.
  • Per-distribution results: 1.74% for Fréchet and Dirichlet, 5.22% for Negative Binomial, and 6.09% for Negative Multinomial rank lowest at KS@100.These distributions involve long tails or correlated multivariate structure.
  • Prompting conditions: Explicit prompting generally outperforms implicit prompting, with Nemotron-3 Super 120B scoring 41.42% versus 26.42% at KS@100.The reported gap is 15 percentage points, although several models perform better implicitly.

G Unimodal vs. Multimodal Distribution Complexity

Multimodal mixture distributions produce model-dependent difficulty: stronger models can outperform on multimodal tasks, while weaker or deterministic models often collapse on them.

  • Model-dependent complexity: 42.50% versus 33.65% at KS@100: Nemotron-3 Super 120B performs better on multimodal than unimodal tasks.GPT-4o shows the same pattern, scoring 38.75% versus 22.96%.
  • Model-dependent complexity: Mixture distributions may be easier for diverse models because their broader support can help imperfect samples pass the KS test.
  • Model-dependent complexity: 0.00% versus 6.31%: Claude Sonnet 4.6 performs worse on multimodal than unimodal tasks.Mercury-2, Phi-3.5 Mini, and both Qwen3.5 MoE variants show the same reversal toward unimodal tasks.
  • Model-dependent complexity: 7.50% versus 18.87%: GPT-5.4 performs markedly worse on multimodal than unimodal tasks, consistent with collapse toward a single point.This failure is especially damaging when the target has two well-separated modes.

H Effect of Distributional Spread

Distributional spread affects models differently: strong models perform similarly on concentrated and spread-out targets, while many weaker models struggle with broad support.

  • Model-dependent effects: 36.36% versus 34.50% at KS100: Nemotron-3 Super 120B performs comparably on concentrated and spread-out distributions.GPT-4o likewise scores 27.27% versus 25.00%.
  • Model-dependent effects: 11.62% versus 0.50%: Grok-4.1-fast performs substantially worse on spread-out than concentrated distributions.Claude Sonnet 4.6, Phi-3.5 Mini, and both Qwen3.5 MoE variants show similar degradation on spread-out targets.
  • Mechanism: Deterministic models may anchor near a distribution’s mode, which works for concentrated targets but fails when support is broad and tail mass is substantial.
  • Model-dependent effects: 21.00% versus 12.12%: Ministral-3B instruct performs better on spread-out than concentrated tasks.A small number of models show this reverse pattern.

I Error Analysis

Repeated-run and ground-truth sensitivity analyses indicate that benchmark results are stable, while model- and category-specific patterns remain consistent across evaluation noise.

  • Evaluation stability: Narrow error bars across three runs confirm stable and reproducible KS@100, JSD, and WDZ results across models and task categories.The reported variance from ground-truth resampling is negligible relative to observed model and category differences.
  • Evaluation stability: The stability analysis supports using a single evaluation run for the paper’s main results.
  • Stable model patterns: Around 59%: Llama-3.2-1B’s RealWorld KS@100 remains high and stable, while its Text JSD is around 0.52 and WDZ around 29.82.
  • Stable model patterns: 40.34% KS@100 and 0.48 JSD: Nemotron-3 Super 120B shows a stable tension between KS-based success and distributional distance on Text.
  • Ground-truth sensitivity: Three independent ground-truth sets of 1,000 samples produce small standard deviations for KS@100, JSD, and WDZ.This confirms robustness to the particular ground-truth sample set.
  • Reproducibility: The evaluation framework is designed to remain agnostic to the specific ground-truth instantiation.The authors release fixed ground-truth samples and generation code for replication and allow alternative sample sets.
  • Generation validity: Invalid generations are discarded and the same prompt is retried up to five times, retaining only valid samples for metric computation.The retry procedure is intended to measure reliable well-formed output without biasing distributional metrics.

L Output Diversity Analysis

Shuffling-task outputs fall below the attainable diversity ceiling, with lower-diversity models showing greater variability and intermittent mode collapse.

  • Observed diversity: All models remain below the attainable ceiling of approximately 39.8 unique items on the shuffling task.The figure measures unique items produced per run across approximately 1,000 runs at temperature 1.0.
  • Observed diversity: 33.5–35.5 unique items: model means cluster within a relatively narrow band despite falling short of full uniform coverage.
  • Variability and collapse: GPT-5.4 and Claude-sonnet-4.6 reach only 25–26 unique items in their worst runs.
  • Variability and collapse: Lower-diversity models show higher variance and longer left tails, indicating intermittent rather than uniformly reduced diversity.

M Additional Qualitative Analysis

Figures 8–11 compare model-generated empirical distributions with ground truth across representative task types, while the benchmark constructs text, code, and multimodal sampling tasks under explicit or implicit concentrated/spread-out settings. Across these examples, many models collapse to narrow or single-point outputs instead of reproducing the target distribution's shape.

  • Qualitative Results: Most models produce narrow spikes or single-point concentrations rather than matching ground-truth distribution shapes.This pattern appears across the four representative figures and is especially pronounced for several evaluated models.
  • Task Design: Benchmark generation covers text, code, and multimodal tasks, varying whether distributions are explicit or implicit and concentrated or spread out.Text and code tasks use four prompt variants, while multimodal tasks use concentrated and spread-out parameter regimes.
  • Evaluation: The benchmark evaluates independent samples from 448 distributions, stochastic code problems, and word problems using the KS@N metric.Answer extraction uses task-specific LLM-based extractors for standard text and code, list, shuffling, and real-world tasks.
  • Qualitative Results: The Fréchet examples expose failures to capture heavy right tails, with samples concentrated near the lower support bound.The figure overlays 100 model samples with the ground-truth density.
  • Qualitative Results: The truncated-normal examples contrast models that approximate a broad bell-shaped target with models that collapse despite wide support.The task uses a textual explicit spread-out setting.
  • Task Design: The Binomial and Poisson implicit tasks require models to infer distributional structure from text or code without an explicit distribution name.The Binomial task uses concentrated discrete support, while the Poisson task uses an integer-valued, right-skewed target.
Loading 2606.06622v3…