UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Failure modes

• Models fail through narrow-output collapse or broader miscalibration, and reasoning interventions help only partially; code-inference and shuffling tasks are especially challenging.Several otherwise strong performers collapse to 0% on shuffling tasks, while Figure 1 illustrates both reasonable-mode collapse and failure to reproduce target structure.Source: Figure 1 · p. 2, §1 · p. 2

Benchmark design

• UNPREDICTABENCH contains 448 distributions and stochastic problems spanning unimodal, multimodal, code, real-world, textual, and shuffling scenarios, with models generating independent samples.The benchmark covers 40 target distributions and evaluates distributional randomness beyond simple numeric prompting.Source: §1 · pp. 2–3

Evaluation metric

• KS@N measures distributional accuracy by testing whether model samples are rejected against black-box ground-truth samples using a fixed-threshold Kolmogorov-Smirnov test, with larger N increasing difficulty.The metric requires only samples from the ground-truth distribution and evaluates repeated generation rather than one-off correctness.Source: §1 · pp. 2–3

Empirical findings

• No model exceeds 40% at KS@100, while most score between 0% and 20%, showing that generating calibrated samples of size 100 remains difficult.Nemotron-3-super-120b-a12b ranks among the strongest, but GPT-5.4 averages 15.18%, Claude-sonnet-4.6 4.7%, and Qwen-3.5-2B 17.67% across tasks.Source: §1 · p. 2

Broader significance

• Distributional fidelity correlates strongly with NoveltyBench and CREATE [34] utility metrics, offering a statistically grounded alternative to LLM-as-a-judge evaluation [41].The benchmark therefore measures a substantive aspect of model quality rather than merely output diversity or one-off plausibility.Source: §1 · p. 2

Limitations and future work

• Because models struggle even with simple direct-output distributions, the benchmark cautions against expecting reliable simulation of more complex distributions such as human behavior.Increasing sample size provides a natural path to harder benchmark variants and future work on randomness and distributional generation.Source: §1 · pp. 2–3

Probabilistic reasoning and randomness generation

• LLMs can identify probabilistic structure but often fail to generate faithful samples, from real-world and latent distributions to uniform randomness and categorical targets [6] [38] [13] [4].These failures include deterministic, value-biased outputs and a cognition–behavior gap, even when models can state correct probabilistic or strategic reasoning.Source: §2 · p. 3
• UnpredictaBench differs from prior single-setting studies by unifying many distributions and tasks in a repeated-output benchmark that tests calibration to target distributions.The evaluation distinguishes calibrated sampling from merely producing diverse responses.Source: §2 · p. 3

Alignment, uncertainty, and behavioral factors

• Post-training and fine-tuning can reduce stochastic behavior: aligned models underperform base models on random generation and creativity, while cross-entropy training systematically lowers diversity [35] [17].Fine-tuning on temperature-shifted self-samples can partially recover diversity [37].Source: §2 · p. 3
• Prompt structure and poor confidence calibration also condition apparent stochastic behavior, with uncertainty organized by semantic similarity among candidate responses [2] [20].Fine-tuning can improve agreement with human opinion distributions but still reduce diversity [3].Source: §2 · p. 3

3.1 Benchmark Construction and Task Types

• UNPREDICTABENCH contains 448 reviewed instances generated from 40 selected distributions and 50 manually constructed stochastic tasks across seven task categories.The construction pipeline uses GPT-5.4 for automatically generated prompts, varies distribution hyperparameters, and removes two tasks during independent quality review.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
• The benchmark’s textual tasks require direct sampling from named distributions or inference of unstated stochastic processes described through real-world scenarios.Text Explicit tasks provide the distribution and parameters, whereas Text Implicit tasks omit the distribution name and require process inference.Source: §3.1 · p. 4
• Code tasks sample outputs from explicit NumPy distributions or implicit programs whose transformations require reasoning about an underlying stochastic process.Implicit programs use operations such as square roots, trigonometric functions, or summations applied to samples from another distribution.Source: §3.1 · pp. 4–5
• Multimodal, shuffling, and real-world tasks test mixture coverage, permutation-level randomness, and simulation of practical nondeterministic systems.Multimodal tasks combine component distributions; shuffling tasks encode permutations with Lehmer codes; real-world scenarios cover MCMC, decisions, concurrency, hashing, networks, and distributed systems.Source: §3.1 · p. 5

3.2 Statistics

• The benchmark includes 398 GPT-5.4-authored and 50 human-authored tasks, with balanced concentrated versus spread parameter settings and 80 multimodal tasks among the automatic instances.Its automatic-task distribution includes 159 Text Explicit, 79 Text Implicit, 80 Code Explicit, and 80 Code Implicit tasks; human-authored tasks include 30 Real-World and 20 Shuffling tasks.Source: §3.2 · p. 5, Table 1 · p. 6

3.3 Evaluation Strategy

• Evaluation compares N = 100 independent model samples against M = 10,000 samples from the ground-truth distribution.The model samples form set A from Dpred, while the reference samples form set B from Dgt.Source: §3.3 · p. 5

4.2 Overall Model Performance on UnpredictaBench

• KS@100 aligns broadly with JSD and WDZ, while performance generally declines with sample size; all models achieve perfect KS@1, making single-sample plausibility insufficient.The strongest model reaches 32.64% at KS@100, whereas Claude Sonnet 4.6 drops from 97.76% at KS@2 to 4.70% at KS@100.Source: §4.2 · pp. 7–8, Table 3 · p. 9
• Nemotron-3 Super 120B leads KS@100 at 32.64%, ahead of GPT-4o at 23.90% and DeepSeek V3.2 at 21.73%, while several large models score below 5%.Figure 3 shows a substantial gap across model families, with Qwen3.5 2B reaching 17.67% and Ministral-3 3B reaching 1.35%.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 The Effect of Instruction Tuning

• Instruction tuning modestly improves KS@100 but often reduces diversity, trading fewer out-of-support values for worse JSD or WDZ.These comparisons use the Text+Code subset and exclude Shuffling and RealWorld tasks.Source: §4.3 · p. 8, Table 4 · p. 10

4.4 The Effect of Reasoning

• Reasoning improves final-output performance overall, but extracted reasoning tokens vary sharply: Qwen3.5-35B-a3b gains +35.18 at KS@20 while Nemotron-3 Super 120B loses −33.17.Qwen3-32B gains +9.30 at KS@50 from reasoning-token extraction, whereas some models repeatedly revisit candidates rather than exploring the support.Source: §4.4 · pp. 9–10, Table 5 · p. 10

4.5 Qualitative Analysis

• Qualitative failures involve either narrow support or incorrect probability mass, and aligned logits show that limited diversity reflects stable model beliefs rather than decoding artifacts.Llama 3.2 1B’s instruct variant covers less of the Poisson-Binomial support than its base variant, while both miss the Beta distribution’s U-shape; instruction tuning suppresses unusual outputs.Source: §4.5 · p. 10, Figure 4 · p. 11

4.6 Alignment with Novelty and Creativity Benchmarks

• KS@100 correlates positively with utility—r = 0.75 for CREATE at p=0.7, r = 0.78∗ at p=0.9, and r = 0.65 for NoveltyBench—while Distinct10 correlates negatively at r = −0.21.Nemotron-3 Super 120B leads across benchmarks, whereas Llama-3.2-1B-instruct leads Distinct10 but ranks near the bottom on utility and KS@100.Source: §4.6 · p. 11

5.1 The Effect of Temperature

• Higher temperatures generally improve KS@100, with Nemotron-3 Super 120B peaking near T=1.2 at 39.57%, whereas weaker models can amplify tail deviations.Nemotron-3 falls to 5.23% at T=0.1; OLMo-3 7B’s WDZ remains high and rises slightly, while Ministral’s optimum is task-dependent.Source: §5.1 · pp. 11–12, Table 7 · p. 13

5.2 Effect of Sampling Budget

• Larger generation budgets improve short-horizon KS@100, but KS@500 and KS@1000 expose deeper biases; Ministral-3 3B leads, while scaling cannot rescue weaker models.The main benchmark’s N=100 protocol is therefore conservative: models passing at KS@100 may fail under stricter scrutiny.Source: §5.2 · p. 12, Table 8 · p. 13

5.3 The Effect of Asking for a List of Samples Instead of One

• List prompting improves KS@100 for capable models but is model-sensitive: 10 outputs add 17.12 and 14.32 for Nemotron-3 and Ministral, while Llama-3.2-1B declines.For Nemotron-3 and Ministral, gains at KS@100 slightly reduce KS@20 and partially reverse at 35 outputs; OLMo-3 benefits consistently, while Llama declines nearly everywhere.Source: §5.3 · pp. 13–14, Table 9 · p. 14

Main conclusion

• 32.64% at KS@100 is the strongest model’s score across 448 instances spanning 40 distributions and four task categories, leaving a large gap to the Random Machine ceiling.No current model comes close to solving the benchmark, and the gap remains unsolved.Source: §6 · p. 14

Failure modes

• Models fail either by lacking a meaningful representation of the target distribution or by understanding its rough shape but collapsing to a narrow set of outputs.Instruction tuning exacerbates output collapse.Source: §6 · p. 14

Interventions

• Reasoning, temperature, and list prompting improve performance modestly but fall far short of closing the distributional-sampling gap.These interventions do not provide an immediate solution to the benchmark’s core challenge.Source: §6 · p. 14

Benchmark significance

• UnpredictaBench aligns with utility metrics from creativity benchmarks while providing a statistically grounded alternative to LLM-as-a-judge evaluation.The cross-dataset analysis supports the benchmark’s broader evaluation value.Source: §6 · p. 14

Limitations and Broader Impact

• UNPREDICTABENCH supports simulation and scientific modeling by testing faithful distributional generation, distinguishing weak distributional understanding from insufficient output diversity.The benchmark offers KS@N as a statistically grounded target for calibration and can inform when models are suitable for simulation-style deployment.Source: Limitations and Broader Impact · p. 14
• The benchmark is limited by English prompts, predominantly GPT-5.4-generated phrasing, Python-only code tasks, reference-sample-dependent targets, and risks of overfitting without implying deployment readiness.It contains no personal or sensitive information and is released under CC BY 4.0, with code and ground-truth values on GitHub.Source: Limitations and Broader Impact · pp. 14–15

A UNPREDICTABENCH Distributions List

• UNPREDICTABENCH covers 40 probability distributions across 8 categories, with highlighted distributions used for the multimodal subtask.The complete distribution list is provided in Table 10.Source: A UNPREDICTABENCH Distributions List · p. 19, Table 10 · p. 19

B Real World Examples

• RealWorld tasks ask models for one plausible output from stochastic code or textual scenarios, including network routing, garbage collection, distributed replicas, and MCMC state transitions.The examples use binary or uniform categorical outcomes, while the MCMC distribution over A, B, and C is empirically estimated from repeated LLM executions.Source: B Real World Examples · pp. 19–21

C.1 Handling Sequence-Valued Tasks via Lehmer Codes

• Sequence-valued tasks are converted through the bijective Lehmer code, with normalized coordinates enabling scalar metrics and Z1 selected for its largest support.Under uniformly random permutations, normalized coordinates are asymptotically uniform on [0, 1], and evaluation applies metrics directly to Z1.Source: §C.1 · pp. 21–22

C.2 Debiased Wasserstein-1 Distance

• The evaluation metrics include debiased Wasserstein-1 distance, using 999 permutation partitions to estimate finite-sample bias and a z-score for systematic mismatch.Values of ZW1 near zero indicate chance-level indistinguishability, whereas larger values indicate distributional mismatch.Source: §C.2 · p. 22

C.3 Jensen–Shannon Divergence

• Jensen–Shannon divergence compares Gaussian-kernel density estimates on a shared 512-point padded grid, capturing symmetric, bounded density-shape mismatches even without overlapping supports.The densities are normalized into discrete distributions before computing JSD, which complements distance-based metrics.Source: §C.3 · p. 22

• Across all models, RealWorld tasks achieve the highest individual scores, while Code and Text remain the most demanding categories.Table 11 reports KS@100, JSD, and WDZ across all four categories.Source: D Extended Model Results · p. 22
• 59.09% KS@100 on RealWorld makes Llama-3.2-1B-instruct an outlier despite near-bottom performance elsewhere.Source: D Extended Model Results · p. 22
• Models with strong overall KS@100 generally show consistently lower JSD and WDZ, while Qwen3.5 MoE variants underperform relative to their parameter counts across categories.Source: D Extended Model Results · p. 22

E Per-Distribution KS@100 Breakdown

• 43.04% for Bernoulli, 34.78% for Categorical, and 16.52% for Discrete Uniform make small-support discrete distributions easiest at KS@100.Their limited support and likely pretraining familiarity may allow sufficient diversity to pass the KS test.Source: E Per-Distribution KS@100 Breakdown · p. 22
• 1.74% for Fréchet and Dirichlet, 5.22% for Negative Binomial, and 6.09% for Negative Multinomial make heavy-tailed and multivariate distributions hardest.These distributions require reproducing long tails or correlated multivariate structure.Source: E Per-Distribution KS@100 Breakdown · p. 22
• Compound Poisson, Erlang, Inverse Gaussian, and Pareto all score below 9% at KS@100, indicating difficulty with precise scale and shape calibration.The results identify calibration of distributional scale and shape as a recurring challenge.Source: E Per-Distribution KS@100 Breakdown · p. 22
• Beta falls from 87.39% at KS@10 to 4.78% at KS@100, consistent with severe support misspecification for bounded continuous distributions.This is among the steepest declines in the per-distribution results.Source: E Per-Distribution KS@100 Breakdown · p. 22

F Explicit vs. Implicit Prompting

• Nemotron-3 Super 120B leads explicit and implicit prompting at 41.42% and 26.42% KS@100, respectively, a 15-point advantage for explicit prompts.Explicit prompts name or describe the distribution, whereas implicit prompts require inferring the stochastic process from context.Source: F Explicit vs. Implicit Prompting · p. 23
• GPT-5.4, DeepSeek V3.2, OLMo-3 7B, and Claude Sonnet 4.6 score higher implicitly than explicitly, with gaps attributed to anchoring on memorized distribution prototypes.Their implicit advantages are 20.13% vs. 14.23%, 23.27% vs. 17.57%, 8.18% vs. 5.02%, and 6.92% vs. 3.78%, respectively.Source: F Explicit vs. Implicit Prompting · p. 23
• Explicit prompting improves KS@100 for most models, while the explicit/implicit gap narrows among weaker models, suggesting fundamental distributional understanding becomes the bottleneck.Naming the distribution reduces the task to parameter estimation and sampling, but weaker models gain less from that format.Source: F Explicit vs. Implicit Prompting · p. 23

Unimodal vs. Multimodal Distribution Complexity

• 42.50% versus 33.65% for Nemotron-3 Super 120B and 38.75% versus 22.96% for GPT-4o show multimodal tasks can outperform unimodal tasks at KS@100.The broader support of mixture distributions may make passing the KS test easier despite imperfect mode coverage.Source: G Unimodal vs. Multimodal Distribution… · p. 23
• Mercury-2, Claude Sonnet 4.6, Phi-3.5 Mini, and both Qwen3.5 MoE variants collapse on multimodal tasks while retaining some unimodal performance.This suggests mixture targets require sustaining multiple output modes, which deterministic models struggle to maintain.Source: G Unimodal vs. Multimodal Distribution… · pp. 23–24
• 7.50% versus 18.87% for GPT-5.4 at KS@100 reverses the stronger-model pattern, consistent with single-point collapse being especially damaging for separated modes.Across target distributions, above-average distributions are relatively easier and below-average distributions are harder.Source: G Unimodal vs. Multimodal Distribution… · p. 24, Table 12 · p. 24

Model-dependent reversal

• At KS100, Nemotron-3 Super 120B (36.36% vs. 34.50%) and GPT-4o (27.27% vs. 25.00%) perform comparably on concentrated versus spread-out distributions.For many strong models, distributional spread is not substantially more challenging.Source: H Effect of Distributional Spread · p. 25
• At KS100, Grok-4.1-fast (11.62% vs. 0.50%) and Claude Sonnet 4.6 (9.64% vs. 0.50%) nearly collapse on spread-out distributions.Phi-3.5 Mini (4.55% vs. 0.50%) and both Qwen3.5 MoE variants (around 6.5% vs. 0.00%) show similar failures, consistent with anchoring near the distribution mode.Source: H Effect of Distributional Spread · p. 25
• At KS100, Ministral-3B instruct (21.00% vs. 12.12%), Llama-3.2-1B (16.50% vs. 6.06%), and Llama-3.1-8B instruct (6.50% vs. 2.02%) score higher on spread-out tasks.These models appear diverse enough to cover broad support but insufficiently precise for concentrated distributions.Source: H Effect of Distributional Spread · p. 25

I Error Analysis

• Around 59% RealWorld KS@100 for Llama-3.2-1B remains high and stable despite around 0.52 Text JSD and around 29.82 Text WDZ.This pattern indicates that the model’s strong RealWorld performance is not an evaluation artifact, while its Text metrics remain poor.Source: I Error Analysis · p. 26
• Nemotron-3 Super 120B combines 40.34% Text KS@100 with 0.48 Text JSD, while OLMo-3 7B has around 59.16 RealWorld WDZ, consistently across runs.These results show stable tension between KS-based and distance metrics and a persistent structural failure rather than sampling noise.Source: I Error Analysis · p. 26

J Ground Truth Sensitivity

• KS@100 standard deviations stay below 0.42 across models under independent ground-truth sets, confirming robustness to the specific ground-truth samples.JSD and WDZ deviations are similarly tight for most models, although variability increases slightly from Random to RealWorld settings.Source: J Ground Truth Sensitivity · p. 26, Table 16 · p. 28
• Qwen-3.5-35B-a3b reaches up to 2.55 JSD instability, whereas Llama-3.1-70B and DeepSeek V3.2 are among the most stable models.The comparison spans all reported metrics and settings in the ground-truth sensitivity analysis.Source: J Ground Truth Sensitivity · p. 26

I Error Analysis

• Narrow error bars across models and metrics show that benchmark results are stable and reproducible, validating use of a single evaluation run.Figure 6 reports mean and min-max ranges across three repeated runs; variance from ground-truth resampling is negligible relative to model and category differences.Source: I Error Analysis · pp. 25–26, Figure 6 · p. 27

J Ground Truth Sensitivity

• Releasing fixed ground-truth samples and generation code enables direct replication, while the metric and framework remain agnostic to alternative ground-truth instantiations.Users can change sample counts, random seeds, or sampling procedures when evaluating under different conditions.Source: J Ground Truth Sensitivity · pp. 27–28

K Instruction Following Analysis of the Models

• RealWorld retry costs follow inverse scaling: Llama-3.2-1B retries on 68.6% of calls at 4.28 attempts, versus 0.0% for GPT-5.4 and 0.2% for Qwen3.5-397B.Failed generations are discarded and retried up to five times, retaining only valid samples so retries measure formatting reliability without biasing metrics.Source: K Instruction Following Analysis of… · p. 28
• Shuffling is nearly retry-free, but Text and Code retries peak for Mercury-2 at 13.6% and Claude-sonnet-4.6 at 10.9%, reflecting format quirks rather than capability.Claude-sonnet-4.6 also has the highest average attempt count overall at 1.406, while Mercury-2 is an outlier across categories.Source: K Instruction Following Analysis of… · p. 28

L Output Diversity Analysis

• 36.99 and 36.38 unique items were the highest mean diversities, while Qwen3.5-397B-a17b and Qwen3.5-35B-a3b achieved only 31.37 and 28.52.All models remained below the attainable ceiling of approximately 39.8 unique items, and the two Qwen models were the only ones whose maximum never exceeded 35.Source: L Output Diversity Analysis · p. 28
• Lower-diversity models tended to have higher variance and longer left tails, with GPT-5.4 and Claude-sonnet-4.6 reaching only 25–26 unique items in their worst runs.This pattern suggests diversity loss is driven primarily by intermittent mode collapse rather than a uniform downward shift.Source: L Output Diversity Analysis · p. 29
• The remaining models clustered within a comparatively narrow mean-diversity band of 33.5–35.5 unique items.Diversity was measured as unique items produced from approximately 40 presented items, aggregated over 1000 runs at temperature 1.0.Source: L Output Diversity Analysis · pp. 28–29

M Additional Qualitative Analysis

• Across four representative tasks, most models collapse to narrow or single-point distributions instead of matching ground truth, with Nemotron-3 Super 120B and DeepSeek V3.2 producing the broadest aligned samples.Figures 8–11 extend the qualitative analysis across the full model pool and multiple task types and distributions.Source: M Additional Qualitative Analysis · p. 29
• The Fréchet task exposes failure to capture heavy right tails, whereas Truncated Normal is more tractable and reveals variation from reasonable approximations to complete collapse.DeepSeek V3.2 and Llama-3.1-70B approximate the Truncated Normal bell shape, while Claude Sonnet 4.6 and GPT-5.4 collapse.Source: M Additional Qualitative Analysis · p. 29, Figure 8 · p. 31, Figure 9 · p. 32
• Implicit binomial and code-based Poisson tasks are harder because models must infer distributional structure, separating models that interpret stochastic code from those collapsing to zero or small integers.Even models performing reasonably on explicit tasks show greater variance and misalignment on the implicit binomial task.Source: M Additional Qualitative Analysis · p. 29, Figure 10 · p. 33, Figure 11 · p. 34

N Prompts

• Prompt N uses explicit or implicit, concentrated or spread-out templates for text and code tasks, mixture-distribution prompts for multimodal tasks, and task-specific LLM extractors for parsing outputs.Sampling prompts are provided separately for text and code evaluation, while extraction covers standard text/code, list, shuffling, and real-world tasks.Source: N Prompts · pp. 29–30
• The templates generate sampling tasks from distribution names and Wikipedia-derived properties, using indirect self-contained scenarios for implicit questions rather than naming the distribution directly.They require questions to focus on drawing random samples instead of asking for means, medians, or other typical statistics.Source: N Prompts · pp. 37–38
• The concentrated variants choose valid, reasonable parameters that minimize the distribution’s support or typical sample range when feasible.The spread-out implicit variant instead maximizes that range while preserving valid parameters.Source: N Prompts · pp. 38–39
• Malformed inputs or missing, inappropriate, or ungeneratable fields trigger an error-only JSON object rather than partial results.Examples include invalid distribution names, irrelevant distribution descriptions, and parameters that cannot be inferred.Source: N Prompts · pp. 38, 41
• Successful outputs contain a question, parameters, context, expected_answer_type, and code_snippet, with numerical_value indicating a single numerical answer.The templates require all fields to be present and contextually appropriate for the selected distribution.Source: N Prompts · pp. 38, 41
• The explicit code template generates runnable Python sampling code, with valid parameters and distribution-specific relevance required.The specified implementation may use NumPy, SciPy, or similar libraries.Source: N Prompts · pp. 38, 40
• Both prompt variants require structured success outputs containing code, parameters, context, and an expected answer type, and require an error-only JSON response for invalid inputs or inappropriate outputs.Inputs must be validated before proceeding, and generated fields must be checked for completeness and contextual appropriateness.Source: N Prompts · p. 41
• Prompt N.6 requests explicit, runnable Python sampling code based on distribution details, with valid parameters chosen to spread the typical sample range when feasible.The prompt prohibits descriptive-statistics queries and targets random sampling procedures.Source: N Prompts · pp. 41–42
• Prompt N.7 requests self-contained implicit sampling code that avoids naming the distribution and produces a single numerical outcome through an indirect scenario or procedure.It also requires valid parameters and focuses exclusively on random sampling rather than descriptive statistics.Source: N Prompts · pp. 42–43
• N.7 instead prefers valid parameters that make the distribution’s support or typical sample range as concentrated as possible.This creates a concentrated counterpart to the spread-out explicit prompt while retaining the sampling-only requirement.Source: N Prompts · p. 43
• Prompt N.8 generates an implicit sampling task by asking for self-contained Python code that indirectly represents a distribution and produces one numerical sample.It selects valid, preferably spread-out parameters while avoiding direct distribution names and descriptive statistics.Source: N Prompts · p. 44
• Both templates require explicit valid distribution parameters and outputs focused on drawing a single numerical sample rather than computing descriptive statistics.The implicit template returns code, parameters, context, and an expected numerical answer type; the multimodal template returns a question plus corresponding sampling metadata.Source: N Prompts · pp. 44–46
• Malformed inputs or missing, inappropriate, or ungenerable fields cause both prompt variants to return only a JSON error object instead of partial results.Each template also requires validating that all output fields are present and contextually appropriate before returning success.Source: N Prompts · pp. 41, 45
• Prompt N.9 targets multimodal sampling by preserving an existing multimodal distribution or constructing a valid two-component version from a unimodal input.It requests concentrated parameter settings, a human-readable sampling question, and metadata identifying the component count and whether multimodality was inherent or constructed.Source: N Prompts · pp. 45–46
• The multimodal task prompt preserves inherently multimodal distributions and converts unimodal inputs into valid two-component constructions before generating sampling questions.It selects valid parameters, favors dispersed support or typical sample ranges, and requires questions that are solvable from the specified distribution rather than asking for typical statistics.Source: N Prompts · p. 47
• The multimodal generator returns a structured object containing the question, parameters, component count, context, answer type, modality flag, and executable sampling code.Missing, malformed, or contextually inappropriate inputs or fields trigger an error-only JSON response instead of partial results.Source: N Prompts · pp. 41, 48
• Text and code sampling prompts require one valid random outcome as plain text, without explanation, code, or formatting.The text prompt requests a sampled value, while the code prompt requests one valid possible output for nondeterministic execution.Source: N Prompts · p. 49
• The answer extractor emits a single JSON object with either a reported number and its exact span and location or null when no final number can be identified.It uses only explicitly presented numbers, ignores unrelated parameter values, and defaults conservatively for incomplete or ambiguous outputs.Source: N Prompts · p. 49
• The numeric extractor returns exactly the requested number of explicitly present final answers in a JSON object, ignoring unrelated numbers and using null when extraction is not confident.It preserves output order, requires JSON numbers, and forbids inference or unstated calculations.Source: N Prompts · pp. 50–51
• All extractors require exactly one valid JSON object with prescribed keys and no additional text, markdown, or code fences.The numeric, list, and textual variants respectively use numbers, arrays, and strings or null as their extracted value types.Source: N Prompts · pp. 51–52
• The shuffling-task extractor returns one JSON list answer or null, choosing the first complete list and preserving its order and element types without synthesizing missing elements.Allowed elements are strings, integers, and floats; Python-style single-quoted strings are converted to JSON strings.Source: N Prompts · p. 51
• The real-world extractor returns one textual answer or null, preserving multiline order and internal newlines while trimming only outer whitespace.When alternatives appear, it selects the first explicit candidate and does not invent missing content.Source: N Prompts · pp. 51–52