The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

Controlled axes

• Sweeping classifier-free guidance, compute, model size, and learning rate does not close the training–generation gap, while relative FID noise remains stable across scale and budget.The learning-rate sweep transfers across widths using µP, whose parameterization makes the optimal learning rate width-invariant.Source: §1 · p. 2

FID lottery

• Retraining moves FID 3.2× more than resampling, while training variability dominates across SiT model sizes and budgets.A single-seed FID can therefore fall below the observed noise floor of genuine seed-to-seed variation.Source: §1 · p. 2, Figure 2 · p. 3

Randomness sources

• The training lottery combines random initialization, data ordering, hardware nondeterminism, and per-step Gaussian loss noise.Unlike sampling noise, flow-matching noise is injected at every gradient step and is resolved differently by independent training runs.Source: §1 · p. 2, Figure 1 · p. 2

Guidance tuning

• Per-cell golden-section classifier-free-guidance tuning tightens the noise floor but substantially increases evaluation cost and reshuffles which seeds rank best.The protocol tunes guidance separately for every training and sampling seed pair.Source: §1 · p. 2

Luck of the draw

• A lucky training seed reaches the same FID with up to 2× less compute than an unlucky seed.This result demonstrates that seed choice can materially alter the compute required to reach a target FID.Source: §1 · p. 3

FID and its limitations

• FID remains dominant [7] [8] [62] despite finite-sample bias, preprocessing sensitivity, class-balance effects, human-rater disagreement, and proposed alternatives such as KID and CMMD.Chong and Forsyth [12] show ranking can flip from sampling noise alone, while Parmar et al. [72] report shifts comparable to claimed state-of-the-art gains.Source: §2 · p. 3

Reproducibility and statistical methodology

• Reproducibility research finds nominally identical systems can diverge across seeds and even invert rankings, motivating variance decomposition, stronger reporting, and classical statistical tests.Bouthillier et al. [10] separate algorithmic, data, and implementation variance; Banerjee et al. [1] and Bench and Thomas [4] are closest methodological precursors.Source: §2 · p. 3

Why runs differ

• Training variability is linked to distinct optimization basins shaped by initialization, batch order, and optimizer stochasticity, although architecture changes basin geometry rather than multiplicity [30] [31] [29] [35] [53] [36] [93] [23].Nagarajan and Kolter [69] formalise why such gaps are intrinsic, while Wenzel et al. [95] and Jordan [45] exploit them for uncertainty quantification.Source: §2 · pp. 3–4

Novelty

• This paper fills that gap by measuring percent-level FID fluctuations among near-identical noise-to-image maps and isolating meaningful training-seed variation from numerical nondeterminism.Figure 3 reports same-seed between-seed σ of 0.047, below the within-seed sampling floor, despite trained-weight differences of approximately 5–6% of their norm.Source: §2 · p. 4, Figure 3 · p. 4

Generative reproducibility, scaling, and seeds

• Generative-model variance remains understudied: prior work examines scaling, function-level reproducibility, or inference-time seed lotteries, but not systematic metric-level variance across training seeds, architectures, and checkpoints.Xu et al. [97] study an inference-time seed lottery, whereas Chong and Forsyth [12] and Bench and Thomas [4] vary neither training seeds, architectures, nor checkpoints.Source: §2 · p. 4

Experimental Setting

• SiT models span four widths on class-conditional ImageNet 256×256, using conditional flow matching and FID in Inception-V3 feature space.The loss redraws Gaussian ε ∼N(0, I) at every gradient step, adding a training-time random-number generator distinct from model weights.Source: §3 · p. 4
• FID is a distribution-level Fréchet distance between Inception-feature Gaussians fit to ImageNet and 50,000 generated images, not a per-image score.Figure 4 shows a 12× FID range from 43 to 3.6, with quality improving as FID decreases.Source: Figure 4 · p. 5

The two-axis panel

• Each experiment forms an N×K panel of independently trained models and multiple generations per model, separating training seeds from sampling seeds.Training seeds control initialization, data-loader order, and per-step flow-matching noise; sampling seeds control generation-time initial noise.Source: §3 · p. 5

Notation

• σwithin measures fixed-model generation noise, σbetween measures training-seed spread, and CoV=σ/µ (%) enables comparison across panels with different absolute FID.The statistics average within-model variation across sampling seeds and compare per-training-seed means across independently trained models.Source: §3 · p. 5

4.1 Training Variability Dominates Evaluation Variability

• 3.2× larger training-seed than sampling-seed effects make single-run FID comparisons unreliable, with only additional training seeds reducing the dominant variance.Across 25 training and 10 sampling seeds, σbetween =0.438 versus σwithin =0.137, while sampling jitter remains about 0.14 FID units.Source: §4.1 · p. 6

4.2 Flow-Matching Noise Leads Init and Data Order

• 0.336 between-seed σ from flow-matching noise exceeds initialization’s 0.294 and data order’s 0.221, while their combined effect is sub-additive and numerical nondeterminism is negligible.Noise alone reproduces 77% of baseline σ, initialization 67%, and data order 51%; the intended random draws, rather than DDP numerical noise, drive the lottery.Source: §4.2 · pp. 6–7

4.3 GS-FID Halves the Floor but Reshuffles Rankings

• 0.67% CoV under per-cell CFG tuning halves the unguided 1.26% spread, but ranking correlation is only ρ=0.73, so tuning changes which seeds appear best.Eight of 25 training seeds shift by at least five ranking places, and tuned sampling jitter still matters because the between-to-within σ ratio falls only to 1.87×.Source: Figure 5 · p. 6, §4.3 · pp. 7–8

4.4 The 1–2 % CoV Floor Survives Scale and Compute

• 1–2% CoV persists across model sizes and training budgets, with a 1.30% median across 76 cells, so gains below roughly 2×CoV require multi-seed confirmation.The spread does not collapse during convergence, and larger architectures do not automatically tighten it because per-step training noise is continually regenerated.Source: Figure 6 · p. 7, §4.4 · p. 8

4.5 The Luck of the Draw: A 1.2–2.0× Convergence Gap

• 1.25–2.0× apparent convergence-speed gaps arise from seed luck alone, with the largest gap on SiT-XL where a lucky seed reaches at 1M what an unlucky seed reaches at 2M.Past approximately 1.5M steps, a seed swap can move FID by amounts comparable to 200–500k additional training steps; N=5 lowers the detectable threshold to approximately 0.25 FID.Source: Figure 7 · p. 8, §4.5 · p. 9

4.6 µP Transfers a 1.7× LR Window, Not a Point

• 1.7× LR windows transfer across SiT widths under GS-FID, whereas unguided selection favors an unstable edge and the CoV does not minimize at the FID-optimal LR.GS-FID identifies flat-bottomed valleys near ω⋆≈2–3×10^-4, while unguided FID selects 5×10^-4; the practical optimum retains a 1.7–2.3% CoV.Source: Figure 8 · p. 9, §4.6 · p. 10

4 Experiments

• The experiments recommend reporting multi-training-seed error bars, treating FID gaps below approximately 2% as inconclusive, and evaluating with the same guided or unguided protocol used for tuning.More sampling cannot replace more training runs, and GS-FID should be paired with its search tolerance when used for comparison.Source: §4.6 · p. 10

Limitations

• The ≈1.3% CoV is a calibration target for SiT, flow matching, class-conditional ImageNet 256×256, and Inception-V3 FID, not a universal constant.Other backbones, objectives, diffusion settings, domains, and Fréchet variants may have different floors.Source: §5 · p. 10
• The panel is finite at 20–25 training seeds and 10 sampling seeds, and behavior beyond SiT-XL or 2M steps remains extrapolative.Appendix analyses found fidelity metrics track Inception FID closely, while recall is an outlier.Source: §5 · p. 11

Outlook

• The proposed measurement framework uses a two-axis training-and-sampling seed panel summarized by per-source variance decomposition, but its 1–2% floor requires validation elsewhere.Future work should test other training methods, model families, and Fréchet-style metrics, and seek cheaper proxies for seed-noise floors.Source: §5 · p. 11

Broader impact

• The ≈1.3% floor can prevent sub-floor retraining waste, but extra seeds can also manufacture more favorable headline results that should be priced into peer review.The paper makes this dual-use temptation explicit in Sec. 4.5.Source: §5 · p. 11

A Overview

• The supplement covers empirical seed-lottery analyses, a theoretical justification of golden-section search, and metric-robustness stress tests.The stress tests replicate the main-paper claims under DINOv2 FID and Inception PRDC.Source: A Overview · p. 19
• 2–3× lucky-seed speedups emerge on SiT-L/XL for precision, density, and coverage, while training randomness still dominates generation randomness across fidelity-axis metrics.Inception recall alone reverses this asymmetry because its k-NN balls are computed on the per-evaluation generated set.Source: A Overview · p. 19
• The noise > init > data hierarchy and training-over-generation lottery asymmetry hold under DINOv2 FID and all four Inception PRDC metrics.These replications sharpen rather than soften the main paper’s conclusions.Source: A Overview · p. 19
• Appendices B–E provide configurations, consolidated headline tables, additional Inception-FID analyses, and theory for Algorithm 1, FID(CFG) unimodality, and optimum-noise sensitivity.The empirical analyses include rank instability, init-seed universality, µP brackets, and an FID-to-confidence-interval lookup.Source: A Overview · p. 19
• Appendices F–G add row-for-row DINOv2/PRDC replications and per-class galleries showing when perceptual quality follows or diverges from the FID gradient.The galleries compare generated samples ordered by FID with and without classifier-free guidance.Source: A Overview · p. 19
• Figure 9 depicts the 3.2× training-versus-generation asymmetry as two slot machines: three training randomness sources produce one network, while ten sampling seeds score it.The visualization pairs the metaphor with the measured SiT-B/2 panel.Source: A Overview · p. 19

B Per-experiment configurations

• 25 independent training runs per cell and 10 sampling seeds per model define the default FID panel, using 50,000 samples and a shared Inception-V3 evaluation pipeline.The protocol uses ImageNet train-set reference statistics and a deterministic ODE sampler; the experiments consumed approximately 100,000 H100 GPU-hours overall.Source: B Per-experiment configurations · p. 19
• The experiments separately vary noise, initialization, and data-order seeds, while also testing guidance, compute, model scale, and µP learning rates across replicated panels.The variance ablations use 24–25 runs at 400k steps; scaling uses 25 networks per size, and the µP sweep covers 10 learning rates across four widths.Source: B Per-experiment configurations · p. 20

C Companion summary tables

• The companion analyses consolidate headline tables on variance decomposition, scaling, µP learning rates, and expected seed-induced confidence intervals.The supplementary analyses also test whether training-seed rankings remain stable under different within-seed summaries and whether good initialization transfers across data-and-noise pairings.Source: C Companion summary tables · p. 20, D Additional analyses on Inception FID · pp. 20–21

D Additional analyses on Inception FID

• The variance sources are not independent: naively summing their contributions gives approximately 0.50, overshooting the observed VARY-ALL σ of 0.44 by 14%.This result qualifies the interpretation of the single-source ablations and accompanies the variance-decomposition table.Source: Table 1 · p. 21
• Across compute and model size, mean FID falls by approximately 1.7–2× while σbetween shrinks 1.7–2.4×, leaving CoVbetween near a 1–2% band.The scaling table compares the beginning and end of the 200k-to-2M-step sweep.Source: Table 2 · p. 21

D.1 Per-train-seed violin panels by variability source

• The four D.1 panels show invariant within-seed jitter, with σwithin remaining between 0.137 and 0.152 regardless of which training randomness varies.Thus, evaluation noise is determined by the converged SiT-B/2 weights rather than by the randomness sources used to train them.Source: §D.1 · p. 21
• Between-seed spread decreases from σ=0.438 when all sources vary to 0.336 for noise, 0.294 for initialization, and 0.221 for data order.The mean-tick range simultaneously drops from approximately 1.7 FID to under approximately 0.9 FID, while violin heights remain essentially unchanged.Source: §D.1 · p. 21, Figure 10 · p. 22
• The panels show evenly distributed training-seed means rather than a heavy-tailed minority of outliers, consistent with well-behaved (max−min)/σ ratios.This supports interpreting the observed spread as broadly distributed across seeds rather than driven by a few anomalous runs.Source: §D.1 · p. 21

D.2 Summary statistics reshuffle the training-seed ranking

• Summary statistics change which training seed appears best: rankings based on per-seed mean, minimum, and maximum FID exhibit dense crossings, despite agreement across Inception and DINOv2.The best-case sampling seed need not be the average-case winner; Inception and DINOv2 agree with Spearman ρ≥0.94 across criteria.Source: §D.2 · p. 22, Figure 14 · p. 24

D.3 Are “good” init seeds universal?

• Initialization seeds are not universally good: their pairwise rankings are only weakly concordant, and the best grand-mean initialization wins just 5 of 15 pairings.Kendall’s W is 0.41 for mean rankings, while the best initialization is top-ranked in only 5/15 data–noise pairings.Source: §D.3 · p. 23, Figure 15 · p. 25

D.4 Detailed numbers for the µP sweep

• Across central learning rates, between-seed CoV remains within 1–3% across model sizes and both FID protocols, but at 5×10^-4 it can rise sharply or coincide with divergence.At the rightmost learning rate, GS-FID CoV jumps by an order of magnitude on SiT-B; SiT-S diverges on three of ten seeds.Source: §D.4 · p. 24
• Unguided and GS-FID protocols often select different per-seed learning rates, and trusting the unguided argmin can be 7.7–11.5% worse in GS-FID with a 5–12× wider seed envelope.Per-seed argmins disagree for 8/10 SiT-S, 6/8 SiT-B, and 5/10 SiT-L seeds.Source: §D.4 · pp. 24–25

D.5 What CI should I expect at my reported FID?

• 1.30% median CoV yields substantial seed-only uncertainty: at FID≈15, one trained model has a ±0.38 95% CI, and reducing it below ±0.1 requires at least 25 training seeds.Across 76 model–step cells, the CoV spans 0.88–1.73%; the CI shrinks only as 1/√N.Source: §D.5 · pp. 25–26

E Theory of golden-section search for FID(CFG)

• The theory section connects Algorithm 1’s global-optimum guarantee to the Gaussian feature model underlying FID, while noting that unequal covariances require additional convexity conditions.The equal-covariance assumption yields strict unimodality; outside it, the observed convergence of 250 searches remains consistent with global unimodality.Source: E Theory of golden-section search for… · p. 26, §E.2 · pp. 27–28

E.1 Convergence

• Golden-section search contracts its CFG bracket by ρ per iteration and required about 12 evaluations theoretically versus a measured median of 14.The setup used bracket width 1 and tolerance 0.01; the small excess reflects an implementation safeguard.Source: §E.1 · pp. 26–27, Figure 17 · p. 27, Figure 18 · p. 27

E.2 Unimodality of FID(CFG) under a Gaussian feature model

• Under equal-covariance Gaussian features, FID(CFG) is a strictly convex quadratic with a unique global minimum, making golden-section search globally valid on every interval.The result assumes µc ≠ µ0; the empirical optimum is w⋆≈0.027, while unequal-covariance experiments also converge to a tight cluster with σw⋆=0.045.Source: §E.2 · p. 28

E.3 Noise robustness

• FID evaluation noise can cause golden-section search to retain the wrong half-bracket with constant probability when probe separation is noise-dominated.The returned optimization error includes a deterministic ε² bracket residual and a noise-induced term.Source: §E.3 · p. 28
• 0.17 predicted noise-induced guidance spread mildly exceeds the observed σ_w⋆ = 0.045, suggesting sharper-than-estimated FID curvature near optimal CFG.The prediction uses σ_y ≈ 0.137 FID and empirical curvature f′′(w⋆) ≈ O(10).Source: §E.3 · p. 29

F Replication across DINOv2 FID and Inception PRDC

• DINOv2 FID strengthens the seed-lottery narrative, while Inception precision, density, and coverage track FID; recall is sampling-dominated and anomalous.These replications reuse the main paper’s panels across five complementary metrics.Source: F Replication across DINOv2 FID and… · p. 29

F.1 Metric definitions

• The appendix evaluates six metrics: Inception-V3 and DINOv2 Fréchet distances plus four Inception PRDC metrics.All use 50k real ImageNet training images and 50k generated samples from a trained SiT, mapped by a frozen feature extractor.Source: §F.1 · p. 29
• DINOv2 FID and Inception FID use Gaussian Fréchet distances in different feature spaces, so their values are not comparable in absolute units; lower is better.DINOv2 FID values are roughly an order of magnitude larger at matched generative quality.Source: §F.1 · p. 29
• Precision, recall, density, and coverage separate fidelity and diversity through k-NN-ball tests, with recall uniquely generated-anchored and coverage real-anchored.Precision and density address fidelity, whereas recall and coverage address diversity; higher is better for all four metrics.Source: §F.1 · pp. 29–30, Table 4 · p. 30
• Recall is the only PRDC metric whose within-seed σ exceeds the binomial estimator floor, by approximately 2× on this panel.Its generated-set k-NN balls are redrawn at every sampling seed, unlike the real-anchored geometry of precision, density, and coverage.Source: §F.1 · p. 30

DINOv2 FID

• 4.82×: DINOv2 FID’s between-to-within ratio exceeds Inception FID’s 3.19×, strengthening the training-lottery asymmetry.The relative between-seed CoV floor tightens from 1.26% to 0.86%.Source: §F.2 · p. 30, Figure 20 · p. 32

PRDC metrics

• 1.14×, 1.41×, and 1.12×: precision, density, and coverage place training and sampling variation near parity.Sampling-only confidence intervals cover roughly half of the true seed-induced envelope for these metrics.Source: §F.2 · p. 30

Recall

• 0.28×: Inception recall inverts the asymmetry, with within-seed sampling variation exceeding between-seed training variation by over threefold.Recall is the only metric for which the sampling-seed confidence interval is the appropriate fixed-model interval.Source: §F.2 · pp. 30–31

Mechanism

• Generated-anchored recall balls shift with each sampling seed, correlating coverage indicators across real points and reducing the effective sample size below 50,000.Real-anchored precision, density, and coverage instead test redrawn generated points against an unchanged ball structure, limiting variation near the binomial floor.Source: §F.2 · p. 31

Empirical check

• Recall’s within-seed σ is roughly twice the binomial floor, while precision and coverage are at or below it; recall also saturates at µ=0.314.The small between-seed σ and structurally inflated within-seed σ produce the denominator inversion.Source: §F.2 · pp. 31–32

F.3 Flow-matching noise dominates for the fidelity metrics

• The F.3 decomposition rescored four single-source seed conditions under each metric, expressing each condition’s between-seed σ relative to the fully stochastic baseline.The conditions are VARY-NOISE, VARY-INIT, VARY-DATA, and VARY-NOISEDATA.Source: §F.3 · p. 32
• About three-quarters of baseline σ comes from flow-matching noise across five well-behaved metrics, versus about two-thirds from initialization and half from data order.The ranking is VARY-NOISE > VARY-INIT > VARY-DATA, and quadrature overshoots the fully stochastic baseline by 7–19%.Source: §F.3 · p. 32
• On Inception Precision, between-seed and within-seed spreads are comparable, with σbetween/σwithin =1.14× and per-seed means spanning 0.480–0.491.A multi-sampling-seed confidence interval therefore covers roughly half of the seed-induced precision envelope, unlike FID’s quarter.Source: Figure 21 · p. 33
• Recall is not decomposable: three single-source conditions exceed the vary-all baseline at 112–126%, because sampling jitter dominates the between-seed component.The framework therefore requires a between-seed component larger than the within-seed component, which recall lacks on this panel.Source: §F.3 · p. 33

F.4 Golden-section search transfers cleanly to DINOv2 FID

• The DINOv2 analysis reruns per-cell golden-section search, then reports all complementary metrics at the CFG optimizing Inception FID rather than separately optimizing each metric.Thus, the supplementary metrics are readouts at the Inception-FID-optimal operating point.Source: §F.4 · p. 33
• Golden-section guidance does not tighten DINOv2 FID: CoVbetween changes from 0.86% unguided to 0.94% at the Inception-FID-optimal CFG.By contrast, GS-FID reduces Inception FID’s CoVbetween from 1.26% to 0.67%; the DINOv2 change is statistically indistinguishable and requires about 14 extra evaluations per cell.Source: §F.4 · pp. 33–34
• The recovered CFG shifts PRDC to a high-fidelity, low-recall regime: precision, density, and coverage roughly double or triple, while recall drops by more than half.These GS-FID PRDC values should not be compared with unguided PRDC across studies because they represent different operating points.Source: §F.4 · p. 34

• 1–2% CoV covers DINOv2 FID, Inception precision, density, and coverage across every checkpoint and model size, while recall remains at 1–1.8% on SiT-B/L.The relative noise floor therefore persists across scales, with recall not shrinking with compute.Source: §F.5 · p. 34
• DINOv2 FID is tighter than Inception FID at every scale and checkpoint past 200k, although both extractors show similar spread reduction with compute.The difference is primarily in the floor itself rather than the absolute reduction across training.Source: §F.5 · p. 34
• Below 0.4% CoV, Inception precision, density, and coverage on SiT-XL reach their lowest floors by 1M steps, with median precision and coverage below 0.3%.Their maxima remain below 0.7%, placing these large-model fidelity-axis metrics an order of magnitude below typical 3–4% recipe-comparison gains.Source: §F.5 · p. 35
• 1–1.8% recall CoV does not shrink with compute: recall σ grows on SiT-B/L, stays flat on SiT-XL, and ranking stability remains essentially absent.Spearman ρ stays below 0.6 before the final checkpoint and dips below zero early on SiT-S/L/XL.Source: §F.5 · pp. 34–35
• 2.9–5.5 for (max−min)/σ across all 76 cells indicates a Gaussian-shaped seed spread for every metric rather than heavy-tailed training failures.This brackets the 3.5–3.9 range predicted for Gaussian samples of size n∈[19, 25].Source: §F.5 · p. 35

Setup

• The protocol defines the unlucky seed as worst at 2M and the lucky seed as best, measuring the first checkpoint where lucky matches the unlucky target.For higher-better metrics, direction is flipped so “lucky” identifies the practically preferred seed.Source: §F.6 · p. 35

Results

• 2.0–2.9× speedups on SiT-L/XL arise for Inception precision, density, and coverage, exceeding the corresponding FID savings.DINOv2 FID transfers the effect with 1.18–1.67× speedups versus 1.18–1.82× for Inception FID.Source: §F.6 · p. 35

Limitation

• 6–10× nominal recall speedups are degenerate because lucky seeds already exceed the 2M unlucky target at 200k–300k.Recall is approximately compute-invariant on this panel, with σ-shrink near or below 1, so the lucky-speedup framing does not apply.Source: §F.6 · p. 35

Metric variability

• Each metric has a distinct scale-invariant relative noise floor, with Inception FID spanning 0.88–1.73% CoV across compute and model scale.DINOv2 FID and Inception precision/coverage are lower, while recall and density fall between them; recall’s floor is wider where σ does not shrink with compute.Source: Figure 23 · p. 36

Setup

• The diagnostic evaluates 10 ImageNet classes and fixed initial-noise tensors across 16 FID-ordered checkpoints using a 50-step flow_euler_sampler.Unguided levels span FID ∈[13.5, 87.4], while guided DiT-XL checkpoints span FID ∈[3.55, 11.3] at selected best CFG.Source: G What does FID look like? · p. 36

What the galleries show

• Unguided samples degrade monotonically as FID increases, progressing from well-formed objects to textureless blobs and noisy patches.The galleries hold initial noise fixed across columns to isolate training-time and guidance effects.Source: G What does FID look like? · p. 37, Figure 35 · p. 49
• With appropriate classifier-free guidance, even the highest-FID checkpoint produces visually clean, recognizable samples.This behavior is consistent with guidance compressing the FID range over which it is computed.Source: G What does FID look like? · p. 37, Figure 25 · p. 38