PaperlayerView exact revision on arXiv

Source-linked AI summary

The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

Nicolas Dufour, Alexei A. Efros, Patrick Pérez

arXiv:2606.20536v1cs.CV

TL;DR

FID is widely reported as a single score, but its reproducibility across training and sampling seeds is unclear. Treating FID as a two-axis random variable, the paper finds retraining changes FID 3.2× more than resampling and that variability remains around a 1–2% CoV floor.

  • Problem

    Most image-generation papers report one FID from one model and sampling seed, leaving its variability across training and generation seeds insufficiently measured.

  • Method

    The paper measures FID across training–sampling seed panels for several hundred SiT networks, decomposing randomness and varying guidance, compute, model size, and learning rate.

  • Results

    Retraining changes FID 3.2× more than resampling, while seed-to-seed variability remains within a 0.74%–2.06% CoV range across compute budgets and model scales.

  • Takeaways & Limitations

    FID comparisons below the measured variability floor should be treated as inconclusive and supported by multi-seed evaluation rather than a single score.

Abstract

from arXiv · show

The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.

1 Introduction

The introduction frames FID as a consequential but seed-dependent arbiter of image-generation quality, governed by separate training and generation lotteries. It proposes measuring FID across both axes because retraining contributes substantially more variability than resampling a fixed network.

  • Motivation: FID anchors central claims and leaderboard rankings, although reported values hide a chain of pseudo-random draws.The introduction notes that half-unit shifts can reorder the leaderboard and that architecture choices may hinge on a few decimals.
  • Two-axis variability: Two lotteries govern FID variance: training produces the network, while generation samples from that fixed network.Training randomness includes initialization, data ordering, and per-step loss noise; generation begins by drawing initial sampler noise.
  • Two-axis variability: An N×K panel of FID evaluations measures variability across N independent training runs and K sampling seeds per run.The panel makes the training and generation axes directly comparable.
  • Main findings: 3.2× more FID movement comes from retraining than from resampling, while relative noise remains stable across model sizes and training budgets.The study treats FID as a random variable over training and sampling seeds and measures it across several hundred SiT networks.
  • Main findings: Up to 2× less compute lets a lucky training seed reach the same FID as an unlucky one.This finding is reported as the luck-of-the-draw effect of training seeds.

2 Related Work

Prior work establishes that FID is widely used but fragile, while evaluation outcomes can vary across random seeds and methodological choices. This paper connects those concerns to generative-model scaling by measuring the residual seed variance that persists within a fixed scale.

  • FID and its limitations: FID has displaced Inception Score, but finite-sample bias, sampling-noise ranking flips, and preprocessing artifacts expose its fragility.FID compares Gaussian moments of Inception-V3 features; prior work proposes FID∞ to address finite-sample bias.
  • Reproducibility and statistical methodology: Nominally identical machine-learning runs can diverge across seeds, motivating stronger reporting and statistical methodology.This reproducibility concern spans reinforcement learning, NLP, language modelling, and broader evaluation critiques.
  • Why runs differ: Initialization, batch order, adaptive optimizers, and stochastic-gradient dynamics can send training through diverse basins, producing run-to-run differences.Prior theory links these effects to lottery-ticket and loss-landscape views, while architecture changes basin geometry rather than multiplicity.
  • Why runs differ: Near-identical noise-to-image maps can still exhibit percent-level FID fluctuations, extending reproducibility concerns to generative models.The cited work also reports diffusion models as unusually well-behaved at the function level.
  • Generative reproducibility, scaling, and seeds: Generative variance studies remain rare, and scaling narratives often treat seed variance as a vanishing residual rather than measuring what remains within a scale.Related work includes equalised-budget GAN comparisons, ImageNet-only protocol design, and scaling laws for diffusion transformers.

3 Experimental Setup

The experiments evaluate SiT models on class-conditional ImageNet 256×256 using an N×K panel of training and sampling seeds. Three nested statistics separate fixed-model generation variability, training-seed variability, and scale-normalized spread.

  • Experimental Setting: All experiments train SiT models at four widths on class-conditional ImageNet 256×256 with conditional flow matching and Inception-V3-space FID.The loss redraws Gaussian ε ∼ N(0, I) at every gradient step, treating this persistent noise as a training-time random-number generator.
  • The two-axis panel: Each experiment forms an N × K panel of N independently trained models, each evaluated under K generation seeds.Training seeds control parameter initialization, data-loader order, and per-step flow-matching noise; sampling seeds control generation-time initial noise.
  • Notation: σwithin measures fixed-model generation noise, while σbetween measures training-seed spread across per-seed mean FIDs.σwithin is computed across K sampling evaluations for each training seed and averaged over N seeds; σbetween is computed across the N per-seed means.
  • Notation: CoV = σ/µ (%) provides a dimensionless spread measure comparable across panels with absolute FID differing by an order of magnitude.The passage gives unguided versus guided panels as an example of differing absolute FID scales.

4 Experiments

FID varies more across retraining seeds than across sampling seeds, chiefly because of training noise, initialization, and data order. Guidance tuning reduces relative spread but reshuffles seed rankings, while compute and model scale leave a persistent 1–2% variability floor and lucky seeds can match unlucky seeds with less training.

  • Training versus generation lottery: 3.2×: retraining changes FID more than resampling, with σbetween =0.438 (CoV ≈1.3%) versus σwithin =0.137 (CoV ≈0.4%).Per-seed mean Inception FIDs span 33.75 →35.42 around a grand mean of 34.74.
  • Sources of training variance: 0.438: varying all training factors produces the largest between-seed spread, followed by noise (0.336), initialization (0.294), and data order (0.221).Noise alone reproduces 77% of baseline σ, initialization 67%, and data order 51%; the sources overlap rather than sum.
  • Determinism control: 0.047: fixing initialization, data order, and training noise across 24 retrains leaves multi-GPU non-determinism with much smaller FID variation than ordinary seed changes.Although EMA weights diverge by ≈5–6% of their norm, FID barely moves.
  • Guidance tuning: 1.26% →0.67%: per-seed CFG tuning nearly halves relative FID spread but changes which training seeds rank best, with Spearman ρ=0.73.GS-FID has σbetween =0.050 and σwithin =0.027, while 8/25 seeds move by at least five ranking places.
  • Compute and scale: [0.74%, 2.06%]: the FID CoV remains within this range across 76 compute-and-model cells, with median 1.30%.At 2M steps, CoV is non-monotonic across scale: SiT-S (0.74%), SiT-B (1.24%), SiT-XL (1.42%), and SiT-L (1.72%).
  • Seed selection and compute: 1.25×–2.0×: lucky training seeds reach the unlucky seed’s 2M-step target this much faster, depending on architecture.The speedups are 1.25× on SiT-S/B, 1.82× on SiT-L, and 2.0× on SiT-XL; N =5 lowers the detectable gap to ≈0.25 FID.

5 Discussion

The study calibrates its ≈1.3% FID variability floor only for SiT, flow matching, class-conditional ImageNet 256×256, and Inception-V3 FID, while proposing broader validation of the evaluation protocol. Its findings also highlight both compute savings and the risk that extra seeds can manufacture headline improvements.

  • Limitations: The ≈1.3% CoV is a calibration target for SiT, flow matching, class-conditional ImageNet 256×256, and Inception-V3 FID—not a universal constant.Other backbones, objectives, latent or pixel diffusion, text-to-image, and Fréchet variants may have different floors.
  • Limitations: Fidelity metrics track Inception FID closely, while recall is the outlier.Appendix F replicates the analyses using DINOv2 FID and Inception precision, recall, density, and coverage.
  • Limitations: The panel contains 20–25 training seeds and 10 sampling seeds, and production-scale behaviour beyond SiT-XL or 2M steps remains an extrapolation.These finite-panel and scale limits constrain how broadly the reported variability can be generalized.
  • Outlook: The proposed protocol uses a two-axis panel of training and sampling seeds summarized by a per-source variance decomposition.The paper establishes this approach for one combination and leaves extension to other methods, model families, and Fréchet-style metrics to future work.
  • Broader impact: The ≈1.3% floor can save compute by avoiding sub-floor retraining, but extra seeds can also manufacture headline improvements that should be priced into peer review.The paper explicitly frames this dual-use temptation in Sec. 4.5.

Supplementary Material · A Overview

The supplementary material extends the main paper with further seed-lottery analyses, a theoretical justification of golden-section search, and metric-robustness stress tests. These tests preserve the main asymmetries while revealing stronger lucky-seed speedups on several metrics and an inversion for Inception recall.

  • A Overview: The supplement extends the study along three axes: seed-lottery analyses, golden-section-search theory, and metric-robustness stress tests.The stress tests replicate the main-paper claims under DINOv2 FID and four Inception PRDC metrics.
  • A Overview: The cross-metric stress test sharpens rather than softens the main paper’s conclusions.It is presented as a replication of Appendix F across alternative metrics.
  • A Overview: Training randomness still dominates generation randomness on every fidelity-axis metric.This preserves the central training-versus-generation lottery asymmetry beyond Inception FID.
  • A Overview: The noise > init > data hierarchy from Sec. 4.2 holds verbatim across the stress-tested metrics.The passage explicitly states that this hierarchy remains unchanged.
  • A Overview: A 2–3× lucky-seed speedup appears for SiT-L/XL on precision, density, or coverage.The passage says this speedup grows from the main-paper result when the benchmark moves to these metrics.
  • A Overview: Inception recall is the lone metric that reverses the training-versus-generation asymmetry.The cited passage identifies this inversion but truncates before giving its detailed explanation.
  • A Overview: The appendices provide configurations, consolidated headline tables, and five additional Inception-FID analyses.These include source-decomposition violins, rank-instability analysis, and a 10×15 factorial initialization-seed test.
  • A Overview: Figure 9 literalizes the 3.2× asymmetry as two slot machines: training randomness produces one network, while ten sampling seeds produce ten FIDs.The left machine spins initialization, data order, and per-step noise; the right machine evaluates ten sampling seeds for that network.

B Per-experiment configurations … D.1 Per-train-seed violin panels by variability source

The study measures FID variability across training and sampling seeds using repeated SiT experiments, then decomposes its sources and examines how it changes with compute, scale, and evaluation protocol. Supplementary panels show that training-time randomness mainly widens between-seed spread, while generation noise remains stable across conditions.

  • B Per-experiment configurations: The default evaluation uses K = 10 sampling seeds per model, Inception-V3 features, 50 000 samples, ImageNet reference statistics, and a deterministic ODE sampler.The appendix specifies the shared FID pipeline and per-experiment configuration fields.
  • B Per-experiment configurations: N ≈25 independent training runs per cell populate the two-axis seed panel, with approximately 100 000 H100 GPU-hours used across reported and preliminary experiments.Training follows the SiT recipe on class-conditional ImageNet 256×256.
  • B Per-experiment configurations: Each single-source ablation varies one training-time RNG across 24–25 SiT-B/2 runs while holding the other two fixed; VARY-ALL is the fully-random 25×10 baseline.The varied sources are parameter initialization, data-loader order, and per-step flow-matching noise.
  • C Companion summary tables: The appendix consolidates headline numbers from the main figures and adds analyses of aggregation choice, initialization transferability, µP learning rates, and expected seed-induced confidence intervals.These analyses span Sections D.1–D.5 and support the paper’s seed-lottery argument.
  • D Additional analyses on Inception FID: Mean FID drops by ≈1.7–2× while σbetween shrinks 1.7–2.4× over 1.8M extra steps, keeping CoVbetween near a 1–2% band.The scaling sweep covers 76 model-step cells across SiT-S/B/L/XL.
  • D.1 Per-train-seed violin panels by variability source: The within-seed σwithin remains in [0.137, 0.152] across conditions, indicating that training randomness changes between-seed spread without changing evaluation jitter.The generation lottery is set by converged SiT-B/2 weights rather than by the training randomness that produced them.
  • D.1 Per-train-seed violin panels by variability source: σbetween declines from 0.438 when all sources vary to 0.336 for noise, 0.294 for initialization, and 0.221 for data, while violin heights stay essentially fixed.The mean-tick range falls from ≈1.7 FID to under ≈0.9 FID, and no panel shows a heavy-tailed minority of training seeds.

D.2 Summary statistics reshuffle the training-seed ranking … E.2 Unimodality of FID(CFG) under a Gaussian feature model

The paper shows that FID-based rankings and uncertainty depend strongly on how sampling and training seeds are summarized, while no initialization seed is universally best. It also establishes a stable seed-variance floor and justifies efficient CFG optimization under a Gaussian feature model.

  • D.2 Summary statistics reshuffle the training-seed ranking: Mean, minimum, and maximum FID across sampling seeds produce dense rank crossings among the 25 training seeds, so best-case and average-case benchmarks measure different quantities.Inception and DINOv2 nevertheless agree on the rankings under each criterion, with Spearman ρ≥0.94.
  • D.3 Are “good” init seeds universal?: 0.78 Inception FID: after averaging across 15 data-noise pairings, the 10 initialization grand means span [34.49, 35.27], making initialization seeds nearly interchangeable.The median per-cell σ is 0.149, close to 0.137 in the 25×10 baseline panel.
  • D.3 Are “good” init seeds universal?: 0.41 Kendall’s W: initialization rankings are only weakly concordant across data-noise pairings, and most seeds visit both the top and bottom three.The best grand-mean initialization is top-1 in only 5 of 15 pairings, while the worst is bottom-1 in 8 of 15.
  • D.4 Detailed numbers for the µP sweep: 1–3%: across central well-conditioned learning rates, the between-seed CoV remains within this band across model sizes under both GS-FID and unguided FID.At 5×10−4, GS-FID CoV rises an order of magnitude on SiT-B, 3–4× on SiT-L and SiT-XL, while SiT-S diverges on three seeds.
  • D.4 Detailed numbers for the µP sweep: 7.7–11.5% worse: selecting learning rates by unguided FID yields higher GS-FID than the GS optimum and a 5–12× wider between-seed envelope on SiT-B/L.The unguided argmin is 5×10−4 for every size, whereas the modal GS argmin is 2.3–3×10−4.
  • D.5 What CI should I expect at my reported FID?: 1.30%: the median between-seed CoV across 76 model-step cells gives a seed-only 95% CI of ±0.38 FID at F≈15 for one trained model.At F≈35 the CI is ±0.89, and reducing uncertainty below ±0.1 at F≈15 requires N≥25 training seeds.
  • E.1 Convergence: 12 FID evaluations: with CFG bracket [1] [2] and tolerance ε=0.01, golden-section search theoretically requires approximately 12 evaluations, versus a measured median of 14.The excess comes from an implementation safeguard when the bracket width straddles the tolerance.
  • E.2 Unimodality of FID(CFG) under a Gaussian feature model: Strictly convex quadratic: under equal-covariance Gaussian conditional and unconditional feature distributions, FID(CFG) has a unique global minimum and is strictly unimodal on every interval.The empirical optimum is w⋆≈0.027, while all 250 searches converged to a tight cluster with σw⋆=0.045, consistent with global unimodality beyond equal covariance.

E.3 Noise robustness · F Replication across DINOv2 FID and Inception PRDC · F.1 Metric definitions

FID-based evaluation is noisy, making CFG optimization vulnerable to sampling-induced errors; the appendix therefore replicates seed-lottery analyses across DINOv2 FID and Inception PRDC while distinguishing their feature spaces and anchor geometries. These metrics differ in scale, bounds, and whether their k-NN balls are anchored on real or generated features.

  • E.3 Noise robustness: FID evaluations are stochastic, so golden-section search minimizes a noisy objective rather than the true FID objective.The noisy objective is written as ˜f(w) = f(w) + η(w).
  • E.3 Noise robustness: Near the optimum, symmetric probes have an O(δ3) true difference that can be dominated by noise, causing the algorithm to retain the wrong half-bracket.This regime occurs when δ≲(2σy/f ′′(w⋆))1/2, and the deterministic residual contributes the ε2 term.
  • E.3 Noise robustness: 0.17 is the predicted noise-induced spread versus 0.045 observed σw⋆, suggesting the FID landscape near optimal CFG is sharper than the back-of-envelope curvature estimate.The prediction uses σy ≈0.137 FID and f ′′(w⋆) ≈O(10).
  • F Replication across DINOv2 FID and Inception PRDC: The appendix re-runs all five seed-lottery questions using DINOv2 FID plus Inception precision, recall, density, and coverage.It reuses the main paper’s panels, with appendix table columns paralleling the main-paper tables row-for-row.
  • F.1 Metric definitions: The evaluation suite contains six metrics: Fréchet distances in Inception-V3 and DINOv2 feature spaces, plus four PRDC metrics comparing feature sets through k-nearest-neighbour balls.Each evaluation uses the same 50k real ImageNet training images and 50k generated samples from a trained SiT.
  • F.1 Metric definitions: DINOv2 FID is roughly an order of magnitude larger than Inception FID at matched generative quality, so their absolute values are not comparable.Both metrics are unbounded above and lower is better; each fits a Gaussian to its two feature sets.
  • F.1 Metric definitions: Recall is the only generated-anchored PRDC metric; precision, density, and coverage use fixed real-anchored ball geometry across sampling evaluations.Precision and density address fidelity, while recall and coverage address diversity; precision, recall, and coverage lie in [0, 1], whereas density is non-negative and unbounded above.

F.2 The training lottery dominates for every fidelity metric, not for recall

The training lottery dominates DINOv2 FID even more strongly than Inception FID, while precision, density, and coverage show near parity between training and sampling variability. Recall reverses the pattern because generated-anchored k-NN geometry changes with each sampling seed.

  • FID: 4.82× is the DINOv2 FID between-to-within ratio, up from 3.19× for Inception FID.The relative CoVbetween floor tightens from 1.26% to 0.86%.
  • PRDC metrics: 1.14, 1.41, and 1.12 are the precision, density, and coverage ratios, respectively, placing the lotteries near parity.Their sampling-only confidence intervals cover roughly half of the true seed-induced envelope.
  • Recall: 0.28× is the Inception recall between-to-within ratio, meaning within-seed sampling σ exceeds between-seed training σ by over threefold.Recall is the only metric for which the sampling-seed CI is the appropriate fixed-model interval.
  • Why recall inverts: Generated-anchored k-NN balls make recall invert the asymmetry because every sampling seed redraws the ball geometry, causing correlated indicator flips across real points.The effective sample size is much smaller than 50k, unlike real-anchored precision, density, and coverage.
  • Empirical check: Recall’s within-seed σ is roughly twice the binomial floor, while precision and coverage are at or below that floor.Only recall is generated-anchored and sits well above the binomial estimator floor.
  • Recall: µ=0.314 is the recall-axis value on the same panel, where similar coverage keeps between-seed σ small while correlated-ball geometry inflates within-seed σ.Together these effects produce the denominator inversion.

F.3 Flow-matching noise dominates for the fidelity metrics · F.4 Golden-section search transfers cleanly to DINOv2 FID

Flow-matching noise is the dominant reproducible source of variance for FID and other well-behaved fidelity metrics, while recall cannot be decomposed reliably. Inception-FID guidance does not improve DINOv2-FID variability but substantially shifts PRDC metrics toward a high-fidelity, low-recall operating point.

  • F.3 Flow-matching noise dominates for the fidelity metrics: Flow-matching noise dominates initialization and data order across Inception FID, DINOv2 FID, precision, density, and coverage.The ranking is VARY-NOISE > VARY-INIT > VARY-DATA, with noise alone reproducing about three-quarters of baseline σ.
  • F.3 Flow-matching noise dominates for the fidelity metrics: Adding single-source variances in quadrature overshoots the fully stochastic baseline by 7–19%, indicating a sub-additive combination.The same sub-additive pattern carries over to DINOv2 FID and Inception precision, density, and coverage.
  • F.3 Flow-matching noise dominates for the fidelity metrics: Inception Precision has comparable training and sampling spreads, with σbetween/σwithin =1.14×.Its per-seed means span 0.480 to 0.491 around a grand mean of 0.485; a multi-sampling-seed confidence interval covers roughly half the seed-induced envelope.
  • F.3 Flow-matching noise dominates for the fidelity metrics: Inception Recall fails variance decomposition: three single-source conditions reach 112–126% of the vary-all baseline, while the naive sum-of-squares overshoots by 90%.Between-seed variation is smaller than within-seed sampling variation, so condition-specific estimates are dominated by sampling jitter.
  • F.4 Golden-section search transfers cleanly to DINOv2 FID: DINOv2 FID’s relative noise floor is 0.86% unguided and 0.94% at the Inception-FID-optimal CFG, so the CoV-halving claim is specific to Inception FID.These values are statistically indistinguishable, and the unguided floor already lies below the 1–2% band.
  • F.4 Golden-section search transfers cleanly to DINOv2 FID: Inception-FID golden-section search reduces Inception FID CoVbetween from 1.26% to 0.67% but does not tighten DINOv2-FID variability.The approximately 14 extra FID evaluations per cell therefore do not buy a tighter DINOv2-FID estimate.
  • F.4 Golden-section search transfers cleanly to DINOv2 FID: Guidance shifts PRDC metrics to a high-fidelity, low-recall operating point: precision, density, and coverage roughly double or triple, while recall drops by more than half.The supplemental metrics are readouts at the Inception-FID-optimal CFG, not separately optimized.

F.5 The relative noise floor is metric-dependent but scale-invariant

The relative FID noise floor is metric-dependent but largely scale-invariant: most metrics remain within a 1–2% CoV band across model sizes, while some fidelity metrics tighten substantially at scale. DINOv2 FID is consistently tighter than Inception FID, whereas recall does not improve with compute.

  • Metric-dependent noise floors: 1–2% CoV covers DINOv2 FID, Inception precision, density, and coverage across every checkpoint and model size.Recall remains within a 1–1.8% CoV band on SiT-B/L without shrinking with compute.
  • Metric-dependent noise floors: DINOv2 FID is tighter than Inception FID at every scale and checkpoint past 200k steps.The DINOv2 FID band sits inside the Inception FID band across all 76 cells, with similar σ-shrink factors.
  • Metric-dependent noise floors: Below 0.4% CoV, Inception precision, density, and coverage on SiT-XL settle by 1M steps.Median precision and coverage CoVs fall below 0.3%, with maxima below 0.7%, producing the tightest seed-induced spread.
  • Metric-dependent noise floors: 1–1.8% recall CoV persists as a metric property: recall spread grows on SiT-B/L, stays flat on SiT-XL, and does not respond to compute.Recall ranking stability is also weak, with Spearman ρ below 0.6 before the final checkpoint across model sizes.
  • Distribution shape: [2.9, 5.5] bounds (max−min)/σ across all 76 cells for every metric, matching the 3.5–3.9 Gaussian expectation.The seed-induced spread is therefore Gaussian-shaped rather than driven by heavy-tailed training failures.

F.6 Cherry-picking saves 1.2–2.9× on every well-behaved metric

Cherry-picking lucky training seeds reduces compute needed to match unlucky-seed targets by 1.2–2.9× across well-behaved fidelity metrics, with the largest savings on larger models’ precision, density, and coverage. Recall is compute-invariant, so its nominal speedups are degenerate and the lucky-seed framing does not apply.

  • F.6 Cherry-picking saves 1.2–2.9× on every well-behaved metric: The speedup generalizes across DINOv2 FID, Inception precision, density, and coverage, while recall remains outside the applicable framing.The largest savings are reported on the fidelity PRDC metrics; recall is explicitly excluded because it is compute-invariant.
  • F.6 Cherry-picking saves 1.2–2.9× on every well-behaved metric: 1.2–1.67× DINOv2 FID speedups match Inception FID’s 1.18–1.82× band, showing the lucky-seed effect is not feature-extractor-specific.The speedup compares 2M training steps with the first checkpoint where the lucky seed matches the unlucky seed’s 2M target.
  • F.6 Cherry-picking saves 1.2–2.9× on every well-behaved metric: 2.0–2.9× speedups occur for Inception precision, density, and coverage on SiT-L/XL, exceeding the FID savings on larger models.The direction is flipped for higher-better metrics so “lucky” identifies the preferred seed.
  • F.6 Cherry-picking saves 1.2–2.9× on every well-behaved metric: Recall yields nominal 6–10× speedups only because lucky seeds already exceed the unlucky-seed-at-2M target at 200k–300k.These entries are degenerate because Inception recall is approximately compute-invariant on this panel, with σ-shrink near or below 1.

G What does FID look like?

Appendix G uses fixed classes and initial-noise tensors to visualize how generated appearance varies across FID-ordered checkpoints, with and without classifier-free guidance. Unguided samples degrade smoothly as FID worsens, whereas guided samples remain visually clean and recognisable even at the highest FID.

  • Setup: 10 ImageNet classes and 10 fixed initial-noise tensors are decoded at 16 FID levels using a 50-step flow_euler_sampler.The unguided levels span FID∈[13.5, 87.4].
  • What the galleries show: Fixed initial noise isolates training-time and guidance changes in the rendered images.The galleries compare guided and unguided panels while holding each class–noise pair constant across columns.
  • What the galleries show: Unguided image quality degrades monotonically along the FID axis, from well-formed objects to textureless blobs and noisy patches.The samples remain class-correct before progressing toward increasingly noisy appearances.
  • What the galleries show: Guided panels show visually clean, recognisable samples even for the highest-FID checkpoint.This contrasts with the monotonic visual degradation observed without guidance.
Loading 2606.20536v1…