PaperlayerView exact revision on arXiv

Source-linked AI summary

Visualizing Distribution Coverage in Generative Diffusion Models

Yifei Wang, Xiaoyu Wu, Tsu-Jui Fu, Chen Chen, Liang-Chieh Chen, Zhe Gan, Chen Wei

arXiv:2609.38853v1cs.LG

TL;DR

Single-sample evaluations may not reveal whether few-step diffusion models preserve their teachers’ distribution coverage. The paper uses pass@k to examine performance across sampling budgets and finds that coverage depends on the distillation objective in both image generation and few-step causal video generation.

  • Problem

    Typical single-sample evaluations may not reveal whether distilled models preserve teacher distribution coverage beyond one draw.

    Source: §1 · p. 1

  • Method

    The paper applies pass@k to benchmark success across sampling budgets, validating the diagnostic on CFG before studying few-step distillation.

    Source: §1 · p. 2

  • Results

    Distribution-matching objectives improve early-hit performance at the cost of large-budget coverage, while consistency- and trajectory-based objectives better preserve teacher coverage across image and causal video generation.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Pass@k reveals coverage differences that single-draw performance alone does not show.

    Source: §1 · p. 2

    Pass@k measures access to successful generations under a specified benchmark criterion, not every form of perceptual or semantic diversity, and large sample pools limit the models and configurations evaluated.

    Source: §7 · p. 9

Abstract

from arXiv · show

Diffusion distillation is widely adopted to accelerate sampling, and the resulting few-step models are broadly believed to match or even surpass their multi-step teachers in generation. However, standard evaluations such as GenEval2 typically draw only one sample per prompt, so improved scores may fail to reveal losses in distribution coverage. We therefore revisit whether distilled models truly match their teachers beyond single-draw performance using \textbf{pass@$\mathbf{k}$}, which measures the probability that at least one of $k$ independent samples satisfies a quality criterion. At $k{=}1$, pass@$k$ reduces to standard single-draw evaluation. As $k$ grows, the curve reveals whether additional draws find genuinely different successes or merely revisit the same modes, directly exposing how broadly a model covers the space of valid outputs. We first show that classifier-free guidance (CFG), whose quality--coverage tradeoff is well established, is the clearest case: higher guidance improves pass@$1$, but its advantage shrinks and reverses at larger $k$. Applying pass@$k$ to few-step distilled models, we find the same tradeoff splits along training objectives: distribution-matching objectives concentrate the student's output distribution, boosting early-hit rates while eroding large-budget coverage, whereas consistency and trajectory-based objectives better preserve the teacher's coverage even at large $k$. We further show that this tradeoff extends to few-step causal video generation. Our findings reveal a previously overlooked cost of diffusion distillation: across both image and video generation, the choice of training objective fundamentally determines whether a few-step model inherits its teacher's distribution coverage or trades it away for single-draw quality.

1 INTRODUCTION

Single-sample benchmarks can miss whether few-step diffusion students preserve the range of valid outputs their teachers produce. Using pass@k, the paper finds that coverage outcomes depend on the distillation objective and that the same pattern appears in few-step causal video generation.

1 INTRODUCTION

  • Single-sample benchmark scores can conceal narrower student distributions, even when distilled models match or exceed teacher scores on compositional tasks.The introduction illustrates this with 256 samples: the few-step student has lower pixelwise standard deviation than its teacher.Source: §1 · p. 1, Figure 1 · p. 2
  • Pass@k tracks benchmark success as the sampling budget grows, revealing whether a small-k advantage shrinks or reverses at larger k.At k=1 it reports mean one-draw success probability; the curve provides nonlinear summaries as k increases.Source: §1 · p. 2
  • Higher classifier-free guidance improves pass@1, but its advantage shrinks and eventually reverses at large k.The paper uses CFG as a sanity check because its quality–coverage tradeoff is already established (Ho & Salimans, 2022).Source: §1 · p. 2
  • Across distillation families, distribution-matching objectives improve early-hit performance at the expense of large-budget coverage, while consistency- and trajectory-based objectives better preserve teacher coverage.The paper also reports reduced within-prompt semantic diversity alongside distribution contraction.Source: §1 · pp. 2–3
  • Few-step causal video generation shows the same objective-dependent pattern: distribution-matching methods have diminishing gains as the budget grows, while consistency-style alternatives better preserve coverage.The paper reports this extension for causal video generation (Huang et al., 2025; Zhao et al., 2026b).Source: §1 · p. 2

2 RELATED WORK

Prior text-to-image benchmarks assess distinct kinds of prompt following, but their usual single-sample or aggregate scores do not show how success is distributed across prompts as sampling grows. Prior coverage metrics offer complementary diagnostics, while this paper uses prompt-level verifiers to assess benchmark success.

2 RELATED WORK

  • Text-to-image benchmarks target different capabilities, including compositional correctness, fine-grained atomic evaluation, dense prompt following, semantic instruction following, and disentangled compositional skills.Examples include GenEval (Ghosh et al., 2023), GenEval2 (Kamath et al., 2025), DPG-Bench (Hu et al., 2024), GenAI-Bench, and T2I-CompBench.Source: §2 · p. 3
  • Unlike typical single-sample or aggregated benchmark scores, pass@k reveals how success probability is distributed across instances as the sampling budget grows.Existing diversity and distribution metrics, including Vendi score (Friedman & Dieng, 2022), precision–recall (Kynkäänniemi et al., 2019), and FID (Heusel et al., 2017), are not directly tied to success under a particular prompt verifier.Source: §2 · p. 3
  • The paper pairs prompt-level verifiers with CLIP-space analysis to assess benchmark-conditioned coverage and within-prompt semantic diversity.The authors note that representation-based distributional metrics can be directly optimized during training, limiting their reliability as independent evaluators (Yang et al., 2026).Source: §2 · p. 3

3 PASS@k AS A LENS FOR DISTRIBUTION COVERAGE

The paper adapts pass@k and best-of-k to benchmark-scored image generation so performance can be examined across sampling budgets rather than at a single draw. These statistics use independent samples per prompt and fixed-budget comparisons to characterize access to successful generations.

3 PASS@k AS A LENS FOR DISTRIBUTION COVERAGE

  • Pass@k estimates the probability that at least one of k independent samples passes a benchmark evaluator, averaged across benchmark instances.Instances follow each benchmark’s protocol and can be a prompt or a prompt–category pair; the estimator uses n≥k samples per instance.Source: §3 · p. 3
  • Pass@1 retains mean success probability, while the pass@k curve summarizes how success probability is distributed across instances as k grows.The estimator corresponds to the probability that a uniformly random size-k subset of the sample pool contains a success.Source: §3 · p. 3
  • For continuous evaluators, best-of-k measures the expected maximum score among k independent generations and probes access to upper-tail scores.For a fixed pool of independent, identically distributed samples, the described estimator is unbiased and reduces to pass@k for binary scores.Source: §3 · p. 3
  • Experiments use independent samples, matched prompts and evaluators, and fixed sampling budgets; k=1 recovers single-sample performance.The authors base claims on moderate-budget crossovers and report checking them against an FP-immune variant and evaluator false-positive calibration.Source: §3 · p. 4

4 PASS@k PROFILES DISTRIBUTION COVERAGE UNDER CFG

The CFG analysis tests whether pass@k distinguishes behaviors that single-sample evaluation cannot. Stronger guidance improves early success but may lose its advantage as the sampling budget increases.

4 PASS@k PROFILES DISTRIBUTION COVERAGE UNDER CFG

  • Fig. 2 compares best-of-k GenEval2 Soft-TIFA GM for CFG values 1, 3.5, and 7, and shows per-prompt success-rate distributions over 256 samples.Its per-prompt success cutoff is Soft-TIFA GM≥0.3, described as a lenient threshold crediting mostly-correct samples.Source: Figure 2 · p. 4
  • Higher CFG improves pass@1, but its advantage shrinks and eventually reverses at large k, demonstrating a CFG quality–coverage tradeoff.The paper frames CFG as a controlled demonstration that pass@k separates behaviors indistinguishable at k=1, not as evidence that guidance is harmful.Source: §4 · p. 5
  • High CFG shifts prompt-level success rates toward both extremes, leaving more prompts unsolved while solving more prompts at high rates.The paper qualifies the result as benchmark-conditioned coverage; perceptual artifacts at extreme guidance may be rejected by a VQA evaluator.Source: §4 · p. 5

5 DISTRIBUTION COVERAGE IN FEW-STEP DISTILLATION

The section asks whether few-step students preserve their teachers’ conditional output coverage, distinguishing distribution-matching objectives from consistency and trajectory-based methods. Across model families, students often lead at small sampling budgets but fall behind at large budgets; a controlled FLUX study associates these differences with training objectives and diversity.

5 DISTRIBUTION COVERAGE IN FEW-STEP DISTILLATION

  • The section groups distillation into distribution matching and consistency distillation; consistency methods train the student to predict the same trajectory endpoint from different states.Distribution matching reduces a discrepancy between teacher and student conditional output distributions, without requiring reproduction of a specific teacher sampling trajectory.Source: §5 · pp. 5–6

5.1 COVERAGE CONTRACTION AND CONDITIONAL MODE COLLAPSE

  • Students lead their teachers at k=1 but the teacher overtakes in 16 of 20 comparisons; LCM-SDXL is the exception, remaining ahead across all four benchmarks.For SD3.5-Large and Turbo, teacher samples had 2.5–3.4 times larger mean pairwise CLIP distance on three prompts both models solved perfectly.Source: §5.1 · p. 6, Figure 4 · p. 6

5.2 WHAT IN DISTILLATION CAUSES THE COLLAPSE? A CONTROLLED LOSS ABLATION

  • In a controlled FLUX.1-dev comparison, DMD2 coverage improves early then declines, ADD peaks early below the teacher, and MeanFlow improves throughout training to finish above it.Higher guidance amplifies DMD2’s late-training decline, while the DMD2–MeanFlow contrast remains under shared standard guidance.Source: §5.2 · p. 8
  • At the shared guidance scale, DMD2 loses coverage and diversity during later training while MeanFlow preserves diversity and improves coverage, linking the objective difference to their distinct trajectories.The paper describes distribution-matching and adversarial objectives as placing pressure on probability mass to concentrate in high-density regions, whereas MeanFlow learns a shortened teacher trajectory without directly reshaping its output distribution.Source: §5.2 · p. 8
  • Along the same training trajectories, DMD2’s declining coverage accompanies sharp CLIP-space contraction, while MeanFlow preserves near-constant diversity as coverage improves.ADD’s diversity remains relatively stable but substantially below the teacher’s, indicating a persistent deficit rather than progressive contraction during the observed training period.Source: Figure 6 · p. 7, §5.2 · p. 8

6 DISTRIBUTION COVERAGE IN VIDEO FORCING MODELS

In few-step video generation, distribution-matching methods initially outperform Wan but lose their advantage as the sampling budget grows, while consistency and trajectory-based methods show better coverage retention. The evaluation uses VBench-20 best-of-k curves over 64 samples per dimension prompt.

6 DISTRIBUTION COVERAGE IN VIDEO FORCING MODELS

  • Self-forcing is distribution-matching, while causal forcing includes both distribution-matching and consistency-distillation variants.This setup allows comparison of objective types against the Wan teacher while retaining causal rollout structure.Source: §6 · p. 8
  • VBench-20 evaluation uses 20 prompts for each of 16 dimensions and 64 independently sampled videos per dimension prompt.Each curve point is the exact expected maximum over uniformly chosen size-k subsets; selection occurs separately within each dimension.Source: §6 · p. 8
  • Fig. 7 plots VBench-20 best-of-k curves, with lower rows showing differences from Wan; positive differences favor Wan.Solid curves mark Self-Forcing DMD and Causal-Forcing, dash-dot marks Causal-CD, and dashed curves mark multi-step baselines.Source: Figure 7 · p. 9
  • Self-Forcing DMD and Causal-Forcing initially outperform Wan, but their advantage quickly reverses as the sampling budget grows.The reversal appears in both quality and semantic best-of-k curves.Source: §6 · p. 9
  • Trajectory-based consistency distillation and multi-step AR-Diffusion show semantic scores tending to catch up as more samples are drawn, despite quality scores remaining below Wan.The authors suggest these approaches may better preserve semantic diversity even when visual quality lags.Source: §6 · p. 9

7 CONCLUSION

The paper uses pass@k to examine whether few-step students inherit teacher coverage and finds that outcomes depend on distillation objectives across image and video generation. It notes that the metric is benchmark-relative and that large sample pools constrain evaluation scale.

7 CONCLUSION

  • Pass@k reveals an objective-dependent coverage tradeoff: distribution-matching and adversarial losses concentrate outputs, while consistency and trajectory-based losses better preserve teacher coverage.The same divide is reported for few-step causal video generation as for images.Source: §7 · p. 9
  • Pass@k measures access to successful generations under a specified evaluation criterion, not every form of perceptual or semantic diversity.Large sample pools also limit how many models and training configurations can be evaluated, particularly proprietary systems.Source: §7 · p. 9

A ADDITIONAL CFG RESULTS

Across CFG sweeps, stronger guidance usually improves small-budget performance, but its advantage narrows and often reverses as k increases. Per-prompt diagnostics show that higher guidance shifts success toward prompts already solved reliably and leaves more prompts unsolved.

A.1 FULL GUIDANCE SWEEP

  • Stronger guidance usually improves small-budget performance, but its advantage narrows and often reverses as k increases.The clearest reversal is on GenAI-Bench; for FLUX.1-dev on GenEval2, CFG=3.5 remains ahead throughout the measured range.Source: §A.1 · p. 13, Figure 8 · p. 13

A.2 PER-PROMPT SUCCESS RATES

  • In five of six settings, CFG=1 leaves fewer prompts in the zero-success bin than CFG=7.Higher guidance shifts probability away from low-but-nonzero success prompts toward the extremes, leaving fewer prompts that can benefit from additional samples.Source: §A.2 · p. 13
  • GenEval2 per-prompt diagnostics use a deliberately lenient Soft-TIFA GM≥0.3 success cutoff to test whether a mostly-correct sample is ever produced.The passage says no quantitative claim depends on this binarization: best-of-k uses raw continuous scores, and Appendix D checks stricter cutoffs.Source: §A.2 · p. 13

B ADDITIONAL DISTILLATION RESULTS

Additional results show family-dependent teacher–student crossovers and prompt-level redistribution of successes. Students often gain repeated early successes on a subset of prompts, while teachers continue to recover prompts outside that set as k grows.

B ADDITIONAL DISTILLATION RESULTS

  • These results provide detailed crossover statistics and per-family diagnostics behind the paper's main distillation results.The section covers crossovers across families and prompt-level coverage diagnostics.Source: B ADDITIONAL DISTILLATION RESULTS · p. 13

B.1 CROSSOVERS ACROSS FAMILIES

  • Fig. 10 breaks aggregate crossovers into benchmark-specific subscores; its cells give the smallest k where the teacher matches or exceeds the student, with NC marking no crossover within N=256.Rows are model families and columns are benchmark tasks, atom counts, categories, or splits.Source: §B.1 · p. 13, Figure 10 · p. 14
  • For SD3.5-Turbo, Qwen-Image-Lightning, and Z-Image-Turbo, teacher crossovers occur across most subscores, while LCM-SDXL remains ahead on every aggregate benchmark.LCM-SDXL is overtaken only on selected subscores, consistent with better coverage preservation under consistency distillation.Source: §B.1 · p. 14, Figure 10 · p. 14

B.2 PROMPT-LEVEL COVERAGE DIAGNOSTICS

  • For SD3.5-Turbo, Sana-Sprint, Qwen-Image-Lightning, and Z-Image-Turbo, student-only successes at small budgets concentrate on prompts the student solves repeatedly, while the teacher later recovers other prompts.LCM-SDXL instead sustains gains across prompts and budgets, consistent with no aggregate crossover under consistency distillation.Source: §B.2 · p. 14, Figure 11 · p. 15, Figure 12 · p. 15

C.1 COVERAGE AT THE FINAL CHECKPOINT

At the final checkpoint, DMD2 and ADD lose their small-budget advantage over the teacher as the sampling budget grows, while MeanFlow remains ahead. Per-prompt success rates show that divergence-based students leave more prompts unsolved, whereas MeanFlow most closely tracks the teacher.

C.1 COVERAGE AT THE FINAL CHECKPOINT

  • LCM retains its advantage over SDXL across the measured range.Source: Figure 15 · p. 16
  • At guidance 3.5, DMD2 and ADD lead at small budgets but fall below the teacher at larger k, while MeanFlow stays above it across the measured range.At guidance 7, DMD2 falls below the teacher from k=8 and trails by more than 11 points at k=64; ADD falls below at k=32 and 64.Source: §C.1 · pp. 15–16, Figure 16 · p. 16

C.2 PER-PROMPT SUCCESS RATES

  • DMD2 and ADD shift success-rate mass toward both high-success and zero-success prompts, while MeanFlow most closely tracks the teacher and leaves the fewest prompts unsolved.The enlarged zero-success bin for DMD2 and ADD limits gains at large k, consistent with the teacher overtaking them.Source: §C.2 · p. 16

D EVALUATOR FALSE POSITIVES AND THE ROBUSTNESS OF PASS@k

The appendix tests whether evaluator false positives undermine pass@k comparisons. Crossovers and endpoint gaps survive an FP-immune re-analysis, although crossover budgets depend on the success threshold and false positives compress extreme upper-tail differences.

Evaluator false positives

  • At large budgets, false-positive mass makes an unsolved prompt’s observed pass@k contribution approach 1 − (1 − ε)^k rather than zero, compressing extreme upper-tail differences.The appendix reports that orderings and crossovers at moderate budgets are more robust than absolute values as k approaches n.Source: D EVALUATOR FALSE POSITIVES AND THE… · p. 17

Two FP diagnostics

  • The FP-immune statistic requires at least two successes, while continuous-evaluator comparisons also sweep the success threshold τ.A single false positive cannot satisfy the two-success requirement; the threshold sweep tests sensitivity to the cutoff.Source: D EVALUATOR FALSE POSITIVES AND THE… · p. 17

Robustness of the reported comparisons

  • Every reported crossover survives the FP-immune pass≥2 @k re-analysis, shifting by at most one budget doubling, and every endpoint gap is preserved.On GenEval, the gaps grow; Table 1 reports crossover budgets and k=256 gaps under both the standard statistic and pass≥2 @k.Source: Table 1 · p. 17, D EVALUATOR FALSE POSITIVES AND THE… · p. 17
  • The ordering of students leading at small budgets and teachers overtaking at larger ones is robust, but the exact crossover budget depends on the success threshold.At τ=0.7, crossovers occur earlier; at τ=0.9, upper-tail compression delays some crossovers toward the measured-budget edge.Source: D EVALUATOR FALSE POSITIVES AND THE… · pp. 17–18

Direct calibration of ε: a null-pair test

  • Measured false-positive rates cannot account for the teacher–student pass@256 gaps, even under the one-sided 95% upper bounds.The null-pair test estimates evaluator false positives using images paired with mismatched prompts; Soft-TIFA’s point estimate is 3.1×10^-4 and VQAScore’s is 0.Source: D EVALUATOR FALSE POSITIVES AND THE… · p. 18, Table 2 · p. 18

Related papers

Loading 2609.38853v1…