PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionBench: On Holistic Evaluation of Diffusion Transformers

Xingjian Leng, Jaskirat Singh, Zhanhao Liang, Ethan Smith, Martin Bell, Aninda Saha, Yuhui Yuan, Liang Zheng

arXiv:2606.24888v1cs.CV

TL;DR

ImageNet-only evaluation may not reveal whether DiT gains reflect broadly better modeling or benchmark overfitting. NanoGen unifies ImageNet and text-to-image training and evaluation, revealing that ImageNet rankings do not reliably predict text-to-image performance and motivating DiffusionBench.

  • Problem

    As DiT methods improve, it becomes increasingly unclear whether ImageNet gains indicate broadly better modeling or overfitting to that benchmark.

  • Method

    NanoGen unifies DiT training and evaluation across ImageNet and text-to-image tasks using shared components, with task changes limited to the dataset and conditioning module.

  • Results

    ImageNet rankings do not reliably predict text-to-image performance across 21 latent diffusion models evaluated under nearly identical settings.

  • Takeaways & Limitations

    DiffusionBench combines ImageNet and text-to-image evaluation, and the paper recommends reporting it rather than ImageNet alone.

  • Takeaways & Limitations

    The ImageNet–text-to-image correlation was measured at the scale and compute budget the authors could afford and may differ at other scales.

Abstract

from arXiv · show

Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: methods that improve DiffusionBench are more likely to reflect broader progress.

1 INTRODUCTION

DiT research has largely relied on class-conditional ImageNet evaluation, but ImageNet gains may not reflect broadly better generative modeling. DiffusionBench addresses this gap by pairing ImageNet with text-to-image evaluation through NanoGen, whose T2I setup requires roughly 12 configuration-line changes.

  • 1 INTRODUCTION: DiT research has converged on a narrow ImageNet-centered evaluation setup, making it increasingly difficult to distinguish broad modeling progress from benchmark overfitting.Text-to-image evaluation is often omitted because it is perceived to require different data pipelines, evaluation procedures, and codebases.
  • 1 INTRODUCTION: NanoGen matches state-of-the-art ImageNet methods and extends from ImageNet to text-to-image training with roughly 12 lines of configuration changes.The changes cover the dataset and conditioning module, challenging the premise that broader evaluation requires a separate high-cost research program.
  • 1 INTRODUCTION: ImageNet rankings do not reliably predict text-to-image performance, so improvements on ImageNet may fail to generalize to T2I generation.NanoGen trains 21 latent diffusion models under nearly identical ImageNet and T2I settings, finding no strong correlation between ImageNet FID and T2I metrics.
  • 1 INTRODUCTION: DiffusionBench combines ImageNet and text-to-image results across multiple metrics and is proposed as the default benchmark for DiT research.The authors argue that methods improving DiffusionBench are more likely to reflect broadly useful progress than methods evaluated on ImageNet alone.

2 THE NANOGEN TRAINING AND EVALUATION FRAMEWORK

NANOGEN provides a unified DiT codebase for class-conditional ImageNet and text-to-image generation, requiring only task-specific data and conditioning changes while sharing the backbone, training, and evaluation infrastructure. It supports reproducible ImageNet baselines and makes T2I evaluation a small configuration change.

  • Design principles: NANOGEN unifies ImageNet and T2I training and evaluation in one DiT framework, making cross-task method evaluation practical without re-engineering.The framework shares one backbone, optimiser, training loop, evaluation harness, and configuration format across tasks.
  • Design principles: Switching from ImageNet to T2I changes only the dataset and conditioning module, while the architecture, loss, optimiser, and EMA remain shared.ImageNet uses class-labelled data and a class embedder; T2I uses captioned images and a frozen text encoder.
  • Backbone architecture: The shared backbone uses DDT, removes encoder AdaLN, and represents timestep and task conditioning as encoder tokens, preserving an identical architecture across tasks.DDT splits encoding and decoding to increase effective width without quadratic FLOPs, while decoder modulation is computed from encoder semantics.
  • Task-specific conditioning tokens: ImageNet uses 4 timestep plus 8 class-conditioning tokens, whereas T2I uses 4 timestep plus 256 text-conditioning tokens.The number and meaning of conditioning tokens are the only task-specific architectural difference.
  • Evaluation protocols: The framework provides online evaluation with ImageNet metrics including FID, IS, FDr, and MIND, and T2I metrics including GenAIBench, DPGBench, and GenEval.Sampling defaults to Euler with 50 NFEs; ImageNet reports with and without CFG, while T2I reports with CFG.
  • ImageNet reproducibility validation: NANOGEN’s unified ImageNet implementation matches or improves published FID and IS across reproduced latent- and pixel-space methods.Six existing methods are retrained with comparable model sizes, inference NFEs, and CFG settings; Table 1 summarizes the reproducibility comparison.

3 DIFFUSIONBENCH: A HOLISTIC BENCHMARK

DiffusionBench evaluates DiTs on both ImageNet and text-to-image generation because strong ImageNet performance does not reliably predict T2I performance, despite broader method-category trends sometimes aligning. The benchmark is practical to run and is recommended over reporting either axis alone.

  • Latent-space generation: FID 1.37 is achieved by FLUX.2-VAE on ImageNet, while REPA-E VAE methods reach about 1.5–1.6 and the better RAE variants reach about 1.7–1.9.Among RAE variants, DINOv3-B has the best FID in the supplied comparison, although its exact value is not provided.
  • Pixel-space generation and MeanFlow: FID 6.60 and 5.40 are obtained by MeanFlow with one and two inference steps, respectively, but it still trails multi-step latent- and pixel-space methods.Pixel-space FID is typically higher than latent-space FID at 80 epochs, and evaluated pixel-space methods do not show accelerated convergence at that training duration.
  • Text-to-image generation: RAE with SpatialPE-L achieves very good ImageNet FID but ranks among the worst on T2I metrics, showing that ImageNet ranking does not robustly predict T2I ranking.Different T2I metrics can also disagree: E2E-Qwen-Image-VAE is strongest on GenEval and DPG-Bench but only second-tier on GenAIBench.
  • Text-to-image generation: 0.691 GenEval is achieved by E2E-Qwen-Image-VAE, compared with 0.633 for RAE using LangPE-L and 0.624 reported by RAEv2 with SigLIP2-B.These results use a small budget of about 10 wall-clock hours on 32 H200 GPUs, with all setups runnable on 8 H200 GPUs.
  • Recommended usage: Future DiT papers should report DiffusionBench, combining ImageNet and T2I results, because single-axis gains may be task-specific rather than broadly useful.Methods improving one axis while regressing on another should be labeled task-specific improvements.

4 BACKGROUND AND RELATED WORK

Diffusion models dominate visual generation, with DiTs, latent representations, and alignment methods extending the framework across image and text-to-image tasks. Existing evaluation relies on imperfect task-specific metrics, motivating holistic assessment across ImageNet and T2I.

  • Diffusion and flow matching: Diffusion models reverse Gaussian-noised data by predicting noise, whereas flow matching regresses the velocity field along the same path.The prediction target may be noise ϵ, clean signal x_0, or velocity v.
  • Diffusion transformers: DiTs replace diffusion U-Nets with transformers over image patches, while later work refines their architectures and extends them to pixel-space generation.Referenced refinements include SiT, long skip connections, MMDiT joint text–image attention, DDT encoder–decoder splitting, PixNerd, JiT, and PixelGen.
  • Tokenizers and representation alignment: Latent diffusion uses pretrained VAE latents to make high-resolution training tractable, while representation alignment aligns DiT features with frozen vision-encoder representations.REPA is identified as a representation-alignment approach for accelerating and improving diffusion training.
  • ImageNet metrics and their limitation: ImageNet generation is commonly evaluated with FID, Inception Score, and precision/recall, but FID is preprocessing-sensitive, domain-dependent, and saturated.The passage also identifies sFID and FDr as improvements over FID.
  • Text-to-image evaluation: Text-to-image evaluation uses MS-COCO FID for fidelity and diversity and CLIPScore for prompt alignment, yet these metrics are coarse and motivate richer holistic benchmarks.Holistic evaluation is framed as a multi-task alternative to single-dimensional leaderboards, covering both ImageNet and T2I for DiT development.

5 CONCLUSION

The paper introduces NanoGen to make joint ImageNet and text-to-image evaluation practical, showing that ImageNet rankings do not reliably predict text-to-image performance and motivating DiffusionBench. It preserves ImageNet and FID as useful tools while presenting DiffusionBench as a refreshable, limited-scope benchmark.

  • Conclusion: NanoGen removes the engineering barrier to training and evaluating diffusion transformers on text-to-image generation, while showing that ImageNet rankings do not reliably predict text-to-image performance.The paper packages ImageNet and text-to-image generation into DiffusionBench and argues for its adoption.
  • What we are not claiming: ImageNet and FID remain useful platforms for generative modeling, and DiffusionBench is not claimed to be permanent because it should be refreshed as methods saturate it.
  • Limitations: The ImageNet–text-to-image correlation may differ at other scales, while the reported benchmark results use 100K iterations and batch size 1,024; longer training improves generated image quality.
  • Future work: Future work will extend DiffusionBench to world models, videos, and 3D, develop hack-resistant text-to-image metrics, and maintain it as a periodically updated community leaderboard.The paper notes that curated-dataset fine-tuning can hack current text-to-image metrics.

6 CONTRIBUTORS

Jaskirat Singh led development of the unified codebase, while Xingjian Leng added T2I evaluation and several method implementations. Singh drafted the paper, Leng and Zhanhao Liang helped finalize it, Liang Zheng advised, and four collaborators provided feedback.

  • Code Development: Jaskirat Singh led most unified-codebase development, including cross-task training, diverse encoder support, conditioning, optimization, evaluation, and T2I functionality.His contributions covered VAE/RAE and diffusion training across ImageNet and T2I, 80+ vision encoders, autoguidance, REPA, a unified dataloader, in-context conditioning, MeanFlow, Gmuon, online gFID/rFID evaluation, and 256-token T2I conditioning.
  • Code Development: Xingjian Leng added online T2I evaluation, REG, pixel-space, and MeanFlow implementations, and ran the paper’s final experiments and reported results.The online evaluation suites included GenEval, DPG-Bench, and GenAIBench.
  • Paper Writing and Advising: Jaskirat Singh wrote the initial draft, Xingjian Leng and Zhanhao Liang helped with most writing and the final draft, Liang Zheng advised, and four collaborators provided feedback.The feedback contributors were Ethan Smith, Martin Bell, Aninda Saha, and Yuhui Yuan.

A ADDITIONAL RESULTS · A.1 IMAGENET SYSTEMATIC COMPARISON WITHOUT CFG

The appendix reports a systematic ImageNet-256 comparison evaluated without classifier-free guidance, while keeping the training setup identical to Table 2.

  • A.1 IMAGENET SYSTEMATIC COMPARISON WITHOUT CFG: The main result is a systematic comparison on ImageNet-256 without classifier-free guidance.The supplied passage identifies Table 4 as the comparison visual, but does not provide its numerical results.
  • A.1 IMAGENET SYSTEMATIC COMPARISON WITHOUT CFG: Evaluation removes classifier-free guidance from the ImageNet-256 comparison.
  • A.1 IMAGENET SYSTEMATIC COMPARISON WITHOUT CFG: The comparison uses ImageNet at 256-pixel resolution.
  • A.1 IMAGENET SYSTEMATIC COMPARISON WITHOUT CFG: All training settings are kept the same as those in Table 2.
  • A ADDITIONAL RESULTS: Table 4 isolates evaluation without classifier-free guidance while holding the training setup constant.
  • A ADDITIONAL RESULTS: The supplied caption specifies the experimental scope but does not establish a winning method or report metric values.

A.2 IMAGENET-FID AND T2I METRICS CORRELATION INCLUDING PIXEL-SPACE METHODS

Including pixel-space methods makes ImageNet FID–T2I correlations appear stronger because these methods perform much worse than latent-space methods on both evaluation types. The main analysis therefore focuses on the latent-space frontier, where the correlation is not driven by pixel-space outliers.

  • Pixel-space methods: Pixel-space methods are much worse than latent-space methods on both ImageNet FID and T2I metrics, artificially raising the overall correlation.The analysis includes JiT (Li & He, 2025), PixNerd (Wang et al., 2025a), and PixelGen (Ma et al., 2026), with and without CFG.
  • Analysis scope: The main analysis excludes pixel-space methods and evaluates correlation on the latent-space frontier to avoid outlier-driven conclusions.This restriction is motivated by the substantially worse ImageNet FID and T2I performance of pixel-space methods.
  • Pixel-space methods: Figure 4 extends the correlation analysis with JiT (Li & He, 2025), PixNerd (Wang et al., 2025a), and PixelGen (Ma et al., 2026), evaluated without CFG and at each method’s best CFG scale.The CFG analysis uses a timestep interval of [0.0, 0.9].

A.3 IMAGENET-FID AND T2I METRICS CORRELATION WITHOUT CFG

Repeating the correlation analysis with ImageNet FID evaluated without classifier-free guidance changes the Pearson r values across metrics but still shows no strong correlation with T2I metrics. Thus, the finding is not dependent on the CFG protocol used for ImageNet evaluation.

  • Without CFG: Without CFG, ImageNet FID still shows no strong correlation with T2I metrics across the evaluated metrics.This result is reported in Figure 5 using the same setup as Figure 1, but without classifier-free guidance.
  • Without CFG: The Pearson r values change across metrics when ImageNet FID is evaluated without CFG.
  • Without CFG: The absence of a strong ImageNet–T2I correlation does not depend on whether CFG is used for ImageNet evaluation.
Loading 2606.24888v1…