PaperlayerView exact revision on arXiv

Source-linked AI summary

GenTron: Diffusion Transformers for Image and Video Generation

Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, Juan-Manuel Perez-Rua

arXiv:2312.04557v2cs.CV

TL;DR

Visual diffusion has primarily used CNN-based U-Nets, while Transformers have become dominant in NLP and visual perception. GenTron adapts Transformer diffusion to text-conditioned image and video generation, reporting improved quality with scaling and motion-free guidance, alongside favorable human evaluations against SDXL.

  • Problem

    Visual diffusion predominantly uses CNN-based U-Nets, unlike the increasingly dominant Transformer architectures in NLP and visual perception.

    Source: §1 · p. 2

  • Method

    GenTron adapts DiTs from predefined class conditioning to free-form text, investigates conditioning strategies, scales the models, and adds temporal self-attention and motion-free guidance for video.

    Source: §1 · p. 2

  • Results

    GenTron reports a 51.1% win rate in visual quality and a 42.3% win rate in text alignment against SDXL, and superior T2I-CompBench performance across compositional criteria.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study reports findings on Transformer conditioning and scaling and evidence of improved generated-video visual quality with motion-free guidance.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this study, we explore Transformer-based diffusion models for image and video generation. Despite the dominance of Transformer architectures in various fields due to their flexibility and scalability, the visual generative domain primarily utilizes CNN-based U-Net architectures, particularly in diffusion-based models. We introduce GenTron, a family of Generative models employing Transformer-based diffusion, to address this gap. Our initial step was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a process involving thorough empirical exploration of the conditioning mechanism. We then scale GenTron from approximately 900M to over 3B parameters, observing significant improvements in visual quality. Furthermore, we extend GenTron to text-to-video generation, incorporating novel motion-free guidance to enhance video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench, underscoring its strengths in compositional generation. We believe this work will provide meaningful insights and serve as a valuable reference for future research.

1. Introduction

Visual diffusion relies predominantly on CNN-based U-Nets, unlike NLP and visual perception, motivating a systematic investigation of Transformers for image and video generation. GenTron adapts DiTs to free-form text, scales them, and extends them to video, where human evaluations and compositional benchmarks report strong results.

1. Introduction

  • The study addresses the contrast between U-Net dominance in visual diffusion and Transformer dominance in natural language processing and visual perception.It investigates Transformer-based diffusion for both image and video generation.Source: §1 · p. 2
  • GenTron replaces DiT’s restricted predefined-class conditioning with language embeddings from open-world, free-form text captions.The study compares adaLN with cross-attention and evaluates generic language-model encoders, multimodal language towers, and their combination.Source: §1 · p. 2
  • GenTron scales its Transformer blocks and hidden dimension; its largest model, GenTron-G/2, has more than 3B parameters and achieves significant visual quality improvement compared with the smaller model.Source: §1 · p. 2
  • GenTron extends text-to-image diffusion to text-to-video by adding temporal self-attention to each Transformer block and introducing motion-free guidance.Motion-free guidance intermittently disables motion modeling during training and integrates with joint image-video training; experiments report improved generated-video visual quality.Source: §1 · p. 2
  • GenTron outperforms SDXL in human evaluations, with a 51.1% win rate in visual quality and a 42.3% win rate in text alignment.It also demonstrates superior performance across T2I-CompBench criteria, including attribute binding, object relationships, and complex compositions.Source: §1 · p. 2

2. Related Work

Related work describes latent diffusion pipelines and prior Transformer-based diffusion architectures, then distinguishes GenTron by its focus on conditioning and scalability for text-to-image diffusion and its extension to video.

Diffusion models for T2I and T2V generation

  • Latent diffusion models use a pretrained variational autoencoder, a text conditioner commonly modeled by cross-attention, and a U-Net backbone.Video latent diffusion models implement the process in a low-dimensional latent space to avoid excessive computing demands.Source: §2 · pp. 2–3

Transformer-based Diffusion

  • Prior Transformer diffusion work includes U-ViT’s combination of Transformer blocks with a U-Net, DiT’s non-hierarchical structure, and efficiency-oriented masking approaches.MDT and MaskDiT incorporate a mask strategy to improve DiT training efficiency.Source: §2 · p. 3
  • GenTron focuses on conditioning strategy and scalability in Transformer text-to-image diffusion, and explores video diffusion beyond PixArt-α’s scope.PixArt-α is described as a concurrent Transformer text-to-image model trained using a three-stage decomposition process.Source: §2 · p. 3

3. Method

GenTron replaces the class-conditioned DiT setup with text-conditioned transformer diffusion, scales the model, and extends it to video with temporal self-attention and motion-free guidance. The method combines image and video training and modulates motion conditioning during inference.

3.1. Preliminaries

  • Diffusion generates samples by progressively adding noise and learning a denoiser to reverse that process; GenTron builds on DiT’s latent-space Transformer backbone.GenTron converts image latents into nonoverlapping tokens before processing them through Transformer blocks.Source: §3.1 · p. 3, §3.2 · pp. 3–4

3.2.1 From Class to Text Condition

  • For text-to-image generation, GenTron investigates text encoders and embedding integration, using cross-attention for text while retaining adaLN to model time conditioning.The study considers CLIP, Flan-T5, and their combinations; pooled text embeddings are also added to the time embedding.Source: §3.2.1 · p. 4

3.2.2 Scaling Up GenTron

  • GenTron-G/2 scales depth, patch-embedding dimensionality, and MLP width, reaching more than 3 billion parameters.Source: §3.2.2 · p. 4

3.3.1 GenTron-T2V Architecture

  • For video, GenTron inserts lightweight temporal self-attention into each Transformer block between cross-attention and the MLP, initializing the new layers to act as identity mappings.The T2I model initializes shared layers, while zero-initialized output projections make the added temporal layers initially produce zero output.Source: §3.3.1 · pp. 4–5

3.3.2 Motion-Free Guidance

  • Joint image-video training addresses video-data scarcity and image-video domain discrepancy, while inference adjusts separate text and motion guidance scales.The authors report that λT = 7.5 and λM ∈ [1.0, 1.3] tend to achieve the best result.Source: §3.3.2 · pp. 5–6
  • Motion-free guidance treats motion as a conditioning signal and intermittently uses an identity temporal-attention mask during training to disable temporal modeling.When motion is omitted, image-text pairs are converted into pseudo videos by repeating the image across frames; otherwise, training uses video clips.Source: §3.3.2 · pp. 5–6

3.2.2 Scaling Up GenTron

  • GenTron-G/2 with CLIP-T5XXL performs best among the compared model variants on T2I-CompBench.The comparison covers conditioning choices and model scale.Source: Table 2 · p. 6

4. Experiments

Experiments assess GenTron’s training setup and text-to-image performance across conditioning, encoders, and model scales, then evaluate text-to-video generation and motion-free guidance.

4.2. Main Results of GenTron-T2I

  • Cross-attention uniformly outperforms adaLN on evaluated metrics for free-form text conditioning, as illustrated by the panda prompt.The first two rows of Table 2 provide the quantitative comparison.Source: §4.2 · p. 7, Figure 4 · p. 7
  • With XL/2 held constant, T5XXL beats CLIP-L on three attribute-binding and spatial-relationship metrics, while combined CLIP-L and T5XXL embeddings improve performance.T5XXL is comparable on the remaining two metrics.Source: §4.2 · p. 7
  • Scaling from approximately 900M to 3B parameters improves T2I-CompBench criteria and produces finer details and more coherent object layouts.The compared models use the same CLIP-T5XXL condition; Figure 5 shows a cat-and-newspaper example.Source: §4.2 · p. 7, Figure 5 · p. 7
  • GenTron exceeds prior methods across T2I-CompBench compositional metrics, including more than 7% over prior state-of-the-art on color binding.Table 3 summarizes alignment results; the paper identifies color binding as a particular strength.Source: Table 3 · p. 6, §4.2 · p. 8
  • Against SDv1.4, GenTron uses 550M versus 2B data and achieves better CLIP-scores and T2I-CompBench results, but not a better FID score.Table 4 compares zero-shot FID-30K, CLIP-Score, and T2I-CompBench metrics.Source: Table 4 · p. 7, §4.2 · p. 8
  • In a blind study with 3,000 responses, participants preferred GenTron over SDXL for visual quality and text faithfulness.The study used 100 images per method from standard PartiPrompt2 prompts.Source: §4.2 · p. 8

4.3. GenTron-T2V Results

  • GenTron-T2V examples are described as visually striking and temporally coherent.Figure 6 shows prompts about a giant tortoise crossing a beach and a swimming dog.Source: §4.3 · p. 8, Figure 6 · p. 8
  • With training conditions held constant, motion-free guidance tends to make the prompted central object more detailed and visually prominent across video frames.Figure 8 compares GenTron-T2V variants with and without motion-free guidance.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

The conclusion reports an exploration of transformer-based diffusion for text-conditioned images and videos, with quality improvements from scaling and motion-free guidance.

5. Conclusion

  • The authors report that scaling improves image quality and motion-free guidance enhances the visual quality of generated videos.They frame the work as exploring transformer-based diffusion for text-conditioned image and video generation.Source: §5 · p. 8

A. Summary of Conditioning Mechanism

For multiple text encoders, GenTron alternates between their embeddings across transformer blocks; for a single encoder, it uses plain cross-attention.

A. Summary of Conditioning Mechanism

  • With multiple encoders, interleaved cross-attention alternates between their text embeddings across transformer blocks, while single-encoder cases use plain cross-attention.The example switches from CLIP embeddings in one block to Flan-T5 embeddings in the next.Source: A. Summary of Conditioning Mechanism · p. 12

B.1. Additional Model Scaling-up Examples

The appendix presents additional qualitative examples of model scaling from the PartiPrompt benchmark.

B.1. Additional Model Scaling-up Examples

  • Figure 9 provides additional qualitative model-scaling examples using PartiPrompt prompts.Source: §B.1 · p. 12

C. Additional GenTron-T2I Examples

This section points to additional GenTron-T2I examples and a figure illustrating model scaling-up effects.

C. Additional GenTron-T2I Examples

D. Additional GenTron-T2V Examples

Additional GenTron-T2V results are available on the authors’ website.

D. Additional GenTron-T2V Examples

Related papers

Loading 2312.04557v2…