Source-linked AI summary
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, Ming-Yu Liu
TL;DR
Text-to-image diffusion models use shared denoisers despite changing reliance on text and visual information across sampling stages, motivating a more stage-aware design. eDiff-I trains specialized expert denoisers from a shared model, combines T5, CLIP text, and CLIP image conditioning, and adds paint-with-words control. It reports improved performance and controllability while maintaining the same inference computation cost, with a best FID of 7.04 among the reported configurations.
Problem
Increasing denoiser capacity at every sampling step can raise test-time complexity, while shared denoisers may not effectively represent the process’s changing temporal dynamics.
Method
eDiff-I pretrains one diffusion model, fine-tunes an ensemble of denoisers for different noise ranges, and conditions generation on T5 text, CLIP text, and CLIP image embeddings.
Results
eDiff-I achieves state-of-the-art text-to-image results, including a best FID of 7.04, while its embeddings provide complementary image formation, style transfer, and paint-with-words control.
Takeaways & Limitations
Stage-specialized denoising preserves inference computation while supporting improved image generation and richer text-, style-, and location-based control.
Takeaways & Limitations
Large-scale text-to-image models can enable harmful manipulation and reflect biases from extremely large, mostly unfiltered training datasets.
Abstract
from arXiv · showhide
Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. Starting from random noise, such text-to-image diffusion models gradually synthesize images in an iterative fashion while conditioning on text prompts. We find that their synthesis behavior qualitatively changes throughout this process: Early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored. This suggests that sharing model parameters throughout the entire generation process may not be ideal. Therefore, in contrast to existing works, we propose to train an ensemble of text-to-image diffusion models specialized for different synthesis stages. To maintain training efficiency, we initially train a single model, which is then split into specialized models that are trained for the specific stages of the iterative generation process. Our ensemble of diffusion models, called eDiff-I, results in improved text alignment while maintaining the same inference computation cost and preserving high visual quality, outperforming previous large-scale text-to-image diffusion models on the standard benchmark. In addition, we train our model to exploit a variety of embeddings for conditioning, including the T5 text, CLIP text, and CLIP image embeddings. We show that these different embeddings lead to different behaviors. Notably, the CLIP image embedding allows an intuitive way of transferring the style of a reference image to the target text-to-image output. Lastly, we show a technique that enables eDiff-I's "paint-with-words" capability. A user can select the word in the input text and paint it in a canvas to control the output, which is very handy for crafting the desired image in mind. The project page is available at https://deepimagination.cc/eDiff-I/
1. Introduction
eDiff-I targets scalable text-to-image generation by specializing denoisers for different stages, where text guidance is strongest early and visual refinement dominates later. It also combines complementary text and image embeddings and adds spatial control through paint-with-words.
- Motivation: Diffusion sampling repeatedly denoises random noise, but increasing denoiser capacity at every step would raise test-time computation.Sampling calls the denoising network many times while solving a reverse generative differential equation.
- Motivation: Early generation relies mainly on text to establish text-aligned content, whereas later denoising increasingly relies on visual features.Attention visualizations and prompt switching support this stage-dependent behavior.
- Method: eDiff-I uses an ensemble of expert denoisers specialized for different generation stages without increasing computation per sampling step.A shared model is pretrained, then specialized models are initialized from it and fine-tuned for fewer iterations.
- Results: The shared pretraining and stage-specific fine-tuning scheme achieves state-of-the-art text-to-image results on the benchmark dataset.The reported approach is designed to increase capacity while reducing the additional training cost of separately training stage-specific models.
- Conditioning: T5 and CLIP text embeddings capture complementary image properties, while their joint use gives the best generation results and CLIP image embeddings support style transfer.CLIP favors global appearance, whereas T5 better reflects individual objects but produces less accurate global looks alone.
- Controllability: A training-free cross-attention modulation extension lets users specify object locations by scribbling semantic regions in a canvas.The resulting model accepts text and semantic masks for additional spatial control.
2. Related Work
Diffusion models are introduced as iterative denoising generators and surveyed across text-to-image generation, downstream image problems, and broader scaling trends. Related work also highlights the computational costs of increasing model and dataset size.
- Diffusion models: Denoising diffusion models generate samples through iterative denoising and are trained with objectives across different noise levels.They are also known as noise-conditioned score networks.
- Text-to-image diffusion models: Text-to-image diffusion models condition denoising on text in image or latent space, often using low-resolution images or latent variables for efficiency.Examples include GLIDE, Imagen, DALL·E 2, Stable Diffusion, and VQ-Diffusion.
- Applications: Diffusion models support downstream applications including super-resolution, inpainting, deblurring, and JPEG restoration.The surveyed applications apply diffusion models to inverse problems and image-to-image translation.
- Scaling: Scaling model parameters and data has improved performance across language understanding, visual recognition, and multimodal reasoning, but increases computational and energy costs.Sparse expert models are one approach discussed for routing inputs to limited subsets of network weights.
3. Background
Text-to-image diffusion models iteratively denoise Gaussian noise while conditioning on text embeddings. Training recovers clean images from noise-corrupted image-text pairs, and sampling follows a generative ODE from high to near-zero noise; super-resolution models additionally condition on low-resolution images.
- Diffusion formulation: Text-to-image diffusion models iteratively denoise samples from a noise distribution using the noisy image, text embedding, and current noise level.The denoiser is written as D(x; e, σ), where x is the noisy image, e is the input embedding, and σ indicates the noise level.
- Training: The denoising model is trained to recover clean images from versions corrupted with Gaussian noise sampled at varying scales.The objective uses training image-text pairs, standard Normal noise, sampled noise levels, and a noise-dependent loss weighting factor.
- Denoiser formulation: The denoiser is preconditioned with a noise-dependent formulation using a trained neural network and σdata = 0.5 for natural-image pixel variability.The noise scale follows a log-normal distribution with Pmean = −1.2 and Pstd = 1.2, while λ(σ) cancels the output weighting.
- Sampling: Sampling starts from x ∼ N(0, σmax^2) and solves a generative ODE backward from high noise toward σmin ≈ 0.The ODE uses the score function obtained from the denoising model to guide samples toward images aligned with the conditioning.
- Super-resolution: Text-conditioned super-resolution denoisers also receive a low-resolution image, corrupted during training to improve generalization to generated-image artifacts.This conditioning distinguishes the super-resolution models from the basic text-conditioned diffusion formulation.
4. Ensemble of Expert Denoisers
eDiff-I exploits the changing roles of text and visual information across denoising by assigning specialized experts to different noise ranges. Its cascade combines multistage resolution enhancement, multiple conditioning embeddings, and training-free spatial control through paint-with-words.
- Temporal dynamics: At high noise levels, denoising relies mainly on text embeddings, whereas at low noise levels it relies mostly on visual features for fine details.Prompt switching during the last 7% of denoising leaves the output unchanged, while switching during the first 40% changes it completely.
- Expert denoisers: An ensemble of expert denoisers specializes separate models for noise ranges, increasing capacity without increasing computation per sampling step.The approach replaces one shared denoiser across all noise levels while preserving the per-step evaluation cost.
- Efficient training: Training begins with a shared model, which initializes specialized experts instead of training each denoiser from scratch.A binary-tree branching strategy recursively splits the noise distribution into equal-area intervals and fine-tunes experts for them.
- Conditioning embeddings: An optional CLIP image embedding enables stylistic control, making outputs follow the style of a reference image rather than the natural style.The embedding is supplied alongside the text conditioning in the eDiff-I pipeline.
- Efficient training: The branching strategy emphasizes low- and high-noise intervals because high-noise denoising supports core text-conditioned image formation while low-noise denoising produces sharp results.The final system uses experts for low noise, high noise, and intermediate noise intervals rather than expanding every binary-tree node.
- Conditioning embeddings: T5 and CLIP text embeddings provide complementary conditioning, with joint use producing the best image-generation results in the model.CLIP favors global appearance, while T5 better reflects individual objects but produces less accurate global looks alone.
- Multistage generation: The eDiff-I cascade generates 64×64 images and progressively upsamples them to 256×256 and 1024×1024 with two super-resolution diffusion models.The super-resolution models are trained with randomly degraded low-resolution inputs to handle artifacts from the base model.
- Paint-with-words: Paint-with-words lets users select prompt phrases and draw masks that modulate cross-attention to control where corresponding concepts appear.Masks are resized for each attention layer, and painted regions are encouraged to attend more strongly to the selected phrase tokens.
5. Experiments
Experiments evaluate eDiff-I’s ensemble denoisers, conditional embeddings, image quality, and controllability. The ensemble improves benchmark performance, while combined T5 and CLIP embeddings support stronger generation and CLIP image embeddings enable style transfer.
- Experimental setup: The experiments evaluate optimization, datasets, quantitative comparisons, ablations, embedding effects, style transfer, and paint-with-words.Evaluation uses MS-COCO and Visual Genome, with FID and CLIP scores as primary metrics.
- Ensemble evaluation: The 2-expert ensemble consistently outperforms the baseline across the entire FID-CLIP trade-off curve.The baseline is trained for 800K iterations and the ensemble for 600K, giving both the same number of training samples.
- Conditional embeddings: Using T5 and CLIP jointly gives the best FID-CLIP trade-offs on both COCO and Visual Genome, while T5 alone is stronger than CLIP alone on Visual Genome.The benefit of T5 alone is attributed to Visual Genome’s longer, more descriptive captions.
- Main results: eDiff-I-Config-D achieves the best reported FID of 7.04, while Config-C beats Imagen and Parti by 0.16 and 0.12 FID, respectively.Config-B achieves an FID of 7.26, slightly better than Imagen.
- Qualitative comparison: eDiff-I better preserves multiple attributes, generates text more accurately, and handles long descriptions better than Stable Diffusion and DALL·E 2.The qualitative comparisons report fewer attribute omissions and misspellings for eDiff-I.
- Controllability: CLIP image conditioning transfers a reference image’s style, and paint-with-words generates high-quality images matching coarse user-specified layouts.Paint-with-words supports multiple concepts in one image generated from scratch, rather than applying one concept to an existing region.
6. Conclusions
The conclusion presents eDiff-I as a high-definition text-to-image model whose stage-specialized denoisers and complementary embeddings improve generation and controllability. It also highlights style transfer, spatial control, creative applications, and risks from misuse and training-data bias.
- Conclusions: eDiff-I combines a base diffusion model with two super-resolution modules to produce 1024 × 1024 high-definition outputs.The paper describes its expert denoisers as providing superior performance compared with previous work.
- Conclusions: Stage-specialized expert denoisers reflect changing synthesis behavior: text alignment dominates early, while visual quality dominates later.The experts specialize these different behaviors across intervals of the iterative synthesis process.
- Conclusions: T5 and CLIP text embeddings capture complementary image aspects, while CLIP image embeddings enable reference-image stylization.The conclusion identifies joint text conditioning and image conditioning as sources of controllability.
- Conclusions: Paint-with-words provides expressive spatial control by allowing users to specify object locations through scribbles.The method is presented as a way to help users craft desired images.
- Societal impact: eDiff-I is envisioned as a tool for designers, photographers, and content creators to produce detailed imagery without specialized skills.The societal-impact discussion frames this as supporting artistic expression and content creation.
- Risks and limitations: The authors caution that these models can support malicious manipulation, deceptive content, and biased outputs from largely unfiltered training data.They point to image validation, representative data collection, and bias correction as possible responses.
A. Network Architecture
The network architecture extends a diffusion U-net with global and cross-attention conditioning from CLIP text, CLIP image, and T5 embeddings. Efficient U-net structures are used for the super-resolution modules.
- Base architecture: The base diffusion model modifies Dhariwal et al.’s U-net architecture.The stated modifications concern conditioning and attention blocks.
- Base architecture: Global conditioning adds projected pooled CLIP text and CLIP image embeddings alongside the timestep embedding.The architecture does not use pooled T5 embeddings for global conditioning.
- Attention blocks: Cross-attention follows every self-attention block and uses concatenated CLIP text, T5, and pooled CLIP image conditioning tokens.The model also includes a learnable null embedding for conditioning dropout.
- Super-resolution architecture: The super-resolution models use Efficient U-net blocks and omit self-attention, retaining cross-attention for efficiency.SR1024 is trained on 256×256 random patches and applied at 1024×1024 inference resolution.
B. Ensemble training schedule
The ensemble training schedule uses binary-tree branching over noise-level intervals. Specialized models are initialized from shared or intermediate models and trained on interval-specific or complementary noise distributions.
- Training strategy: The ensemble is trained with a binary-tree-based branching strategy.Tables 5 and 6 provide the exact schedules for the base and super-resolution models.
- Noise distributions: Negative interval notation denotes the complementary distribution containing all noise levels except the indicated interval.For example, −(9, 511) represents all noise levels other than the specified interval.
- Base-model schedule: Each configuration identifies a tree level and a noise-interval id used to define the model’s training distribution.The schedule also records the initialization model and training iterations.
- Super-resolution schedule: The same level-and-interval configuration scheme is used for the super-resolution training schedule.Table 6 lists initialization sources and iteration counts for the super-resolution models.
B.1. Hyper-parameters
This section identifies Table 7 as the location of the hyperparameters used to train the models.
- The hyperparameters used for training all models are provided in Table 7.
- The section directs readers to Table 7 for the models’ training settings.
- Table 7 is labeled “Hyperparameters.”