Source-linked AI summary
Text-to-image Diffusion Models in Generative AI: A Survey
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, In So Kweon, Junmo Kim
TL;DR
Text-to-image diffusion research has expanded rapidly, creating a need for a focused account of its key breakthroughs. This survey synthesizes diffusion-based text-to-image methods and their extensions, finding broad progress from image generation to video, 3D, and editing while identifying ongoing challenges.
Problem
The rapid growth of text-to-image diffusion research makes its key breakthroughs difficult to learn from existing surveys with limited task-specific or diffusion-focused coverage.
Method
The survey introduces diffusion and text-conditioned image synthesis, reviews pioneering methods and improvements, and covers applications, evaluation, challenges, and future directions.
Results
The survey organizes progress in text-to-image diffusion and extends its coverage to text-guided video generation, 3D object generation, and image editing.
Takeaways & Limitations
Text-to-image diffusion is presented as a leading approach whose progress can be understood alongside related text-guided generation and editing tasks.
Takeaways & Limitations
Generated content can be inappropriate, including offensive, insulting, or threatening outputs, because of dataset-related issues.
Abstract
from arXiv · showhide
This survey reviews the progress of diffusion models in generating images from text, ~\textit{i.e.} text-to-image diffusion models. As a self-contained work, this survey starts with a brief introduction of how diffusion models work for image synthesis, followed by the background for text-conditioned image synthesis. Based on that, we present an organized review of pioneering methods and their improvements on text-to-image generation. We further summarize applications beyond image generation, such as text-guided generation for various modalities like videos, and text-guided image editing. Beyond the progress made so far, we discuss existing challenges and promising future directions.
1. Introduction
Text-to-image generation has progressed from early GAN and autoregressive approaches to diffusion models, which now lead the task. This survey organizes that progress, reviews methods and evaluations, covers extensions beyond images, and identifies challenges and future opportunities.
- Text-to-image synthesis helps visualize information from language, supporting human understanding and enjoyment.
- Early methods progressed from AlignDRAW and text-conditional GANs to autoregressive models, but faced limited realism, small-scale training, computation costs, and sequential error accumulation.
- Diffusion models have emerged as the leading approach for text-to-image generation, with DALL-E2 producing imaginative images with high fidelity.
- Existing surveys provide limited coverage of text-to-image synthesis or focus on GAN-based methods without introducing diffusion-based approaches.
- This work presents a comprehensive survey of diffusion-based text-to-image generation, including background, pioneering methods, subsequent advances, and evaluation.
- The survey broadens its scope to text-to-X applications such as video and 3D object generation, while discussing challenges and future opportunities.
2. Background on diffusion models
Diffusion models generate samples by learning to reverse a noise-adding process, with DDPM providing a milestone implementation and later work connecting related formulations and guidance mechanisms. This background covers the development of DPMs and SGMs, DDPM’s forward and reverse passes, and conditional guidance for incorporating labels or text.
- Diffusion probabilistic models: Diffusion models are Markov-chain generative models trained to reverse data perturbation, transforming complex distributions toward simple noise and learning the reverse mapping for generation.
- Development before DDPM: DDPM, published in 2020, sparked rapidly increasing interest and offers a self-contained framework for understanding unconditional image synthesis and conditional guidance.
- Development before DDPM: DPMs estimate the reversal of a Markov diffusion chain, while SGMs perturb data with Gaussian noise at multiple magnitudes and estimate score functions for noisy distributions.
- Development before DDPM: DDPM and SGM share a similar optimization objective and can be unified through stochastic differential equations, enabling more flexible sampling methods.
- How DDPM works for image synthesis: DDPM adds Gaussian noise to images over forward steps until they are destroyed, then learns reverse transition kernels that recover clean images from noise.
- Guidance in diffusion-based image synthesis: Conditional diffusion models use class or text inputs, whereas guided models use class-induced gradients during sampling; classifier-free guidance obtains guidance without an auxiliary classifier.
- Guidance in diffusion-based image synthesis: Classifier-free guidance achieves a quality–diversity trade-off similar to classifier guidance and supports modalities such as text for text-to-image generation.
3. Pioneering text-to-image diffusion models
Pioneering text-to-image diffusion models are organized by whether diffusion occurs in pixel space or compressed latent space, with representative systems including GLIDE, Imagen, Stable Diffusion, and DALL-E 2. Their advances include classifier-free guidance, pretrained language encoders, latent-space modeling, and multimodal latent priors.
- Text-to-image diffusion models are categorized by performing diffusion directly in pixel space or after compressing images into a low-dimensional latent space.
- GLIDE introduced text-conditioned diffusion for image generation and outperformed DALL-E in both FID and human evaluations of sample quality.
- Imagen uses a pretrained frozen language model as its text encoder, and increasing language-model size improves image fidelity and image-text alignment more than enlarging the diffusion model.
- Stable Diffusion performs diffusion in latent space, reducing complexity while preserving detail, and classifier-free guidance further improves latent text-to-image models.
- DALL-E 2 uses CLIP representations and learns a text-image latent prior with diffusion, whose removal causes a large performance drop.
4. Model advancements
Model advancements improve text-to-image diffusion through architecture, denoising, acceleration, representation learning, personalization, spatial control, and versatile conditioning. These methods target fidelity, text alignment, sampling efficiency, controllability, and support for complex or personalized content.
- Denoising process: Multiple denoiser models can specialize across generation stages because early sampling prioritizes text alignment while later sampling prioritizes image quality.
- Model architecture: Free-U improves image quality by re-weighting U-Net skip-connection and backbone feature contributions without additional training or fine-tuning.
- Model architecture: PixArt-α supports high-resolution synthesis up to 1024 × 1024, while DiT replaces the commonly used U-Net backbone with a diffusion transformer.
- Model acceleration: V-prediction distills an N-step DDIM sampler into N/2 sampling steps without hurting generation quality, addressing diffusion models’ slow sampling.
- Model acceleration: REPA accelerates SiT training by over 17.5× by incorporating representations from self-supervised models such as DINO v2.
- Textual inversion for concept control: Personalization methods learn subject concepts from reference images, enabling users to control the subject’s status through prompts.
- Spatial control: Spatial-control methods use spatio-textual representations, sketches, boxes, or scribbles to provide fine-grained layout control beyond natural-language prompts.
- Versatile content control: ControlNet adds conditioning signals such as edges, depth, and segmentation to pretrained diffusion models, while related systems unify multiple control signals.
5. Model evaluation
The survey describes common metrics and benchmarks for evaluating text-to-image models, alongside ethical, security, and privacy evaluations. These studies show that diffusion models can achieve strong image quality while introducing bias, inappropriate content, and detection challenges.
- Quantitative metrics: FID measures the distance between synthetic and real-world images, with lower values indicating higher image fidelity.CLIP scores measure text-image alignment, while IS and R-precision provide additional evaluation perspectives.
- Image quality: Diffusion models outperform autoregressive models in image quality, with lower FID on MS-COCO.The survey summarizes representative methods and their MS-COCO results in Table 1.
- Evaluation benchmarks: Evaluation resources increasingly test complex scenes, multilingual prompts, visual reasoning, social biases, infrequent entities, and multiple capabilities.Examples include UniBench, PartiPrompts, PaintSkills, EntityDrawBench, and a multi-task benchmark with thirty-two tasks.
- Ethical risks: Large-scale unfiltered training data can reinforce biases and produce inappropriate or unfair content, including stereotypes associated with traits, descriptors, occupations, or objects.Ethical benchmarks examine interventions across gender, skin color, and culture, while Safe Latent Diffusion suppresses inappropriate content with additional guidance.
- Security and privacy: Diffusion-generated images are harder for existing detectors to identify than GAN-generated images, partly because of high-frequency mismatches.DE-FAKE also reports shared features and model-specific fingerprints that support source-model tracking.
- Security and privacy: Security and privacy evaluations include backdoor attacks that activate attributes through trigger words and membership-leakage attacks that infer whether an image trained the model.The proposed membership attacks achieve impressive results, highlighting the threat of membership leakage.
6. Applications beyond image generation
The survey reviews applications of diffusion models beyond text-to-image generation, including artistic creation, video synthesis, 3D generation, and text-guided image editing. Across these modalities, methods add controls for style, temporal or geometric consistency, inversion, and flexible textual edits.
- Overview: Text-to-X applications extend diffusion models to modalities such as video, while text-guided image editing broadens their use beyond image generation.The survey introduces pioneering work across these application areas.
- Artistic image generation: Artistic generation methods improve control over content, aesthetics, style, diversity, quality, and computational efficiency.Examples include multimodal guidance, dual diffusion models, retrieval-augmented diffusion, and supervised or self-style guidance.
- Video generation: Video-generation methods target consistency across frames and scenes, while recent systems improve text-to-video and image-to-video performance.Make-A-Story uses visual memory, AR-LDM uses image-caption history, Stable Video Diffusion introduces staged training, and Sora compresses videos and images into latent patches.
- Video generation: Sora generates a minute of high-fidelity video by representing videos and images as compressed patches, then applying diffusion to spacetime patches with a transformer backbone.Sora is not open-sourced, motivating open-access efforts such as Open-Sora.
- Text-guided image editing: Image-editing methods range from noise-based inversion to flexible text control, automatic mask generation, and tuning-free high-fidelity editing.EDICT couples noise vectors, LEDITS++ avoids tuning and optimization, DiffEdit automatically creates masks, and Prompt-to-Prompt uses cross-attention maps.
7. Challenges and outlook
The survey identifies challenges involving ethical bias, data and computation, security, and multilingual access. It points toward safer and more capable systems through balanced data, improved detection and robustness, unified multimodality, and collaboration with other fields.
- Ethical issues and dataset bias: Large-scale unfiltered datasets can generate inappropriate or unfair content, while English-dominant inputs may disadvantage people who do not understand English.The survey links these concerns to offensive, threatening, or socially unfair outputs and unequal language access.
- Security risks: Improved generation makes synthetic images harder to distinguish from real ones, creating risks of malicious uses such as falsifying electronic evidence.The survey also identifies backdoor attacks and privacy issues as security concerns.
- Data and computation: Training text-to-image models requires hundreds of millions of image-text pairs and substantial computation, limiting training from scratch to large companies.Efficient deployment remains challenging, including in environments such as edge devices.
- Safe and fair applications: Promising directions include more diverse and balanced datasets, advanced generated-image detection, and diffusion models robust against various attacks.These directions directly target the survey’s ethical and security concerns.
- Unified multi-modality framework: Unified multimodality models could combine tasks such as text-to-image generation and image captioning, potentially improving representations across modalities.The survey presents multi-task unification as a promising trend rather than an established result.
- Collaboration with other fields: Collaboration with language, generative, and vision models is identified as an open direction for addressing ethical and security issues and extending diffusion applications.Examples include GPT-3 integration, autoencoder decompositions, restoration, depth estimation, enhancement, and classification.