State of the Art on Diffusion Models for Visual Computing

Ryan Po; Wang Yifan; Vladislav Golyanik; Kfir Aberman; Jonathan T. Barron; Amit H. Bermano; Eric Ryan Chan; Tali Dekel; Aleksander Holynski; Angjoo Kanazawa; C. Karen Liu; Lingjie Liu; Ben Mildenhall; Matthias Nießner; Björn Ommer; Christian Theobalt; Peter Wonka; Gordon Wetzstein

Source-linked AI summary

State of the Art on Diffusion Models for Visual Computing

Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T. Barron, Amit H. Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, C. Karen Liu, Lingjie Liu, Ben Mildenhall, Matthias Nießner, Björn Ommer, Christian Theobalt, Peter Wonka, Gordon Wetzstein

arXiv:2310.07204v1Oct 11, 2023cs.AIcs.CVcs.GRcs.LG

TL;DR

Visual computing needs an organized account of rapidly expanding diffusion-model research across media and applications. This STAR combines mathematical foundations with a scoped literature survey, discusses datasets, metrics, challenges, and social implications, and concludes that it offers an intuitive starting point while remaining non-exhaustive and constrained by unresolved data, evaluation, and controllability issues.

Problem
Diffusion-model research in visual computing is expanding rapidly, while applications beyond 2D face data and computation constraints and generated-content evaluation remains imperfect.
Method
The STAR introduces diffusion mathematics and sampling, surveys selected visual-computing applications, and reviews datasets, metrics, open challenges, and social implications.
Results
The report provides a comprehensive overview of diffusion-based generation and editing across generated media, together with discussions of datasets, metrics, open challenges, and social implications.
Takeaways & Limitations
The STAR is intended as an intuitive starting point for researchers, artists, and practitioners exploring diffusion models for visual computing.
Takeaways & Limitations
The report does not claim completeness, and diffusion applications remain constrained by the quality and quantity of training data, especially paired datasets for conditioned motion models.

Abstract

from arXiv · show

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion-based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state-of-the-art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

1. Introduction

Generative AI is reshaping visual computing by making photorealistic and stylized image, video, and 3D content generation and editing accessible through prompts or high-level guidance. This progress also exposes challenges in extending models beyond 2D, reducing computational demands, and tracking rapidly expanding research.

Generative AI enables photorealistic and stylized generation and editing of images, videos, and 3D objects from text prompts or high-level guidance.These tools automate laborious processes that previously required specialized domain knowledge.
Foundation models such as Stable Diffusion, Imagen, Midjourney, and DALL-E underpin current visual generative tools through large-scale text–image training.The passage describes models trained on hundreds of millions to billions of text–image pairs with billions of learnable parameters.
Extending image foundation models to video and 3D remains unclear, largely because suitable training data are scarce compared with the abundance of web images.The supplied passage begins to contrast the volume of 2D images with fewer examples for higher-dimensional domains.
The STAR responds to rapid field growth by introducing diffusion fundamentals, organizing applications, and outlining open challenges.Its organization includes mathematical foundations, higher-dimensional generation, datasets, and metrics.

2. Scope of this STAR

This STAR surveys diffusion models for visual computing, emphasizing generation and editing beyond 2D images while defining a selected, non-exhaustive scope. It covers mathematical foundations and applications across video, 3D, and 4D data, and excludes non-diffusion generative pipelines.

The STAR focuses on diffusion-based generation and editing for images, videos, 3D objects or scenes, and multi-view consistent 4D dynamic scenes.It begins with mathematical underpinnings and emphasizes data beyond the image domain.
The report excludes works using generative pipelines other than diffusion models.
The report refers readers to related surveys for GANs, broader generative methods, foundation models, and language-model connections to text-to-image generation.
The report surveys selected papers from major computer vision, machine learning, and computer graphics venues and arXiv preprints published from 2021–2023.Selection is based on relevance to the report’s visual-computing scope.
The authors do not claim completeness and direct readers to cited works for in-depth discussion and details.

3. Fundamentals of Diffusion Models

Diffusion models generate data by gradually adding noise and then reversing that process through learned score estimates and numerical solvers. The section explains SDE-based formulations, denoising, latent diffusion, and tradeoffs among sampling strategies.

Diffusion process: Diffusion models define noisy data distributions across continuous noise levels, then generate samples by sequentially denoising an initial Gaussian noise image.For sufficiently large σmax, the noised distribution is practically indistinguishable from Gaussian noise.
Diffusion process: An Itô SDE models forward image corruption using drift and diffusion coefficients, with specific choices determining the diffusion process.The Variance Exploding SDE increases variance continuously and adds Gaussian noise with variance σ2(t).
Reverse process and score estimation: The reverse-time SDE requires the score function, which points toward higher data likelihood and can be estimated by training a denoiser with denoising score matching.The denoiser-to-score relationship is ∇x log p(x;σ) = (D(x;σ) − x)/σ2, and noise-prediction and denoising parameterizations are recoverable from one another.
Numerical sampling: Sampling starts from Gaussian noise and solves the reverse dynamics numerically; Euler–Maruyama approximates trajectories through small tangent steps.Many diffusion samplers can be viewed as modifications of Euler–Maruyama.
Numerical sampling: Higher-order solvers can reduce truncation error and maintain image quality with fewer network evaluations, while deterministic probability-flow ODE sampling can accelerate inference.Stochastic sampling often remains preferred when many denoising iterations are available and image quality is paramount, because stochasticity can correct earlier errors.
Latent diffusion and computational tradeoffs: Diffusion models require recurrent denoising passes, increasing training and inference costs compared with single-pass generative models; latent diffusion reduces cost by operating in compressed latent space.Latent diffusion retains perceptually relevant details through an encoder–decoder architecture but depends on a well-trained autoencoder.

4. The Challenge of Moving Beyond Images

Although 2D image synthesis benefits from mature architectures and abundant data, higher-dimensional video and 3D synthesis remains substantially more difficult. Key challenges include complex representations, long-range information flow, and costly or difficult-to-annotate data.

2D diffusion models benefited from mature image-processing architectures, including established denoising building blocks.
By the end of the 2010s, 2D diffusion had a mathematical framework, expressive function approximators, and abundant training data.
Higher-dimensional video and 3D synthesis is substantially more difficult than 2D image synthesis because of additional technical and data-related issues.
Models: No consensus network architecture currently exists for high-dimensional data requiring complex representations and long-range information flow.These requirements are important for modeling temporal dynamics in videos and spatial relationships in 3D structures.
Data: 3D data acquisition is time consuming and expensive, while annotating video motion and temporal dependencies is difficult.Creating a 3D model involves scanning, processing, and reconstruction, each requiring specialized expertise and resources.

5. Video Generation and Editing

Video diffusion extends image-generation models into the temporal domain, but video data requirements, computational cost, and consistency remain substantial challenges. Recent methods reuse 2D image priors, factorize spatial and temporal processing, and support increasingly controlled generation and editing across video and 3D content.

5. Video Generation and Editing: Video generation is difficult because it requires substantially more curated data than images and scales to much higher-dimensional inputs.A two-minute video at 30 fps contains 3600× more pixels than a single frame.
5. Video Generation and Editing: Video diffusion extends 2D U-Nets with temporal modules to model motion, while factorized space-time processing enables joint training on images, videos, and text.The Video Diffusion Model uses factorized space and time modules for more efficient computation.
5. Video Generation and Editing: Cascaded and pretrained-model approaches improve video-generation efficiency by combining spatio-temporal modeling with spatial and temporal super-resolution or learned 2D image priors.Imagen Video uses cascaded super-resolution models, while Make-A-Video and related methods extend pretrained text-to-image models.
5. Video Generation and Editing: Temporal convolutions and cross-frame attention are common architectural choices, but cross-frame attention promotes temporal consistency without guaranteeing it.These modules are added to residual and self-attention blocks to model dynamics across frames.
5. Video Generation and Editing: Video editing methods must preserve consistency across frames; zero-shot feature manipulation avoids single-video fine-tuning but highly consistent edits remain challenging.Single-video tuning can overfit and is computationally costly, whereas attention-based methods preserve layout and encourage consistency implicitly.

7. Towards 4D Spatio-temporal Diffusion

Diffusion models are being extended from static 3D content to motion, articulated objects, and higher-dimensional scene understanding, while controllability, physical realism, and data availability remain limiting factors.

7. Towards 4D Spatio-temporal Diffusion: Video generation often lacks long-range temporal consistency and consistent 3D structure, while 3D tools mainly model static scenes with rigid camera motion.The section motivates 4D methods that combine temporal generation with understanding of underlying 3D structure.
7. Towards 4D Spatio-temporal Diffusion: Articulated-object methods represent locally rigid deformations and generate controllable humans, animals, and other objects using parametric models, tree structures, or learned 2D priors.Examples include avatar generation with human body models, NAP’s tree parameterization, and ARTIC3D’s 2D diffusion prior.
7. Towards 4D Spatio-temporal Diffusion: Diffusion supports flexible motion editing during inference, including masked replacement of known regions, and methods such as TEDi and DoubleTake target long motion trajectories.These approaches address editability or sequence length without requiring the same type of retraining described for conventional pipelines.
7. Towards 4D Spatio-temporal Diffusion: Motion diffusion remains constrained by scarce high-quality paired datasets and cannot guarantee precise constraint satisfaction or controllability.The report identifies both data requirements and recurring physical, geometric, and control violations as open issues.
7. Towards 4D Spatio-temporal Diffusion: Motion-generation systems condition diffusion on audio, music, text, trajectories, keypoints, obstacles, or object interactions, with examples spanning dance synthesis and spatially constrained motion.Figure 10 shows audio-conditioned Locking and Krumping dance synthesis.
7. Towards 4D Spatio-temporal Diffusion: Physics-guided diffusion adds a motion-projection step that mitigates floating, foot-floor penetration, and foot sliding by enforcing constraints in a physical simulator.PhysDiff uses a human motion imitation policy to control a virtual character during projection.

8. Data

Training and evaluation data vary sharply by modality: images are abundant, whereas video, 3D, 4D, and human-motion datasets are smaller, costlier, or heterogeneous.

8. Data: Image datasets support diffusion training and validation through large scale, diversity, and semantic labels that promote generalization across image modalities.The passage emphasizes class or instance-level semantic information as useful dataset structure.
8. Data: Text–video datasets such as WebVid-10M and HD-VILA-100M contain 10M and 100M text–video pairs, respectively, far fewer than available text–image pairs.This scarcity motivates combining image and video training or using pre-trained text-to-image models.
8. Data: 3D datasets remain constrained by acquisition costs, although Objaverse and Objaverse-XL exceed 10M models with associated text corpora.Large-scale 3D collections still present heterogeneous quality, uneven text descriptions, and non-uniform distributions.
8. Data: Human-motion datasets are scarce because 3D capture is expensive, with many sizable collections based on AMASS and typically limited to several hundred captured people.Dance, indoor interaction, speech-aligned motion, and object-manipulation datasets broaden available conditions.

9. Metrics

Diffusion systems are evaluated with modality-specific measures of quality, diversity, fidelity, identity, and motion realism, but standard metrics often require human-study complements or ground truth.

9. Metrics: Image quality and diversity are commonly measured with IS, FID, and KID, while Zero-Shot FID evaluates generated images from unseen prompts against validation references.PSNR, SSIM, LPIPS, DreamSim, and CLIP provide additional comparison measures when appropriate data or embeddings are available.
9. Metrics: Video quality and diversity are often assessed with FVD and video IS on datasets such as UCF101 and MSR-VTT.FVD captures frame realism and short-segment motion but not long-term realism, and implementation differences hinder direct comparison.
9. Metrics: 3D object fidelity can be evaluated with FID or KID over renderings or latent representations, including P-FID based on PointNet++ features.The choice of representation determines whether the evaluation operates on renderings, points, or other encoded geometry.
9. Metrics: Animated and articulated objects use measures such as Instantiation Distance, physics-based realism, diversity, multimodality, and Beat Alignment Score.These metrics address part-level geometry, motion patterns, physical plausibility, and alignment with music.
9. Metrics: Human evaluations assess perceived quality, faithfulness, or pairwise preference because FID and FVD are unreliable proxies, but user studies are difficult to replicate and resource-intensive.Metric limitations motivate combining automated scores with subjective evaluation.
9. Metrics: Prompt fidelity is measured with cosine similarity between CLIP embeddings of prompts and generated content, while identity preservation uses ArcFace or CLIP similarities.These metrics target text alignment and consistency across views or against reference identities.

10. Open Challenges

Open challenges include evaluation, data, efficiency, physical grounding, reproducibility, and controllability, with progress limited by scarce higher-dimensional data and imperfect interfaces.

10. Open Challenges: Existing image and video metrics can diverge from human judgment, while ground-truth metrics need matching pairs and user studies are costly.The report calls for automated metrics applicable across images, video, 3D, and dynamic 3D scenes and aligned with human preferences.
10. Open Challenges: Captioned images are abundant, but labeled data for 3D, video, and 4D generation remains scarce, creating a quality-versus-quantity and data-efficiency trade-off.The report highlights both collecting higher-dimensional data and learning useful priors from lower-dimensional projections.
10. Open Challenges: Diffusion sampling may require up to thousands of network evaluations per result, making speed a fundamental bottleneck despite distillation and single-step methods.The unresolved goal is to generate the highest-quality output efficiently.
10. Open Challenges: Training specialized diffusion models remains difficult because most models are trained at scale by corporations with large computational resource pools.Reducing training compute is identified as an open problem for broader research access.
10. Open Challenges: Diffusion models remain difficult to control precisely: text prompts often require extensive engineering, and existing conditioning and guidance methods are not polished solutions.The report frames predictable alignment with human intentions as an open interface and modeling problem.
10. Open Challenges: Physical grounding can embed scene rules in training or network design to encourage more plausible 3D and motion generations.The challenge is especially relevant for modalities constrained by the physics of the underlying scene.
10. Open Challenges: A discrepancy persists between officially publicized results and independent reproductions, sometimes bridged through prompt, hyper-parameter, and random-seed manipulation.This motivates stronger robustness and reproducibility practices.

11. Social Implications and Ethical Concerns

Generative AI broadens access to photorealistic content creation while raising concerns about harmful content, copyright, bias, environmental costs, employment, explainability, and accountability.

Distribution of Harmful Content: Unrestricted generative AI access can enable harmful deepfakes that damage real people’s reputations.Proposed responses include content-generation safeguards and forensic detection techniques.
Copyright, Legal Exposure, and Privacy Concerns: Foundation models may incorporate scraped, copyrighted, personally identifiable, or sensitive images, creating legal and privacy exposure.Artists have already filed copyright-infringement lawsuits against companies developing visual-computing foundation models.
Bias and Fairness: Diffusion models can perpetuate training-data biases, motivating further research on mitigation methods.The report frames this as a significant fairness and ethical concern.
Environmental Concerns: Training foundation models requires substantial computational resources, creating significant environmental concerns.Stable Diffusion was reportedly trained on 2.3 billion images using 256 Nvidia A100 GPUs for 150,000 GPU-hours.
Economic Impacts: Generative AI may displace some creative and other workers while also creating new creative roles and monetization opportunities.The report highlights concern about accelerated worker displacement as generative systems become integrated across sectors.
Explainability, Trust, and Accountability: Because generative models learn correlations rather than causality and can synthesize untruthful data, explainability, trust, and accountability remain difficult.These issues can also have legal consequences, requiring dialogue among policymakers, industry, and researchers.

12. Discussion and Conclusion

The report synthesizes diffusion models’ theory, practice, applications, evaluation resources, open challenges, and social implications. Despite providing an intuitive starting point, it emphasizes that the rapidly evolving field still has substantial challenges ahead.

Discussion and Conclusion: The STAR reviews diffusion-model mathematics, implementation choices, finetuning, sampling, conditioning, inversion, applications, datasets, metrics, challenges, and social implications.Its coverage is organized by the type of generated medium.
Discussion and Conclusion: The report aims to provide an intuitive starting point for artists, practitioners, and researchers exploring this rapidly changing field.The authors note that papers and commercial models appear weekly or daily.
Discussion and Conclusion: Many open challenges remain, including the broader goal of helping novice and advanced users intuitively turn imagination into images, videos, or 3D scenes.The report describes current generative AI tools as a major step forward but says substantial work remains.