PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Resolution Image Synthesis with Latent Diffusion Models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer

arXiv:2112.10752v2cs.CV

TL;DR

Pixel-space diffusion models make high-resolution synthesis computationally demanding because training and inference require costly sequential evaluations. This paper moves diffusion into a perceptually equivalent autoencoder latent space and adds cross-attention conditioning, achieving efficient sampling and favorable results across conditional image-synthesis tasks.

  • Problem

    Pixel-space diffusion models require costly sequential evaluations, making high-resolution training and inference computationally demanding.

  • Method

    The approach separates compression from generation by training diffusion models in a computationally reduced, perceptually equivalent autoencoder latent space with cross-attention conditioning.

  • Results

    LDMs significantly improve training and sampling efficiency without degrading quality, achieving favorable results across conditional image-synthesis tasks while lowering computational costs.

  • Takeaways & Limitations

    Latent diffusion provides an efficient, general framework for conditional image synthesis without task-specific architectures.

  • Takeaways & Limitations

    LDM sampling remains slower than GAN sampling, and autoencoder reconstruction can bottleneck tasks requiring fine-grained pixel accuracy.

Abstract

from arXiv · show

By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .

1. Introduction

Diffusion models deliver strong, flexible image synthesis but remain computationally expensive, especially for high-resolution generation. The paper addresses this by moving diffusion modeling into efficient autoencoder latent spaces while preserving fidelity and enabling broad conditioning and high-resolution synthesis.

  • Motivation: Diffusion models achieve state-of-the-art image synthesis and support tasks including class-conditional generation, super-resolution, inpainting, colorization, and stroke-based synthesis.Their flexibility extends beyond unconditional generation to multiple forms of image conditioning and manipulation.
  • Motivation: 50k samples take approximately 5 days on a single A100 GPU, while training and sequential inference demand substantial computational resources.These costs restrict accessibility, increase time and memory requirements, and contribute to a large carbon footprint.
  • Method: The proposed approach trains diffusion models in lower-dimensional, perceptually equivalent autoencoder latent spaces, avoiding excessive spatial compression and reducing computational complexity.The autoencoder is trained once and reused across multiple diffusion-model trainings and tasks.
  • Contributions: Latent diffusion models provide more faithful reconstructions, lower computational and inference costs, and competitive performance across unconditional synthesis, inpainting, and stochastic super-resolution.The method is also designed for high-resolution synthesis of megapixel images.

2. Related Work

Prior image-synthesis methods trade off sampling efficiency, optimization, density estimation, sample quality, or computational cost. The proposed LDMs address two-stage compression tradeoffs through convolutional scaling in higher-dimensional latent spaces while preserving high-fidelity reconstructions.

  • Generative Models for Image Synthesis: GANs sample high-resolution images efficiently with good perceptual quality but are difficult to optimize and struggle to capture the full data distribution.Likelihood-based methods optimize density estimation more tractably, while VAEs and flow-based models synthesize high-resolution images efficiently but produce lower-quality samples than GANs.
  • Generative Models for Image Synthesis: Diffusion models achieve state-of-the-art density estimation and sample quality, leveraging UNet inductive biases for image-like data.Their best synthesis quality is usually obtained with a reweighted training objective.
  • Two-Stage Image Synthesis: Two-stage approaches combine different generative methods, but VQ-VAE systems face a compression-versus-computation tradeoff.High compression enables autoregressive training but introduces billions of trainable parameters, whereas less compression increases computational cost.
  • Two-Stage Image Synthesis: LDMs scale more gently to higher-dimensional latent spaces through a convolutional backbone, enabling compression levels that balance first-stage learning, perceptual compression, and reconstruction fidelity.This design is intended to preserve high-fidelity reconstructions without placing excessive perceptual compression on the diffusion model.
  • Two-Stage Image Synthesis: Jointly learned encoding-decoding models require difficult reconstruction–generation weighting, while separately learned score-based priors focus on highly structured images such as human faces.The passage states that the joint approach is outperformed by the proposed method.

3. Method

The method separates perceptual compression from generative learning by training diffusion models in a low-dimensional latent space produced by a pretrained autoencoder. A UNet backbone and cross-attention conditioning enable efficient, flexible image synthesis from modalities such as text and semantic maps.

  • Perceptual Compression: An autoencoding model learns a perceptually equivalent image representation with substantially lower computational complexity, separating compression from generative learning.This avoids costly diffusion-model evaluations directly in high-dimensional pixel space.
  • Perceptual Compression: Perceptual and patch-based adversarial losses constrain reconstructions to the image manifold and avoid bluriness associated with pixel-space L2 or L1 losses.Latent regularization uses either a KL penalty toward a standard normal or vector quantization in the decoder.
  • Perceptual Compression: The encoder maps x to z = E(x), while the decoder reconstructs x̃ = D(E(x)) from a latent z ∈ R^h×w×c downsampled by f = H/h = W/w.The method investigates downsampling factors f = 2^m, with m ∈ N.
  • Latent Diffusion: Diffusion operates in the efficient latent space, abstracting away imperceptible high-frequency details so likelihood-based models focus on semantic content and train with reduced computational demands.The model uses image-specific inductive biases, including a primarily 2D-convolutional UNet, while decoding samples with one pass through D.
  • Conditioning Mechanism: Cross-attention augments the UNet to condition latent diffusion on varied modalities, with a domain-specific encoder τθ projecting inputs y into representations mapped into intermediate UNet layers.The conditioning encoder and denoising network are jointly optimized, and τθ can use domain-specific experts such as unmasked transformers for text prompts.

4. Experiments

Experiments show that LDMs achieve a favorable trade-off between perceptual compression, sample quality, sampling speed, and computational cost. Across unconditional, conditional, super-resolution, and inpainting tasks, they attain competitive or state-of-the-art results while reducing resource requirements.

  • Unconditional synthesis: 5.11 FID on CelebA-HQ establishes a new state of the art for unconditional image synthesis, while LDMs outperform prior diffusion approaches on all but LSUN-Bedrooms.On LSUN-Bedrooms, the score is close to ADM despite using half its parameters and requiring 4-times less training resources.
  • Unconditional synthesis: LDMs consistently improve Precision and Recall over GAN-based methods, supporting stronger coverage of the data manifold through likelihood-based training.The experiments evaluate unconditional models on CelebA-HQ, FFHQ, LSUN-Churches, and LSUN-Bedrooms using FID and Precision-and-Recall.
  • Conditional synthesis: A 1.45B-parameter text-conditional LDM trained on LAION-400M uses transformer-derived language codes and UNet cross-attention to support text-to-image synthesis.With classifier-free guidance, the text-conditional LDM-KL-8-G directly synthesizes images larger than 256^2 pixels.
  • Conditional synthesis: On class-conditional ImageNet, the best LDMs with f ∈{4, 8} outperform ADM while significantly reducing computational requirements and parameter count.The comparison is reported in Table 3, Figure 4, and supplementary Table 18.
  • Super-resolution and inpainting: LDM-SR outperforms SR3 in FID for 4× bicubic ImageNet super-resolution, although SR3 achieves better IS.LDM-SR uses the f = 4 VQ-regularized autoencoder pretrained on OpenImages and concatenates low-resolution conditioning with the UNet input.

5. Limitations & Societal Impact

LDMs reduce computational requirements but remain slower than GANs and can bottleneck pixel-precise tasks. Their broader accessibility also raises concerns about misuse, privacy, bias, and unresolved ethical effects.

  • Limitations: LDM sequential sampling remains slower than GAN inference despite substantially reducing computational requirements compared with pixel-based approaches.Their f = 4 autoencoding models show very small image-quality loss, but reconstruction can bottleneck tasks requiring fine-grained pixel accuracy.
  • Societal Impact: Lower training and inference costs may democratize generative technology while facilitating manipulated data, misinformation, spam, and deepfakes that disproportionately affect women.The passage characterizes generative media models as having both creative benefits and harmful dissemination risks.
  • Societal Impact: Generative models can reveal training data, posing privacy risks for sensitive or personal information collected without explicit consent, though applicability to image DMs remains unclear.The extent to which this concern applies specifically to diffusion models of images is not yet fully understood.
  • Societal Impact: Deep learning models may reproduce or exacerbate dataset biases, while the extent of misrepresentation from LDMs’ two-stage training remains an important research question.Diffusion models provide better data-distribution coverage than GAN-based approaches, but the effects of combining adversarial training with a likelihood-based objective remain unresolved.

6. Conclusion … B. Detailed Information on Denoising Diffusion Models

The paper concludes that latent diffusion models improve diffusion-model training and sampling efficiency without degrading quality, while cross-attention enables strong conditional synthesis across tasks. The appendices and changelog document scalable convolutional sampling, model updates, added evaluation, and the mathematical formulation of denoising diffusion models.

  • 6. Conclusion: Latent diffusion models significantly improve denoising diffusion models’ training and sampling efficiency without degrading quality.Cross-attention conditioning supports favorable results across diverse conditional image-synthesis tasks without task-specific architectures.
  • Appendix: Convolutional sampling produces semantic-landscape samples and enables a 1.45B-parameter text-to-image model to render images beyond its native 2562 resolution.The text-to-image model was trained at 2562 resolution.
  • A. Changelog: 1.45B parameters updated the text-to-image synthesis results and added comparisons with recent competing methods.The updated results came from training a new, larger model.
  • A. Changelog: Larger-batch retraining updated ImageNet class-conditional results, while classifier-free guidance was added to both updated text-to-image and class-conditional models.Corresponding qualitative results in Figures 26 and 27 were also updated.
  • A. Changelog: A user study added evaluation for the inpainting and superresolution models.The study followed the scheme suggested by Saharia et al.
  • B. Detailed Information on Denoising Diffusion Models: Diffusion models are specified through a signal-to-noise ratio and a forward Markov diffusion process, which denoising diffusion models reverse with a backward Markov structure.The forward process starts from a data sample x0 and proceeds over discrete time steps.
  • B. Detailed Information on Denoising Diffusion Models: The ELBO decomposes over discrete time steps, using a standard-normal prior and an estimated xθ(xt, t) in place of the unknown x0.The reverse transition is parameterized using the true posterior with x0 replaced by the current-step estimate.
  • B. Detailed Information on Denoising Diffusion Models: Reparameterization expresses the reconstruction term as a denoising objective, while equal reweighting assigns the ELBO terms the same weight and yields Eq. (1).The derivation follows the stated reparameterization and reweighting procedure.

C. Image Guiding Mechanisms … D.4. Class-Conditional Image Synthesis on ImageNet

The paper extends test-time diffusion guidance from classifier conditioning to general image-to-image translation, including upsampling and perceptual guidance. Additional experiments analyze latent-space scaling, layout-to-image synthesis, and class-conditional ImageNet generation, emphasizing competitive results with reduced computational demands.

  • C. Image Guiding Mechanisms: Diffusion models can be conditioned at test time, and the paper introduces post-hoc image-guiding beyond classifier-based conditioning.The approach builds on classifier guidance applied to diffusion trajectories.
  • C. Image Guiding Mechanisms: The guiding distribution is reinterpreted as a general-purpose image-to-image translation objective using a target image and a differentiable transformation T.T may be the identity, downsampling, or another task-specific transformation.
  • C. Image Guiding Mechanisms: A 256^2 unconditional model guides convolutional synthesis of 512^2 images through 2× bicubic downsampling, using either L2 or perceptual LPIPS guidance.The Gaussian guider with fixed variance σ^2 = 1 yields an L2 regression objective; LPIPS replaces L2 in an additional experiment.
  • D.1. Choosing the Signal-to-Noise Ratio for High-Resolution Synthesis: Latent-space signal-to-noise ratio substantially affects convolutional sampling, with high ratios allocating more semantic detail early in reverse denoising.Component-wise standard-deviation rescaling decreases the latent SNR.
  • D.2. Full List of all First Stage Models: The paper provides a complete list of autoencoding models trained on OpenImages and evaluated on ImageNet-Val, including attention-free variants.Table 8 denotes attention-free autoencoders with †.
  • D.3. Layout-to-Image Synthesis: For layout-to-image synthesis, models are trained on COCO and OpenImages, with additional COCO finetuning; the COCO model matches recent state of the art, while finetuning surpasses it.The quantitative comparison follows the cited works’ training and evaluation protocol.
  • D.4. Class-Conditional Image Synthesis on ImageNet: The class-conditional ImageNet LDM-8 achieves very competitive performance with significantly fewer parameters and compute, and classifier guidance can further improve results.The classifier is trained cheaply in latent space, following the guidance procedure described in Section C.

D.5. Sample Quality vs. V100 Days (Continued from Sec. 4.1) … E.2. Implementation Details

The paper evaluates LDM quality against training resources, demonstrates strong and generalizable super-resolution behavior under diverse degradations, and documents the trained models’ hyperparameters. Across these analyses, latent-space diffusion offers comparable or better performance with faster sampling, while fixed degradation limits real-world generalization.

  • D.5. Sample Quality vs. V100 Days (Continued from Sec. 4.1): Reporting FID and IS over V100 days produces qualitatively similar training-progress trends to reporting them over train steps.Figure 17 evaluates class-conditional ImageNet LDMs for 35 V100 days using 100 DDIM steps and κ = 0.
  • D.5. Sample Quality vs. V100 Days (Continued from Sec. 4.1): An LDM trained with matched steps and comparable parameters achieves better performance than a pixel-space diffusion model while sampling significantly faster.The comparison appears in the last two rows of Table 11; exact architectural matching is impossible because the diffusion model operates in pixel space.
  • D.6.1 LDM-BSR: General Purpose SR Model via Diverse Image Degradation: LDM-BSR generalizes to arbitrary inputs and functions as a general-purpose upsampler, including scaling class-conditional LDM samples to 1024^2 resolution.Using a fixed degradation process hinders generalization.
  • D.6. Super-Resolution: LDM-SR is evaluated on synthetic class-conditional ImageNet samples and internet images to assess generalization across input types.The evaluation reveals that bicubicly conditioned LDM-SR does not generalize well to images that do not follow that preprocessing.
  • D.6.1 LDM-BSR: General Purpose SR Model via Diverse Image Degradation: Because real-world images can contain complex combinations of camera noise and compression, diverse degradation is needed for a broadly applicable super-resolution model.Training only with bicubicly downsampled conditioning does not generalize well beyond that preprocessing.
  • E. Implementation Details and Hyperparameters: The paper provides hyperparameter overviews for all trained LDM models in Tables 12–15.Tables 12 and 13 cover unconditional and ImageNet-conditional LDMs, respectively, with all models trained on a single NVIDIA A100.
  • E.1. Hyperparameters: Tables 12 and 13 specify hyperparameters for unconditional LDMs and ImageNet-conditional LDMs, and both model groups were trained on a single NVIDIA A100.The unconditional models correspond to Table 1, while the conditional models support the analysis in Section 4.1.

E.2.1 Implementations of τθ for conditional LDMs · E.2.2 Inpainting · E.3. Evaluation Details

Conditional LDMs encode text, layouts, and classes with dedicated conditioners and inject the resulting representations through cross-attention in the UNet. Inpainting evaluation follows a fixed Places validation/testing protocol with specified crop sizes, while E.3 introduces additional evaluation details for Sec. 4 experiments.

  • E.2.1 Implementations of τθ for conditional LDMs: For text-to-image and layout-to-image synthesis, τθ is an unmasked transformer that maps tokenized inputs y to ζ := τθ(y) ∈ R^M×dτ.It consists of N transformer blocks with global self-attention, layer normalization, and position-wise MLPs.
  • E.2.1 Implementations of τθ for conditional LDMs: The conditioner representation ζ is injected into the UNet through cross-attention using T-block transformers with self-attention, position-wise MLP, and cross-attention layers.This modifies the ablated UNet architecture by replacing its self-attention layer.
  • E.2.1 Implementations of τθ for conditional LDMs: The implementation omits timestep conditioning of τθ because it would reduce inference speed, leaving analysis of that modification for future work.The paper notes that timestep conditioning could increase τθ’s representational power.
  • E.2.1 Implementations of τθ for conditional LDMs: The text-to-image model uses a publicly available tokenizer, while the layout-to-image model encodes each bounding box as a discretized (l, b, c)-tuple containing positions and class information.The class-conditional model instead uses a learnable 512-dimensional embedding layer mapping classes y to ζ ∈ R^1×512.
  • E.2.2 Inpainting: Inpainting experiments use synthetic masks generated with code from [88] and fixed Places sets containing 2k validation and 30k testing samples.The protocol follows [88] and reproduces its reported metrics.
  • E.2.2 Inpainting: Inpainting trains on random 256 × 256 crops and evaluates on 512 × 512 crops, following the training and testing protocol in [88].Additional qualitative results are provided for LDM-4 variants with and without attention.
  • E.3. Evaluation Details: The evaluation details section provides additional information for the experiments presented in Sec. 4.

E.3.1 Quantitative Results in Unconditional and Class-Conditional Image Synthesis … E.3.6 User Study

The paper evaluates synthesis quality with task-specific protocols, standardized sample comparisons, and human preference testing. It also reports metric sensitivity to evaluation pipelines and sample counts, alongside efficiency-oriented analyses.

  • E.3.1 Quantitative Results in Unconditional and Class-Conditional Image Synthesis: FID, Precision, and Recall are estimated from 50k generated samples and each dataset’s entire training set, using torch-fidelity and an additional evaluation script.The authors note that the two pipelines mainly coincide and emphasize the importance of unified sample-quality assessment.
  • E.3.1 Quantitative Results in Unconditional and Class-Conditional Image Synthesis: ImageNet FID scores are 7.76 with torch-fidelity versus 7.77 with the Nichol-and-Dhariwal script, while LSUN-Bedrooms scores are 2.95 versus 3.0.These are the datasets where the evaluation pipelines produce slightly varying scores.
  • E.3.2 Text-to-Image Synthesis: Text-to-Image models are evaluated on MS-COCO by comparing generated samples with 30000 validation-set samples using FID and Inception Score.Both metrics are computed with torch-fidelity following the protocol of [66].
  • E.3.3 Layout-to-Image Synthesis: Layout-to-Image FID is computed against 2048 unaugmented COCO Segmentation Challenge examples and 2048 center-cropped OpenImages validation images.For COCO, the evaluation uses the exact same samples as [37] for comparability.
  • E.3.4 Super Resolution: Super-resolution models are evaluated on ImageNet after removing images shorter than 256 px and generating low-resolution inputs with anti-aliased bicubic interpolation.FID uses torch-fidelity, validation samples, and reference features computed on the training split.
  • E.3.5 Efficiency Analysis: Efficiency-analysis metrics use 5k samples, so results may differ from Tables 1 and 10; compared models have comparable parameter counts and independently maximized stable learning rates.Learning rates therefore vary slightly between runs.
  • E.3.6 User Study: The user study follows [72] and uses a 2-alternative force-choice paradigm to measure human preferences across two tasks.In Task-1, subjects compare a synthesized image conditioned on a low-resolution or masked image with the corresponding ground-truth high-resolution or unmasked image.

F. Computational Requirements · G. Details on Autoencoder Models

The paper compares computational requirements and sample quality against recent generative models, converting A100 training time to V100-days using a 2.2× speedup assumption. Its autoencoders use adversarial training with latent regularization, while diffusion training handles KL- and VQ-regularized latent spaces differently.

  • F. Computational Requirements: The authors compare their best CelebA-HQ, FFHQ, LSUN, and ImageNet models with recent state-of-the-art methods using reported compute figures and FID scores.Their A100 training times are converted to V100-days for comparison.
  • F. Computational Requirements: A 2.2× A100-versus-V100 speedup is assumed when converting training time into V100-days.The factor corresponds to U-Net speedup measurements reported in.
  • G. Details on Autoencoder Models: The autoencoders are trained adversarially with a patch-based discriminator distinguishing original images from reconstructions D(E(x)).The discriminator is optimized according to the adversarial training procedure in [23].
  • G. Details on Autoencoder Models: A regularizing loss Lreg centers latent z around zero and keeps its variance small, avoiding arbitrarily scaled latent spaces.The paper investigates two regularization methods, including a low-weighted Kullback–Leibler term.
  • G. Details on Autoencoder Models: The full autoencoding objective is introduced for jointly training the encoder E and decoder D.The supplied passage states the objective but does not include its expanded formula.
  • G. Details on Autoencoder Models: For KL-regularized latents, diffusion training samples z as Eµ(x)+Eσ(x)·ε with ε ∼ N(0, 1), while rescaling uses component-wise variance estimates.The latent is sampled from the encoder’s mean and variance parameterization.
  • G. Details on Autoencoder Models: For VQ-regularized latents, diffusion training extracts z before quantization and incorporates quantization into the decoder as its first layer.This treats the quantization operation as part of D.

H. Additional Qualitative Results

Additional qualitative results demonstrate that LDMs support high-resolution semantic synthesis, general-purpose upsampling, class-conditional ImageNet generation, and strong unconditional samples across face and scene datasets. These visual results include resolution generalization beyond training sizes and reported FID values for several best-performing models.

  • General-Purpose Upsampling: LDM-BSR generalizes to arbitrary inputs as a general-purpose upsampler, including LSUN-Cows samples upscaled to 1024^2 resolution.The result is shown in Figure 19.
  • Semantic Synthesis: Semantic landscapes models trained at 256^2 generalize to substantially larger resolutions, producing spatially conditioned samples at 1024 × 384.The model is fine-tuned on 512^2 images for additional qualitative results and can generate larger-resolution landscape samples.
  • Class-Conditional ImageNet: Class-conditional LDM-4 ImageNet samples are shown with classifier-free guidance scales s = 5.0 and s = 3.0 using 200 DDIM steps and η = 1.0.Figures 26 and 27 present random samples under the two guidance scales.
  • Unconditional Generation: The best-performing unconditional models achieve FID = 5.15 on CelebA-HQ, FID = 4.98 on FFHQ, FID = 4.48 on LSUN-Churches, and FID = 2.95 on LSUN-Bedrooms.These samples use LDM-4 except for LSUN-Churches, which uses LDM-8, with 200 or 500 DDIM steps depending on the dataset.
Loading 2112.10752v2…