PaperlayerView exact revision on arXiv

Source-linked AI summary

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, Robin Rombach

arXiv:2307.01952v1cs.CVcs.AI

TL;DR

Text-to-image diffusion models faced limitations in image-size handling, composition, and visual quality. SDXL addresses these with a larger backbone, added conditioning, multi-aspect training, improved autoencoding, and an optional refinement model, with refined SDXL preferred over earlier Stable Diffusion versions in user studies.

  • Problem

    Latent diffusion models struggle with variable image sizes and aspect ratios, while previous Stable Diffusion models can produce cropped synthesized objects.

  • Method

    SDXL combines a 3× larger UNet, additional conditioning, multi-aspect training, an improved autoencoder, and a separate diffusion-based refinement model.

  • Results

    48.44% of users preferred SDXL with refinement, compared with 36.93% for base SDXL, 7.91% for Stable Diffusion 1.5, and 6.71% for Stable Diffusion 2.1.

  • Takeaways & Limitations

    SDXL provides an open text-to-image model with performance competitive with black-box image generation models.

  • Takeaways & Limitations

    The best SDXL samples require a two-stage pipeline that loads two large models, reducing accessibility and sampling speed.

Abstract

from arXiv · show

We present SDXL, a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder. We design multiple novel conditioning schemes and train SDXL on multiple aspect ratios. We also introduce a refinement model which is used to improve the visual fidelity of samples generated by SDXL using a post-hoc image-to-image technique. We demonstrate that SDXL shows drastically improved performance compared the previous versions of Stable Diffusion and achieves results competitive with those of black-box state-of-the-art image generators. In the spirit of promoting open research and fostering transparency in large model training and evaluation, we provide access to code and model weights at https://github.com/Stability-AI/generative-models

1 Introduction

The report introduces SDXL, a drastically improved latent text-to-image diffusion model, and describes design choices behind its performance gains. It also emphasizes open access as a response to the reproducibility and evaluation challenges posed by opaque black-box models.

  • Introduction: SDXL is presented as a drastically improved version of Stable Diffusion, a latent text-to-image diffusion model.The report situates SDXL within recent advances in deep generative modeling across natural language, audio, and visual media.
  • Introduction: User studies show SDXL consistently surpasses previous Stable Diffusion versions by a significant margin.The reported design choices include a 3× larger UNet backbone, two additional conditioning techniques without additional supervision, and a separate diffusion-based refinement model.
  • Introduction: Opaque black-box visual-media models hinder faithful performance assessment, validation, reproducibility, innovation, and community progress.The introduction identifies transparency as a major concern in visual media creation.

2 Improving Stable Diffusion

SDXL improves Stable Diffusion through a redesigned UNet and text-conditioning setup, plus conditioning on image size, cropping parameters, and diverse aspect ratios. It also improves local detail and sample quality with a stronger autoencoder and a specialized refinement stage.

  • Architecture: SDXL shifts transformer computation to lower-level UNet features, using 2 and 10 blocks at lower levels while omitting the highest-level block and lowest 8×-downsampling level.This heterogeneous distribution is designed for efficiency.
  • Text conditioning: SDXL combines OpenCLIP ViT-bigG with CLIP ViT-L, concatenates their penultimate outputs, and additionally conditions on the pooled OpenCLIP embedding.These changes produce a model size of 2.6B parameters.
  • Image-size conditioning: Conditioning on original image height and width lets SDXL retain varied-size training images and allows users to set the desired apparent resolution at inference.The height and width are independently Fourier-embedded and added to the timestep embedding.
  • Cropping conditioning: Conditioning on cropping parameters turns random-cropping artifacts into controllable augmentation, enabling inference-time control over how much of an image is cropped.The method is easy to implement online during training and avoids additional data preprocessing.
  • Aspect-ratio training: SDXL supports the varied landscape and portrait aspect ratios found in real-world data rather than restricting generation to square 512 × 512 or 1024 × 1024 outputs.The paper motivates multi-aspect training by the widespread use of formats such as 16:9.
  • Autoencoder and refinement: A larger-batch autoencoder improves local high-frequency detail, while a separate high-resolution refinement LDM applies SDEdit-based noising and denoising to base-model samples.The autoencoder uses batch-size 256 vs 9, and the refinement model specializes in the first 200 discrete noise scales.
  • Evaluation: 48.44% was the user-study win rate for SDXL with refinement, compared with 36.93% for SDXL base, 7.91% for Stable Diffusion 1.5, and the reported Stable Diffusion 2.1 result.The refinement-stage model was the highest-rated choice and outperformed Stable Diffusion 1.5 and 2.1 by a significant margin.

3 Future Work · Appendix

The report presents a preliminary analysis of Stable Diffusion improvements while identifying remaining areas for future improvement. Current limitations include the two-stage sampling setup’s memory and speed costs and the need to further improve text rendering.

  • 3 Future Work: SDXL improves synthesized image quality, prompt adherence, and composition, but the report characterizes these results as preliminary.The authors identify additional aspects in which the model may be improved further.
  • 3 Future Work: The report frames further model development around improving capabilities that remain incomplete despite significant gains.The stated areas include image quality, prompt adherence, composition, and text rendering.
  • 3 Future Work: The strongest SDXL samples currently require a two-stage pipeline with an additional refinement model.This approach generates the best samples described in the passage.
  • 3 Future Work: Loading two large models hampers accessibility and sampling speed.These costs arise from the current two-stage generation setup.
  • 3 Future Work: Future work should seek a single-stage system with equal or better quality.The proposed direction is to avoid the current two-stage requirement without sacrificing quality.
  • 3 Future Work: Scaling and the larger OpenCLIP ViT-bigG text encoder improve text-rendering capabilities over previous Stable Diffusion versions.The passage attributes improved text rendering to both model scale and the larger text encoder.

B Limitations

SDXL still has limitations in handling complex structures and prompts, achieving perfect photorealism, separating concepts, rendering long text, and avoiding social and racial biases. These shortcomings identify areas for further research and responsible use.

  • Complex structures and prompts: SDXL sometimes struggles with complex prompts involving detailed spatial arrangements and with accurately generating human hands.The paper suggests that fine-grained anatomical details may require further scaling and targeted training techniques.
  • Photorealism: Despite its realism, SDXL does not achieve perfect photorealism because subtle lighting effects and minute texture variations may be missing or less faithful.This limitation warrants caution in applications requiring high visual fidelity.
  • Bias: Large-scale training datasets can introduce social and racial biases that SDXL may exacerbate when generating images or inferring visual attributes.The limitation arises from the model’s reliance on large-scale datasets.
  • Concept bleeding: SDXL may exhibit concept bleeding, unintentionally merging or swapping attributes among distinct objects or subjects in complex scenes.The paper attributes this potentially to pretrained text encoders that compress information into a single token and may fail to bind attributes to the correct objects.
  • Text rendering: SDXL still has difficulty rendering long, legible text, which may contain random characters or inconsistencies.The paper identifies improved text-generation techniques for extended textual content as a direction for future work.

C Diffusion Models · D Comparison to the State of the Art · E Comparison to Midjourney v5.1

The paper summarizes continuous-time diffusion models as sequential denoising from high-variance Gaussian noise and describes numerical sampling, score-model training, and classifier-free guidance. It also presents a qualitative SDXL comparison with several state-of-the-art image generators using randomly selected prompts and controlled seeds where available.

  • C Diffusion Models: Diffusion models sample high-variance Gaussian noise and sequentially denoise it through decreasing noise levels until the result follows the data distribution.The process uses x_M ∼ N(0, σ_max^2), with σ_i < σ_i+1 and σ_0 = 0 for a well-trained model.
  • C Diffusion Models: Sampling can numerically simulate either the Probability Flow ODE or an SDE, which in principle produce samples from the same distribution.The Probability Flow ODE is introduced as a practical implementation of iterative denoising, while the SDE provides an alternative.
  • C Diffusion Models: Training learns a noise-conditioned score model that estimates ∇x log p(x; σ), optionally through a denoiser predicting clean data from noisy observations.The denoiser parameterization is sθ(x; σ) = (Dθ(x; σ) − x)/σ^2, and denoising score matching is used for training.
  • C Diffusion Models: The implementation uses a discrete distribution over 1000 noise levels with weighting function λσ = σ−2, while conditioning may include class labels or text prompts.The conditioning signal c can be a class label, a text prompt, or a combination thereof.
  • C Diffusion Models: Classifier-free guidance steers sampling by mixing conditional and unconditional predictions, improving sampling quality while trading off diversity.The unconditional branch can be trained jointly by replacing the conditioning signal with a null embedding, for example 10% of the time.
  • D Comparison to the State of the Art: SDXL is qualitatively compared with DeepFloyd IF, DALLE-2, Bing Image Creator, and Midjourney v5.2 using randomly selected Parti prompts to reduce cherry-picking bias.Seed 3 was applied uniformly where possible; otherwise, the first generated image was used.

E.1 Overall Votes

A user study on PartiPrompts compared SDXL with Midjourney v5.1 using AWS GroundTruth votes for prompt adherence. Across 17,153 comparisons, SDXL was favored 54.9% of the time, indicating a slight overall preference for SDXL.

  • Study design: AWS GroundTruth voters evaluated prompt adherence on images generated from randomly sampled PartiPrompts prompts by SDXL and Midjourney v5.1.The study used five random prompts per category and four 1024 × 1024 images per prompt from each model.
  • Overall preference: 17,153 user preference comparisons found SDXL favored 54.9% of the time over Midjourney v5.1.The comparisons covered all categories and challenges in the PartiPrompts benchmark.

E.2 Category & challenge comparisons on PartiPrompts (P2)

On the PartiPrompts benchmark, SDXL compares favorably with Midjourney V5.1 across both text categories and complex-prompt challenges. SDXL outperforms Midjourney in four of six categories and is equal to or better than it in seven of ten challenges.

  • Category comparisons: SDXL outperforms Midjourney in four out of six PartiPrompts categories.The comparison uses SDXL without the refinement model.
  • Evaluation setup: The comparisons use Midjourney V5.1 because it was the best available version at that time.The benchmark organizes each prompt into a category and a challenge targeting different difficult aspects of generation.
  • Challenge comparisons: SDXL either outperforms or is statistically equal to Midjourney V5.1 in seven out of ten complex-prompt challenges.This comparison uses SDXL with the refinement model.

F On FID Assessment of Generative Text-Image Foundation Models · G Additional Comparison between Single- and Two-Stage SDXL pipeline

The section questions whether zero-shot FID and CLIP scores on small natural-image datasets adequately assess foundation text-to-image models. It also contrasts SDXL samples with and without the refinement model while reporting that human preferences diverge from standard metrics.

  • F On FID Assessment of Generative Text-Image Foundation Models: SDXL shows only slightly improved text alignment by CLIP score compared with previous versions.The figure notes that this metric-based improvement does not align with human evaluator judgments.
  • F On FID Assessment of Generative Text-Image Foundation Models: FID scores for SDXL are worse than those for both SD-1.5 and SD-2.1.This comparison is reported across different classifier-free guidance scales.
  • F On FID Assessment of Generative Text-Image Foundation Models: Human evaluators clearly prefer SDXL generations over those of SD-1.5 and SD-2.1 despite the reported FID results.The figure therefore presents a divergence between FID rankings and human evaluation.
  • F On FID Assessment of Generative Text-Image Foundation Models: Zero-shot FID and CLIP scores are commonly measured on complex, small-scale natural-image datasets such as COCO.This practice was established for generative text-to-image models.
  • F On FID Assessment of Generative Text-Image Foundation Models: Foundation text-to-image models target visual compositionality alongside deep text understanding and fine-grained artistic-style distinctions.These broader capabilities motivate reassessing conventional evaluation datasets and metrics.
  • G Additional Comparison between Single- and Two-Stage SDXL pipeline: Figure 13 compares SDXL samples generated without and with the refinement model.The examples include a futuristic-woman headshot prompt and a New Year’s Eve dinner scene.

H Comparison between SD 1.5 vs. SD 2.1 vs. SDXL

The section compares SDXL with previous Stable Diffusion versions through additional qualitative output results. For each prompt, three random samples are shown using 50 DDIM sampling steps and cfg-scale 8.0.

  • Qualitative comparison: The figures compare SDXL outputs with previous versions of Stable Diffusion across multiple prompts.The displayed prompts include a vibrant Salvador Dalí portrait with a robotic half face and a voxel-made capybara.
  • Evaluation setup: 50 DDIM steps and cfg-scale 8.0 define the sampling setup for the comparison.Each prompt includes three random samples from each respective model.

I Multi-Aspect Training Hyperparameters

The section specifies image resolutions for mixed-aspect-ratio finetuning. The associated table organizes these settings by height, width, and aspect ratio.

  • Multi-Aspect Training Hyperparameters: Mixed-aspect-ratio finetuning uses a specified set of image resolutions.The passage introduces these resolutions as the settings used for finetuning.
  • Multi-Aspect Training Hyperparameters: The resolution settings are organized by height and width.These are the first two columns identified in the table.
  • Multi-Aspect Training Hyperparameters: Each resolution setting is associated with an aspect ratio.Aspect ratio is identified as a table column alongside height and width.

J Pseudo-code for Conditioning Concatenation along the Channel Axis

This section provides Python pseudo-code for concatenating the additional conditioning signals introduced in Sections 2.1–2.3 along the channel dimension.

  • Conditioning Concatenation: Figure 16 presents Python code that concatenates the additional conditionings introduced in Sections 2.1 to 2.3 along the channel dimension.The code illustrates the conditioning-concatenation procedure.
Loading 2307.01952v1…