PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Visual Generation with Composable Diffusion Models

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, Joshua B. Tenenbaum

arXiv:2206.01714v6cs.CVcs.AIcs.LG

TL;DR

Text-guided diffusion models struggle with complex compositions such as binding attributes to the correct objects. The paper composes diffusion models by interpreting them as energy-based models, enabling zero-shot generation of more complex combinations across several domains. The method generates images containing the requested concepts and generalizes to combinations rarely seen in the real world, while remaining limited when combining models trained on different datasets.

  • Problem

    Text-conditioned diffusion models can generate photorealistic images but fail on complex descriptions involving object attributes and relations.

  • Method

    The method factorizes compositional specifications across diffusion models and explicitly composes them using an energy-based-model interpretation.

  • Results

    The method generates images conditioned on sentence descriptions, objects, object relations, and facial attributes, including combinations substantially more complex than those seen during training.

  • Takeaways & Limitations

    Composable diffusion models support zero-shot combinatorial generalization and generate images containing multiple requested concepts without additional training.

  • Takeaways & Limitations

    The approach has limited success composing diffusion models trained on different datasets because its composable models are instances of the same model.

Abstract

from arXiv · show

Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions. While such models are highly flexible, they struggle to understand the composition of certain concepts, such as confusing the attributes of different objects or relations between objects. In this paper, we propose an alternative structured approach for compositional generation using diffusion models. An image is generated by composing a set of diffusion models, with each of them modeling a certain component of the image. To do this, we interpret diffusion models as energy-based models in which the data distributions defined by the energy functions may be explicitly combined. The proposed method can generate scenes at test time that are substantially more complex than those seen in training, composing sentence descriptions, object relations, human facial attributes, and even generalizing to new combinations that are rarely seen in the real world. We further illustrate how our approach may be used to compose pre-trained text-guided diffusion models and generate photorealistic images containing all the details described in the input descriptions, including the binding of certain object attributes that have been shown difficult for DALLE-2. These results point to the effectiveness of the proposed method in promoting structured generalization for visual generation. Project page: https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/

1 Introduction

The paper addresses failures of text-conditioned diffusion models on complex compositions by explicitly factorizing and composing diffusion models. This enables zero-shot generation across domains and combinations more complex than those seen during training.

  • Motivation: Text-conditioned diffusion models can generate photorealistic images but often fail on complex descriptions, including object attributes and relations.The paper attributes a possible underlying difficulty to encoding increasingly complex descriptions in fixed-size latent vectors.
  • Approach: The method factorizes compositional generation across diffusion models that capture different subsets of a compositional specification.These models are explicitly composed to generate an image.
  • Approach: Interpreting diffusion models as implicitly parameterized energy-based models enables their explicit composition while improving training stability and image quality.This connects diffusion-model composition to prior energy-based-model approaches without directly training EBMs.
  • Capabilities: The framework supports zero-shot composition of language descriptions, objects, object relations, and human facial attributes.The paper applies it to pre-trained Stable Diffusion and GLIDE models as well as object and facial-attribute composition.
  • Results: 24.02% higher accuracy than the best baseline is reported for composing three objects in specified positions on CLEVR.The reported accuracy measures generated images containing all requested concepts.

2 Related Work

The related work situates the paper among controllable image-generation methods and diffusion models, emphasizing compositional control of image content. Diffusion sampling provides an iterative, gradient-based basis for composing multiple factors.

  • Controllable Image Generation: Prior controllable image-generation methods specify image content through text or manipulate attributes in existing images.The paper distinguishes these approaches from its focus on compositionally controlling underlying image content.
  • Diffusion Models: Diffusion models formulate generation as iterative denoising and support image editing, text conditioning, and inpainting.Their iterative, gradient-based sampling enables composition of multiple factors.

3 Background

The background introduces diffusion models as iterative denoising processes and energy-based models as unnormalized density models. Their similar gradient-based sampling procedures motivate composing diffusion models through the energy-based-model perspective.

  • Denoising Diffusion Models: DDPM generation starts from Gaussian noise and performs T denoising steps to produce a sharp image x_0.The intermediate states are x_T, x_T−1, ..., x_0, with decreasing noise levels.
  • Denoising Diffusion Models: DDPMs model forward noising and reverse denoising as products of Markov transition probabilities.The forward process gradually adds Gaussian noise, while the learned reverse process removes it.
  • Denoising Diffusion Models: Each reverse diffusion step uses a Gaussian transition with learned mean μ_θ(x_t,t) and covariance σ_t^2 I.The mean is represented through a perturbation function ε_θ(x_t,t).
  • Denoising Diffusion Models: The denoising update gradually predicts a less noisy x_t−1 from x_t, and generated images become more realistic over multiple iterations.Sampling proceeds from t = T to t = 1 using p_θ(x_t−1|x_t).
  • Compositional Generation: Compositional generation computes scores for multiple concepts at each iteration, combines them with operators such as conjunction, and denoises for T iterations.The method can generate images containing all concepts without further training.
  • Energy-Based Models: Energy-based models define an unnormalized image density proportional to e^-E_θ(x) and sample from it using gradient-based Langevin dynamics.The energy function E_θ(x) is a learnable neural network.
  • Connection: Diffusion and EBM sampling both iteratively refine Gaussian-noise images while adding small amounts of noise at each step.This functional similarity motivates treating a trained diffusion model as an implicitly parameterized EBM.

4 Our approach

The approach interprets diffusion models as implicitly parameterized energy-based models and composes their learned scores to generate images conditioned on multiple concepts. It introduces conjunction and negation operators that combine concepts during inference without additional training.

  • Diffusion Models as Energy Based Models: Diffusion models and energy-based models use functionally similar iterative refinement procedures, allowing a trained diffusion model to be viewed as an implicitly parameterized energy-based model.Both start from Gaussian noise and iteratively refine images with learned score or energy-gradient information.
  • Composing Diffusion Models: Multiple energy-based models can be composed by multiplying their distributions, equivalently summing their energy functions and combining their score contributions.The composed distribution is sampled through iterative refinement using the summed energy gradients.
  • Compositional Generation through Diffusion Models: The method represents each concept with a conditional diffusion model and uses an unconditional diffusion model to define conjunction and negation operators.Conjunction combines multiple concepts under a conditional-independence assumption; negation combines a desired concept with the absence of another concept.
  • Compositional Generation through Diffusion Models: For conjunction, the composed score is formed from conditional-minus-unconditional score differences weighted by concept-specific temperature parameters, then used in the diffusion sampling process.The resulting transition samples from a Gaussian centered at the current image minus the composed score, with covariance σ_t^2 I.
  • Compositional Generation through Diffusion Models: For negation, the composed distribution uses a ratio between the desired concept likelihood and the negated concept likelihood, with a weight controlling negation strength.The resulting composed score combines the unconditional score with the difference between the desired and negated conditional scores.
  • Compositional Generation through Diffusion Models: The operators compose pre-trained diffusion models during inference without additional training, while a non-conservative score field remains a stated complication of the energy-based interpretation.The paper reports that explicitly parameterizing the score as an energy gradient achieves similar performance to the non-conservative diffusion parameterization.

5 Experiment Setup

The experiments use synthetic object and relation datasets alongside a labeled real-world face dataset, evaluating concept composition beyond the single-concept training setting. Generalization is tested by composing two or three concepts at test time.

  • Datasets: CLEVR contains 30,000 synthetic 128 × 128 images with one to five objects, varying shapes, colors, sizes, and 2D coordinate labels.Relational CLEVR adds descriptions of relations between objects, such as one object being left of another.
  • Datasets: The face experiments use FFHQ, a real-world dataset of 70,000 human face images, with 51,067 images annotated for smile, gender, and glasses.The annotations are three binary attributes produced using pre-trained classifiers.
  • Evaluation Settings: Testing compares 1 Component, 2 Components, and 3 Components settings, where the latter two compose concepts using conjunction and negation operators.These settings evaluate generalization to new combinations beyond the single-concept training distribution.
  • Evaluation Metrics: Each method generates 5,000 images per test setting, and concept accuracy is the percentage of images capturing all concepts.Image quality is additionally evaluated with Clean-FID using features from a pre-trained Inception model.

6 Experiments

Experiments evaluate compositional generation across language descriptions, objects, and object relations against GAN, EBM, and diffusion baselines. The method supports composition at inference time, improves concept and relation fidelity, and reports stronger compositional generalization with realistic images.

  • Experimental Setup: The evaluation compares compositional generation across natural language descriptions, objects, object relations, and human facial attributes.
  • Objects: The method outperforms baselines on CLEVR across three test settings using binary classification accuracy and FID evaluation.The three settings test one, two, and three-component compositions.
  • Language Descriptions: Composed GLIDE combines multiple short textual descriptions without further training, capturing details that original GLIDE misses, including polar bears and overwater bungalows.
  • Objects: The proposed model composes multiple objects at specified positions, whereas baselines may miss objects or generate incorrect objects and positions.
  • Results Analysis: The method achieves 24.02% higher accuracy than EBM in the more challenging 3 Components setting and produces lower FID scores indicating greater similarity to real images.
  • Object Relations: On Relational CLEVR, the method composes multiple object relations during inference and outperforms StyleGAN2, LACE, and GLIDE on binary classification accuracy across all test settings.EBM has strong relational accuracy but substantially worse FID, while StyleGAN2-ADA has better accuracy and FID but cannot compose multiple concepts.

7 Conclusion

The paper composes diffusion models through an energy-based interpretation to generate complex, unseen combinations across visual domains. It demonstrates broad compositional generation while noting limited success when combining models trained on different datasets.

  • 7 Conclusion: Diffusion models can be explicitly composed as energy-based models to generate combinations substantially more complex than those seen during training.The approach uses concept conjunction and negation during inference without additional training.
  • 7 Conclusion: The method composes sentence descriptions, objects, object relations, and human facial attributes, including combinations rarely seen in the real world.These results support compositional visual generation across multiple domains.
  • 7 Conclusion: The approach has limited success when composing diffusion models trained on different datasets, whereas energy-based models can compose separately trained models.The authors identify additional structure, such as a conservative score field, as a promising direction.
  • 7 Conclusion: Figure 6 categorizes failures as concept misunderstanding, confused object attributes, or composition failure, with the latter usually occurring when objects are centered.The figure also includes successful examples generated by the proposed method.

A Additional Results

Additional experiments apply the method to 3D asset generation and facial-attribute composition. On FFHQ, the method produces high-fidelity images matching all input attributes and is comparable with the best baseline across test settings.

  • A Additional Results: Point-E is composed to generate point clouds from language descriptions, which are then converted into 3D meshes using marching cubes.The results demonstrate composition of language descriptions for 3D asset generation.
  • A Additional Results: On FFHQ, the method generates high-fidelity faces containing all input attributes, unlike baselines that either mismatch labels or produce lower-quality images.StyleGAN2 and LACE produce high-fidelity images that can fail to match labels, while EBM image quality is much worse.

A.3 More Qualitative Results

The qualitative results show that compositional operators support multiple concepts, facial attributes, and language-driven 3D generation beyond the training distribution. Combining descriptions can also produce styles distinct from directly encoding one long sentence.

  • A.3 More Qualitative Results: Point-E language compositions are used for point-cloud generation and subsequent 3D mesh construction.The 3D results are presented as qualitative examples of hybrid objects.
  • A.3 More Qualitative Results: The approach reliably generates images conditioned on multiple concepts, including combinations outside the training distribution.Additional qualitative results cover language descriptions and object compositions.
  • A.3 More Qualitative Results: Facial attributes can be composed recursively during inference without further training using conjunction and negation operators.The model combines Not Male and Glasses to generate females wearing glasses.
  • A.3 More Qualitative Results: Composing multiple textual descriptions produces image styles different from GLIDE’s direct encoding of a single long sentence.For “a dog” and “the sky,” the method generates a dog-shaped cloud, while GLIDE generates a dog under the sky.

B Details of Binary Classifiers

The appendix describes classifier-based evaluation and implementation details for CLEVR, Relational CLEVR, and FFHQ. It also summarizes qualitative and quantitative facial-attribute comparisons with baselines.

  • B Details of Binary Classifiers: The CLEVR classifier evaluates whether an object appears at a specified position, while the Relational CLEVR classifier evaluates object-relational descriptions.Their validation accuracies are 99.05% and 99.80%, respectively.
  • B Details of Binary Classifiers: FFHQ evaluation uses separate binary classifiers for smiling, glasses, and gender concepts.The classifiers are trained on 30,000 image-label pairs from CelebA-HQ and assess generated facial attributes.
  • B Details of Binary Classifiers: The implementation trains conditional and unconditional diffusion models by replacing labels with null labels for 10% of data points.CLEVR coordinate labels are embedded with a linear layer and fused with the time embedding to estimate the score.

E Implementation Details

The experiments use consistent training setups across model families, with dataset- and model-specific training durations, optimizers, and diffusion configurations.

  • Each model is trained on a single Tesla V100 32GB GPU.
  • StyleGAN2-ADA: StyleGAN2-ADA models are trained for two days using Adam with β1 = 0 and β2 = 0.99.
  • StyleGAN2: StyleGAN2 models are trained for two days on CLEVR and Relational CLEVR using Adam with β1 = 0 and β2 = 0.99.
  • EBMs: EBM training uses Adam with learning rate 10^-4, while MCMC sampling uses step size 300 and 80 iterations.
  • Ours: Diffusion models use 1,000 diffusion steps with a cosine noise schedule and AdamW with β1 = 0.9 and β2 = 0.999.

F Derivation

The derivation composes independently conditioned diffusion models by combining their probability distributions and corresponding score predictions, including conjunction and negation operators. Qualitative results show that the method composes language descriptions, objects, and facial attributes, including more objects at inference than appeared during training.

  • F.1 Conjunction Operator (AND): For independent concepts, the joint distribution factorizes into the unconditional image distribution and concept likelihoods.
  • F.1 Conjunction Operator (AND): The conjunction distribution can be rewritten as the unconditional distribution multiplied by a product of likelihood ratios p(x|c_i)/p(x).
  • F.1 Conjunction Operator (AND): Taking the logarithmic gradient converts the factorized distribution into a sum of conditional and unconditional score terms.
  • F.1 Conjunction Operator (AND): The modified score prediction weights each concept-specific score difference by w_i, controlling the temperature of each implicit classifier.
  • F.1 Conjunction Operator (AND): With one conditioned concept, the expression reduces to classifier-free guidance with temperature scaling w > 1.
  • F.2 Negation Operator (NOT): Negating c1 while retaining c2 uses a ratio that removes c1's likelihood and preserves c2's contribution.
  • F.2 Negation Operator (NOT): The negation score combines the unconditional score with c2's conditional score minus c1's conditional score, scaled by w.
  • Composing Language Descriptions: Composed GLIDE combines textual descriptions without further training and produces qualitative image results.
Loading 2206.01714v6…