Source-linked AI summary
Composer: Creative and Controllable Image Synthesis with Composable Conditions
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, Jingren Zhou
TL;DR
Generative image models remain difficult to control despite producing high-quality images. Composer decomposes images into composable representations and recombines them with a multi-conditional diffusion model, supporting novel combinations and many generation tasks; it achieves a zero-shot FID of 9.2 on COCO text-to-image synthesis using captions alone.
Problem
Generative image models can produce photorealistic and diverse images but provide designers limited controllability for practical applications.
Method
Composer decomposes images into representations and trains a multi-conditional diffusion model to reconstruct them from representation subsets.
Results
Composer enables novel combinations of representations and a broad range of image generation and manipulation tasks, achieving a zero-shot FID of 9.2 on COCO text-to-image synthesis with captions alone.
Takeaways & Limitations
Composable conditions vastly expand generative-model control and allow traditional and previously unexplored tasks without retraining.
Takeaways & Limitations
Joint training of multiple conditions can downweight single-conditional performance, and incompatible conditions may conflict during generation.
Abstract
from arXiv · showhide
Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity. With compositionality as the core idea, we first decompose an image into representative factors, and then train a diffusion model with all these factors as the conditions to recompose the input. At the inference stage, the rich intermediate representations work as composable elements, leading to a huge design space (i.e., exponentially proportional to the number of decomposed factors) for customizable content creation. It is noteworthy that our approach, which we call Composer, supports various levels of conditions, such as text description as the global information, depth map and sketch as the local guidance, color histogram for low-level details, etc. Besides improving controllability, we confirm that Composer serves as a general framework and facilitates a wide range of classical generative tasks without retraining. Code and models will be made available.
1. Introduction
Composer addresses limited controllability in generative image models through compositionality: it decomposes images into components, then recombines them with multi-conditional diffusion to support flexible customization and novel combinations.
- Generative image models support photorealistic, diverse outputs but still offer designers limited controllability for practical applications.Existing approaches add conditions such as segmentation maps, scene graphs, sketches, depth maps, and inpainting masks, or fine-tune models on subject-specific data.
- Compositionality expands control by enabling many novel combinations from a limited set of known visual components.The paper argues that compositionality can exponentially expand the control space.
- Composer decomposes images into representations and trains a multi-conditional diffusion model to reconstruct them from representation subsets.Its training alternates between decomposition with computer vision algorithms or pretrained models and composition through reconstruction optimization.
- Composer can decode novel images from unseen combinations of representations, including components from different and potentially incompatible sources.
- Composer supports diverse generation and manipulation tasks, including text-to-image generation, style and pose transfer, image translation, virtual try-on, reconfiguration, colorization, and localized editing.An orthogonal masking representation restricts edits to user-specified regions, and the model achieves a zero-shot FID of 9.2 on COCO text-to-image synthesis using only captions.
2. Method
Composer decomposes images into multiple representations and recomposes them with a multi-conditional diffusion model. Its conditioning and training design supports flexible combinations of global and localized controls, including text, color, sketches, depth, segmentation, intensity, and masks.
- Decomposition and composition: Composer separates images into decoupled representations and reassembles them with a conditional diffusion model.The framework uses decomposition and composition phases, with representations extracted on-the-fly during training.
- Guidance directions: The model uses classifier-free guidance to vary emphasis between two condition sets during sampling.Conditions unique to one set are emphasized or suppressed, while shared conditions receive weight 1.0.
- Guidance directions: Composer supports bidirectional guidance by reversing an image under one condition set and resampling it under another.The difference between the two condition sets defines the manipulation direction.
- Representations: Global conditions include captions, image semantics and style, and color statistics represented through embeddings or a CIELab histogram.Caption representations use CLIP sentence and word embeddings, while color uses a smoothed CIELab histogram.
- Representations: Localized conditions include sketches, instance masks, depthmaps, intensity images, and masked images that encode spatial or regional information.Masked images use three RGB channels plus a binary-mask channel; localized embeddings are spatially aligned with the noisy latent.
- Model architecture: Composer modifies GLIDE conditioning modules to combine global and localized representations within a UNet diffusion architecture.Global representations condition timestep and cross-attention inputs, while localized embeddings are summed and concatenated with the noisy latent.
- Joint training: Independent condition dropout trains Composer to decode images from varied combinations of conditions.The reported configuration uses 0.5 dropout per condition, 0.1 probability of dropping all conditions, and 0.1 probability of retaining all conditions.
- High-resolution generation: Composer generates 64 × 64 base images and uses two unconditional diffusion upsamplers to reach 256 × 256 and 1024 × 1024 resolutions.The upsampling models are modified from unCLIP with increased low-resolution capacity and self-attention blocks.
3. Experiments
Composer supports diverse image generation and manipulation tasks by recombining selected representations, and it can be applied without retraining. Its multi-task model also achieves competitive text-to-image quality on COCO.
- Image Manipulation: Composer creates controlled variations by conditioning on selected subsets of an image’s representations.Adding more conditions yields more accurate reconstructions than unCLIP, which uses only image embeddings.
- Image Manipulation: Composer interpolates global representations between images while controlling which elements change and which remain fixed.This produces multiple distinct interpolation directions.
- Image Manipulation: Composer reconfigures images by DDIM inversion followed by sampling with modified representations, changing selected attributes while preserving other content.The difference between original and modified representations defines the variation direction.
- Image Manipulation: Masked-image conditioning restricts variations to an editable region and extends editing flexibility beyond inpainting.The editable-region operation is described as orthogonal to other generation and manipulation operations.
- Traditional Task Reformulation: Composer reformulates colorization, style transfer, image translation, pose transfer, and virtual try-on within one framework.The reported virtual try-on results are of moderate quality but demonstrate the framework’s application to a difficult task.
- Image Generation Quality: 9.2 FID and 0.28 CLIP are reported for Composer on COCO despite multi-task training, with performance comparable to leading text-to-image diffusion models.The evaluation compares Composer with state-of-the-art text-to-image models using multiple sampling configurations.
4. Related Work
Diffusion models provide a strong image-generation foundation, while related work adds individual conditions or subject-specific fine-tuning. Composer differs by supporting composable conditions for greater flexibility and controllability.
- Diffusion Models: Diffusion models have emerged as a successful image-generation paradigm and outperform GANs.Related systems include hierarchical diffusion models that upscale low-resolution outputs with additional diffusion models.
- Composer’s Position: Unlike text-to-image models described here, Composer supports composable conditions and exhibits better flexibility and controllability.The distinction is presented as a contrast with models using text conditioning alone.
- Conditional Generation: Recent methods extend pretrained text-to-image diffusion models with masks, sketches, scene graphs, keypoints, segmentation, text compositions, depthmaps, or subject-specific fine-tuning.These approaches typically introduce conditions or adapt parameters for customized generation.
5. Conclusion and Discussion
The paper concludes that composing conditions expands generative control and supports many tasks through one architecture. It also identifies training conflicts and misuse risks as important limitations.
- Conclusion: Composable conditions vastly expand the control space of generative models compared with using conditions independently.The paper connects this expansion to reformulating traditional tasks and exposing previously unexplored abilities.
- Limitations: Joint multi-condition training may reduce single-conditional performance and produce conflicts between incompatible conditions.When image and text embeddings have different semantics, text embeddings are often downweighted in generated results.
- Risks: Improved controllability also raises risks of generating deceptive or harmful content, motivating investigation of mitigation and filtering.The authors state that they intend to study misuse mitigation before public release.
A. Architecture Details
Table 1 lists Composer’s hyperparameters and specifies DPM-Solver++ as the sampling algorithm for all diffusion models.
- Architecture Details: Table 1 provides Composer’s hyperparameters and identifies DPM-Solver++ as the sampling algorithm used across its diffusion models.The supplied caption does not enumerate the individual hyperparameter values.
B. Conditioning Modules
Composer uses separate modules for global and local conditions. Global inputs affect timestep and cross-attention context, while local inputs are spatially aligned and concatenated with the noisy image.
- Global conditioning: Global conditions are projected into the timestep embedding and, for image embeddings and color palettes, into eight extra cross-attention tokens.These tokens are concatenated with CLIP word embeddings as context for the cross-attention layers.
- Local conditioning: Local conditions are converted into spatial embeddings matching the noisy image before being summed and concatenated to it.The module supports segmentation maps, depthmaps, sketches, grayscale images, and masked images.
C. Additional Samples
Composer’s additional samples examine conflicts between disparate conditions and provide further colorization and style-transfer results. In conflicts, the model typically gives less weight to conditions containing fewer details.
- Conflicting conditions: Conflicting conditions are illustrated using extreme cases in which the inputs come from disparate sources.The examples include comparisons involving segmentation maps versus sketches and depthmaps, and text embeddings versus image embeddings.
- Additional results: Additional colorization results are visualized at 256 × 256 resolution to reduce file size.The figure presents supplementary colorization examples rather than a quantitative evaluation.
- Additional results: Additional style transfer results are visualized at 256 × 256 resolution to reduce file size.The figure presents supplementary style-transfer examples rather than a quantitative evaluation.