Source-linked AI summary
More Control for Free! Image Synthesis with Semantic Diffusion Guidance
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, Trevor Darrell
TL;DR
The paper addresses fine-grained, continuous control of diffusion-based image synthesis using language, reference images, or both. It introduces Semantic Diffusion Guidance, which steers a pretrained unconditional diffusion model through CLIP-based image-text or image-matching gradients without retraining the diffusion model. Experiments on FFHQ and LSUN demonstrate text-guided, content- and style-guided, and multimodal synthesis, including datasets without text annotations.
Problem
Fine-grained control of diffusion-based image synthesis is needed for text, reference-image, and combined guidance, while many text-guided methods require text annotations.
Method
Semantic Diffusion Guidance injects language, content, style, or multimodal guidance into an unconditional diffusion model during sampling using CLIP-based matching gradients.
Results
Experiments on FFHQ and LSUN show fine-grained text-guided synthesis, diverse content- and style-related image synthesis, and images aligned with both language and reference-image guidance.
Takeaways & Limitations
The framework provides flexible semantic control that can be added to off-the-shelf unconditional diffusion models, including datasets without language annotations.
Abstract
from arXiv · showhide
Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from a reference image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffusion model using the gradient of image-text or image matching scores, without re-training the diffusion model. We explore CLIP-based language guidance as well as both content and style-based image guidance in a unified framework. Our text-guided synthesis approach can be applied to datasets without associated text annotations. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained text-guided image synthesis, synthesis of images related to a style or content reference image, and examples with both textual and image guidance.
1. Introduction
The paper targets fine-grained control of image synthesis when text annotations or sufficiently general image-conditioned methods are unavailable. It introduces Semantic Diffusion Guidance, which injects language, image, or combined guidance into pretrained unconditional diffusion models without retraining them.
- Fine-grained control over generated images is important for assisting users with art creation and design.
- Most prior text-to-image methods require image-caption pairs and therefore do not generalize to datasets without text annotations.
- Existing image-conditioned techniques mainly transfer reference style or target structured domains, limiting diverse generation across poses, structures, and layouts.
- Semantic Diffusion Guidance unifies text-guided and image-guided synthesis by steering an unconditional DDPM during sampling.The framework uses guidance functions with CLIP-based image-text matching for language and content or style matching for image guidance.
- The framework supports fine-grained text queries, content or style reference images, and joint multimodal guidance without retraining the diffusion model.Experiments are conducted on FFHQ and LSUN, with validation focused on generated-image quality, diversity, and controllability.
2. Related Work
Related work covers controllable synthesis with text, images, and diffusion-based or CLIP-based guidance. Prior approaches established conditional generation but often required paired annotations, struggled with diversity, or relied on domain-specific settings.
- Text-guided Synthesis: Text-to-image research progressed from conditional GANs to transformer and discrete-VAE approaches using large-scale data.
- Image-guided Synthesis: Image-guided synthesis seeks diverse outputs resembling a reference in content or style, while style-transfer methods often struggle to generate diversity.
- Diffusion Models: Diffusion models use forward signal-to-noise and reverse noise-to-signal processes for generation.DDPMs gradually transform Gaussian noise into signal through a denoising autoencoder.
- Diffusion Models: Diffusion models have achieved image quality comparable or superior to GANs while offering better mode coverage and training stability.
- CLIP-guided Generation and Manipulation: CLIP provides a joint vision-language representation used for zero-shot tasks, text-guided manipulation, and domain adaptation without domain-specific image-text pairs.
3. Semantic Diffusion Guidance
Semantic Diffusion Guidance (SDG) injects language, image, or multimodal semantic control into a pretrained unconditional DDPM during iterative sampling. It uses noisy-image-aware CLIP guidance for language and image matching, while preserving flexible content, style, and modality control without retraining the diffusion model.
- Framework: SDG incorporates language, image, or both forms of guidance into a pretrained unconditional diffusion model without retraining it.Only the guidance network is finetuned, using a self-supervised scheme that does not require paired language data.
- Framework: The DDPM generates images by iteratively removing noise, while SDG is injected at every sampling timestep through a guidance function.The guidance shifts the sampling distribution using gradients of the guidance score, with scaling factor s controlling guidance strength.
- Language Guidance: Language guidance uses a CLIP image-text matching score computed between a noisy-image encoder and the text embedding.The image encoder is adapted to accept timestep information and finetuned on noisy images.
- Image Guidance: Image guidance supports content matching through image embeddings and style matching through Gram-matrix alignment of intermediate feature maps.Embedding-based guidance allows varied pose and layout, while spatial feature alignment adds structural similarity and Gram matrices provide style guidance.
- Multimodal Guidance: Multimodal guidance combines language and image guidance functions with a weighted sum, letting users balance the two modalities.The weighting factors serve as user-controlled scaling factors for each guidance signal.
- Evaluation: Table 1 evaluates SDG against prior work on FFHQ using FID, diversity, and other scores, with lower being better only for FID.The table covers both image-guided and text-guided synthesis.
4. Experiments
Experiments evaluate SDG on FFHQ and LSUN using separate language- and image-guided setups, with quantitative metrics and qualitative comparisons. Results indicate controllable, diverse synthesis across guidance types and settings.
- Evaluation Setup: Experiments use FFHQ and LSUN subsets with unconditional DDPMs, evaluating language and image guidance separately against prior work.Language-guided evaluation uses 400 FFHQ text instructions and 10,000 generated images; image-guided evaluation compares against ILVR.
- Quantitative Evaluation: FID measures image quality, and SDG outperforms compared methods for both image-guided and language-guided synthesis.FID is computed on 10,000 generated images per dataset, with lower scores indicating better quality.
- Quantitative Evaluation: LPIPS results indicate that SDG generates more diverse images than ILVR and StyleGAN+CLIP.Unlike the baselines, SDG produces variations in pose, structure, and layout rather than mainly changing details.
- Ablation Study: Higher guidance scaling improves semantic consistency but reduces diversity, allowing users to choose the desired control-diversity trade-off.The scaling factor s is user-controllable and determines guidance strength.
- Qualitative Results: Image content guidance produces semantically similar images with different poses, layouts, and structures on LSUN and FFHQ.The qualitative results show new images that match the guidance image semantically while varying their organization.
- Qualitative Results: Language guidance handles fine-grained descriptions, while joint language-image guidance aligns generated images with both inputs.Examples include modifying a reference bedroom with windows or adding attributes such as smiling, short hair, or sunglasses.
- Other Applications: Style guidance synthesizes bedrooms in an unseen style, while content guidance preserves reference structure, pose, and layout.The framework also generates photo-realistic images semantically similar to out-of-domain cartoon guidance images.
5. Conclusion and Discussions
Semantic Diffusion Guidance unifies language, image, and multimodal control for diffusion-based image synthesis. The framework supports comparisons with prior methods and applications spanning style, structure preservation, and out-of-domain guidance.
- Semantic Diffusion Guidance unifies diffusion-based image synthesis with language, image, or multimodal guidance.
- Joint image and language guidance provides complementary information, enabling generated images to match both sources.
- Image-guided synthesis is compared with ILVR, while text-guided synthesis is compared with StyleGAN+CLIP.
- SDG supports style-guided synthesis, structure-preserving synthesis, and photorealistic synthesis from out-of-domain image guidance.