Source-linked AI summary
Collaborative Diffusion for Multi-Modal Face Generation and Editing
Ziqi Huang, Kelvin C. K. Chan, Yuming Jiang, Ziwei Liu
TL;DR
Existing diffusion models mainly provide uni-modal control, motivating a framework for combining multiple conditions without retraining. Collaborative Diffusion coordinates pre-trained uni-modal models through a dynamic diffuser for multi-modal face generation and editing. The framework achieves reported superiority in image quality and condition consistency across synthesis and editing tasks.
Problem
Existing diffusion explorations largely use one modality at a time, while multi-modal conditioning remains under-explored and limits controllability.
Method
Collaborative Diffusion combines pre-trained uni-modal diffusion models without retraining, using a dynamic diffuser to predict spatial-temporal influence functions for their contributions.
Results
The method achieves reported superiority in image quality and condition consistency for both multi-modal face synthesis and editing.
Takeaways & Limitations
The framework extends uni-modal diffusion capabilities to multi-modal face generation and editing, including combinations of existing editing approaches.
Takeaways & Limitations
Performance depends on the capabilities of the pre-trained diffusion models used as collaborators.
Abstract
from arXiv · showhide
Diffusion models arise as a powerful generative tool recently. Despite the great progress, existing diffusion models mainly focus on uni-modal control, i.e., the diffusion process is driven by only one modality of condition. To further unleash the users' creativity, it is desirable for the model to be controllable by multiple modalities simultaneously, e.g., generating and editing faces by describing the age (text-driven) while drawing the face shape (mask-driven). In this work, we present Collaborative Diffusion, where pre-trained uni-modal diffusion models collaborate to achieve multi-modal face generation and editing without re-training. Our key insight is that diffusion models driven by different modalities are inherently complementary regarding the latent denoising steps, where bilateral connections can be established upon. Specifically, we propose dynamic diffuser, a meta-network that adaptively hallucinates multi-modal denoising steps by predicting the spatial-temporal influence functions for each pre-trained uni-modal model. Collaborative Diffusion not only collaborates generation capabilities from uni-modal diffusion models, but also integrates multiple uni-modal manipulations to perform multi-modal editing. Extensive qualitative and quantitative experiments demonstrate the superiority of our framework in both image quality and condition consistency.
1. Introduction
Diffusion models offer flexible modality conditioning, but existing systems largely use one modality at a time, limiting multi-modal controllability. Collaborative Diffusion combines pre-trained uni-modal models without retraining, using dynamic influence prediction for multi-modal face generation and editing.
- Existing diffusion systems largely support one conditioning modality at a time, leaving multi-modal control under-explored.
- Training a unified multi-modal model from scratch is expensive, and fixed modality combinations require retraining when conditions change.
- Collaborative Diffusion synergizes pre-trained uni-modal diffusion models for multi-modal face generation and editing without retraining.
- The dynamic diffuser predicts spatial-varying and temporal-varying influence functions to enhance or suppress modality contributions at each iterative step.
- The framework extends to multi-modal face editing, with qualitative and quantitative results reporting superior image quality and condition consistency in synthesis and editing.
2. Related Work
Diffusion models use score-matching objectives across noise levels and generate images through iterative denoising. Collaborative Diffusion instead focuses on exploiting existing models for multi-modal synthesis and editing without large-scale retraining.
- Diffusion models are trained with score-matching objectives at various noise levels and sampled through iterative denoising.
- Collaborative Diffusion exploits existing diffusion models to provide multi-modal synthesis and editing without large-scale retraining.
- The framework overview shows a dynamic diffuser predicting spatial-temporal influence functions during reverse-process steps to selectively enhance or suppress modalities.
3. Collaborative Diffusion
Collaborative Diffusion combines multiple pre-trained uni-modal diffusion models for multi-modal face generation and editing without retraining. A dynamic diffuser predicts spatial-temporal influence functions to regulate each model’s contribution during denoising, and the framework extends this collaboration to editing while preserving identity.
- Framework: Collaborative Diffusion combines pre-trained uni-modal diffusion models for multi-modal generation and editing without altering the pre-trained models.The framework is designed to integrate models driven by different modalities within one collaborative process.
- Dynamic Diffuser: Dynamic diffuser predicts spatial-temporal influence functions that selectively enhance or suppress each modality’s contribution at every denoising step.The influence depends on the diffusion state, timestep, spatial location, and modality condition.
- Multi-Modal Collaboration: The learned influence functions combine the collaborators’ noise predictions through pixel-wise weighting at each denoising step.The influence maps are normalized across modalities before controlling each model’s contribution.
- Framework: The framework can integrate models with different weights, architectures, and modalities through learned dynamic diffusers.This flexibility distinguishes it from composition based on identical text-to-image model instances.
- Multi-Modal Collaborative Editing: The framework supports collaborative editing by combining multiple facial manipulations, each guided by a different modality.The demonstrated editing procedure fine-tunes models for identity capture and then combines them with trained dynamic diffusers.
- Results: Qualitative comparisons report outputs consistent with both mask and text conditions, while maintaining identity in face editing.The comparisons describe competing methods failing on mask or text consistency in representative examples.
4. Experiments
Experiments show that Collaborative Diffusion produces diverse, realistic, condition-consistent face generation and editing, while spatial-temporal influence variation is necessary for effective collaboration.
- Quantitative Comparison: Users selected the method as best in 69.40% of generation trials and 84.37% of editing trials for quality, consistency, and identity preservation.These results are reported in the user study comparing three methods.
- Qualitative Comparison: The framework synthesizes realistic faces under diverse text-mask combinations, including relatively rare conditions, while preserving condition consistency.Examples include unconstrained hair color and skin tone diversity and a man with long hair.
- Qualitative Comparison: In editing, the method maintains identity while satisfying both mask-driven and text-driven target conditions.Compared methods fail on either hair-mask consistency or text-specified beard generation in the cited example.
- Quantitative Comparison: The method outperforms TediGAN and Composable Diffusion on all three objective metrics for face generation, with lower FID and higher text-mask consistency.The table caption identifies lower FID as better image quality and higher consistency with both conditions.
- Influence Function Analysis: Spatial influence variation assigns mask-driven influence to facial contours and text-driven influence to skin regions, matching their respective control roles.The ablation reports deterioration in output quality and condition consistency when spatial variation is removed.
- Influence Function Analysis: Temporal influence variation gives stronger mask-driven influence at earlier diffusion stages and increases text-driven influence at later stages.Removing temporal variation causes drops in both image quality and condition consistency.
5. Conclusion
The conclusion presents Collaborative Diffusion as a re-training-free framework for multi-modal face generation and editing, while noting dependence on the capabilities of its pre-trained collaborators and potential misuse.
- Conclusion: Collaborative Diffusion combines pre-trained uni-modal diffusion models for multi-modal face generation and editing without re-training.The framework uses dynamic influence prediction to extend uni-modal approaches toward multi-modal control.
- Conclusion: The framework is intended to support comprehensive multi-modal conditioning and inspire related applications in motion and 3D generation.The paper frames these domains as potential future directions rather than demonstrated applications.
- Limitations and Future Work: Performance depends on the capabilities of the pre-trained diffusion models used as collaborators.The authors identify training each collaborator on large-scale datasets as an orthogonal direction for performance gains.
- Potential Negative Societal Impacts: The facial manipulation capability could be applied maliciously to real human faces, so the authors advise recreational use only.This statement appears in the paper’s discussion of potential negative societal impacts.
Supplementary
The supplementary file expands implementation and experimental details, adds qualitative results and visualizations, and discusses potential societal impacts.
- Supplementary: The supplementary material covers framework implementation, experimental details, additional qualitative results and visualizations, and societal impacts.These topics are organized in Sections A through D.
A. Implementation Details
The framework combines latent-space uni-modal diffusion models with dynamic diffusers for multi-modal synthesis and extends the setup to collaborative editing. Implementation uses VAE-compressed 64×64×3 latents, spatial-temporal influence prediction, and modality-specific condition optimization.
- Multi-Modal Collaborative Synthesis: LDM-based uni-modal diffusion models operate in VAE-compressed latent space to balance image quality and sampling speed.The VAE maps 256×256×3 images to 64×64×3 latents, which diffusion models denoise before decoding.
- Multi-Modal Collaborative Synthesis: The framework supports 256×256 and 512×512 resolutions, while dynamic diffusers remain much smaller than conditional diffusion models.The implementation discussions use the 256×256 version for simplicity.
- Multi-Modal Collaborative Synthesis: The VAE objective combines reconstruction, perceptual, and KL losses, with the KL term weighted by 10^-6.The weak KL regularization prioritizes reconstruction because the VAE is used as an image-compression tool rather than a generative model.
- Multi-Modal Collaborative Synthesis: Dynamic diffusers take the noisy latent, timestep, and modality condition to predict a spatial influence function for each uni-modal model.They use UNets for matching spatial resolution and inject timesteps through adaptive layer normalization.
- Multi-Modal Collaborative Synthesis: Cross-attention feeds modality conditions and intermediate activations into dynamic diffusers through projected queries, keys, and values.The projections are Q = WQ · hin, K = WK · cm, and V = WV · cm.
- Collaborative Editing: Collaborative editing combines text-driven and mask-driven Imagic edits by optimizing conditions, fine-tuning modality-specific models, and integrating them on one input image.Imagic reconstructs the input, improves fidelity through fine-tuning, interpolates conditions, and synthesizes the edited image.
B.1. Dataset
Experiments use CelebA-HQ images with multi-modal annotations from CelebAMask-HQ and CelebA-Dialog. Training uses 27,000 images, while reported results use conditions from a 3,000-image validation split.
- Dataset: CelebA-HQ provides 30,000 high-resolution images, split into 27,000 training and 3,000 validation images.Uni-modal diffusion models and dynamic diffusers train on the training split; reported results use the validation split.
- Dataset: CelebAMask-HQ supplies segmentation masks with 19 facial-component and accessory classes.Classes include skin, hair, eyes, mouth, eyeglasses, earrings, necklaces, and clothing.
- Dataset: CelebA-Dialog supplies fine-grained text descriptions for Bangs, Eyeglasses, Beard, Smiling, and Age.Descriptions for Bangs, Eyeglasses, and Smiling are trimmed because masks already describe those attributes.
- Dataset: TediGAN is used as a StyleGAN-based baseline for text- and mask-driven generation and editing, including style-code mixing for multi-modal control.Official implementations are used for the uni-modal generation and editing comparisons.
- Dataset: Composable and Collaborative Diffusion use the same pre-trained uni-modal conditional diffusion models and 50-step DDIM sampling for fair comparisons.The setting applies to quantitative, qualitative, and user-study experiments involving these methods.
C. More Qualitative Results
Supplementary figures provide additional face-generation and face-editing examples, together with visualizations showing how influence functions vary across spatial regions and diffusion timesteps.
- More Qualitative Results: Figures A11 and A12 provide additional face-generation results, while Figure A13 provides additional face-editing results.These figures are located at the end of the supplementary file.
- More Qualitative Results: Figures A14 and A15 visualize spatial-temporal influence-function variation across DDIM sampling steps.The visualizations include modality-specific influence maps and decoded intermediate diffusion results.
D. Potential Societal Impacts
The paper highlights both the capabilities and societal risks of Collaborative Diffusion. It cautions that realistic multi-modal editing can facilitate malicious face manipulation and that celebrity-based training data may not represent the general population.
- Potential Societal Impacts: High-quality editing of real faces can be used to maliciously manipulate human faces, motivating restricted recreational use.The authors advise users to apply Collaborative Diffusion only for proper recreational purposes.
- Potential Societal Impacts: Training on CelebA-HQ celebrity faces could produce biases that deviate from the appearance of the general population.The paper also identifies misinformation and possible training-data disclosure as broader generative-model concerns.