Source-linked AI summary
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, Kwan-Yee K. Wong
TL;DR
Text-to-image diffusion models produce realistic images but struggle to express detailed controls through complex text. Uni-ControlNet adds two lightweight adapters to frozen pretrained models, supporting composable local and global controls in one model. Experiments report improved controllability, fidelity, and composability with reduced fine-tuning cost and model size.
Problem
Text descriptions are inefficient or insufficient for conveying detailed controls such as fine-grained layouts, while models also struggle with complex text descriptions.
Method
Uni-ControlNet uses two shared adapters on frozen pretrained text-to-image diffusion models to integrate diverse local and global conditions compositionally.
Results
Uni-ControlNet demonstrates improved controllability, generation fidelity, and composability while requiring only two adapters for different conditions.
Takeaways & Limitations
The framework reduces fine-tuning cost and model size while enabling simultaneous use of multiple conditions.
Abstract
from arXiv · showhide
Text-to-Image diffusion models have made tremendous progress over the past two years, enabling the generation of highly realistic images based on open-domain text descriptions. However, despite their success, text descriptions often struggle to adequately convey detailed controls, even when composed of long and complex texts. Moreover, recent studies have also shown that these models face challenges in understanding such complex texts and generating the corresponding images. Therefore, there is a growing need to enable more control modes beyond text description. In this paper, we introduce Uni-ControlNet, a unified framework that allows for the simultaneous utilization of different local controls (e.g., edge maps, depth map, segmentation masks) and global controls (e.g., CLIP image embeddings) in a flexible and composable manner within one single model. Unlike existing methods, Uni-ControlNet only requires the fine-tuning of two additional adapters upon frozen pre-trained text-to-image diffusion models, eliminating the huge cost of training from scratch. Moreover, thanks to some dedicated adapter designs, Uni-ControlNet only necessitates a constant number (i.e., 2) of adapters, regardless of the number of local or global controls used. This not only reduces the fine-tuning costs and model size, making it more suitable for real-world deployment, but also facilitate composability of different conditions. Through both quantitative and qualitative comparisons, Uni-ControlNet demonstrates its superiority over existing methods in terms of controllability, generation quality and composability. Code is available at \url{https://github.com/ShihaoZhaoZSH/Uni-ControlNet}.
1 Introduction
Text-to-image diffusion models generate high-quality images but struggle to express and interpret detailed controls through complex text. Uni-ControlNet addresses this gap with two lightweight adapters that support diverse, composable controls in a single model.
- Motivation: Text descriptions inadequately convey fine-grained layouts and complex controls, motivating additional visual control modes for text-to-image diffusion models.Examples include user-drawn sketches and semantic masks used alongside text descriptions.
- Contribution: Uni-ControlNet uses lightweight adapters on pretrained text-to-image models to provide precise control without training a new model from scratch.The framework supports different conditions within one model and composable control.
- Contribution: Local and global conditions are handled by two shared adapters regardless of the number of controls involved.Local controls are injected through multi-scale modulation, while global controls become conditional tokens concatenated with text tokens for cross-attention.
- Results: Training on 10 million text-image pairs for 1 epoch yields promising fidelity and controllability, with visual examples covering single and multi-condition use.Ablations also compare the adapter designs against ControlNet, GLIGEN, and T2I-Adapter.
2 Related Work
Text-to-image generation has evolved from domain-specific GANs and autoregressive models toward diffusion models, while controllable diffusion methods mainly differ in training strategy. Existing approaches trade off control quality against substantial training cost or adapter complexity.
- Text-to-Image Generation: Text-to-image generation seeks realistic images from text, with early GAN approaches limited by training instability and poor open-domain generalization.Autoregressive methods later used web-scale image-text pairs for open-domain zero-shot generation.
- Controllable Diffusion Models: Controllable diffusion models extend text-to-image systems with additional user controls for guiding generation.The literature describes this as an increasingly important direction.
- Controllable Diffusion Models: Existing methods either train a large model from scratch or fine-tune lightweight adapters on frozen pretrained models.Composer exemplifies the first strategy, while ControlNet, GLIGEN, and T2I-Adapter exemplify the second.
- Controllable Diffusion Models: Composer supports single and multi-condition control but requires substantial training resources because it trains a large diffusion model from scratch.The caption defines fine-tuning cost as the number of fine-tunings required across N conditions.
3 Method
Uni-ControlNet adds dedicated local and global control adapters to a frozen Stable Diffusion model, supporting diverse conditions and their composition. Local controls are injected across resolutions, while global controls are converted into tokens for cross-attention.
- Preliminary: Uni-ControlNet uses Stable Diffusion’s UNet encoder, middle block, decoder, skip connections, and text cross-attention as its base architecture.The decoder combines middle-block and encoder features through skip connections, while cross-attention uses projected noise features and text embeddings.
- Control Adapter: The framework supports seven local conditions and one global condition based on a CLIP embedding of a reference image.Local examples include edges, sketches, pose, depth, and segmentation masks; the global embedding provides semantic content without explicit spatial guidance.
- Local Control Adapter: A local control adapter copies the base encoder and middle-block structure, then integrates control features into the decoder through zero convolutions.Zero convolutions begin with zero weights and gradually incorporate control information while preserving the remaining model components.
- Local Control Adapter: Local conditions are concatenated, extracted at multiple resolutions, and injected through Feature Denormalization using spatially varying scale and shift coefficients.The feature extractor produces resolution-specific condition features that modulate normalized noise features through learnable convolutions.
- Global Control Adapter: The global control adapter projects a global signal into condition embeddings, reshapes them into tokens, and concatenates them with text tokens for cross-attention.The extended prompt is supplied to cross-attention layers in both the main diffusion model and the control adapters.
- Training Strategy: The local and global adapters are fine-tuned separately because jointly fine-tuning them can produce poor controllable generation performance.Condition dropout is used during each adapter’s fine-tuning to regulate condition use.
4 Experiments
Uni-ControlNet is evaluated against existing controllable diffusion models on generation quality, controllability, single-condition alignment, composable multi-conditions, and adapter design choices. It achieves strong results while using two adapters for all conditions, with ablations supporting its injection and separate-training strategies.
- Quantitative comparison: Uni-ControlNet shows superior FID performance across most evaluated conditions compared with existing approaches.FID is reported on COCO2017-generated images under single-condition evaluation.
- Quantitative comparison: Uni-ControlNet outperforms other baseline methods in 4 out of 8 controllability metrics, while ControlNet leads 3 and T2I-Adapter leads 1.The metrics cover structural similarity, pose, depth, segmentation, and CLIP-based content evaluation.
- Qualitative comparison: Uni-ControlNet uses two adapters for all conditions, whereas ControlNet, GLIGEN, and T2I-Adapter fine-tune eight adapters for eight single conditions.This comparison concerns the qualitative single-condition setting.
- Composable multi-conditions: Uni-ControlNet fuses two local conditions better than Multi-ControlNet and CoAdapter, which sometimes omit elements such as a podium or car.For unrelated local-global inputs, CoAdapter can produce unrealistic, poorly integrated results, whereas Uni-ControlNet generates an integrated Minion-shaped bus example.
- Condition injection ablation: The proposed local-condition injection strategy performs better under most settings, while interpolation-based Injection-S1 damages condition alignment and Injection-S2 performs poorly for composite control.The feature extractor projects conditions into corresponding latent spaces at different layers to improve alignment with noise features.
- Condition injection ablation: Without injecting the global condition into the main diffusion model through the extended prompt, Injection-S3 cannot transfer that condition into final generated images.The global condition is otherwise added only to the control adapter.
- Training-strategy ablation: The default separate-fine-tuning strategy and Train-S2 consistently outperform Train-S1, while further joint fine-tuning provides no obvious gain in most cases.Train-S1 jointly fine-tunes both adapters from scratch; Train-S2 jointly fine-tunes them after separate fine-tuning.
5 Conclusion and Social Impact
Uni-ControlNet integrates diverse local and global controls using two adapters, reducing fine-tuning cost and model size while facilitating simultaneous composability. Experiments show improved controllability, generation fidelity, and composability, alongside considerations for responsible deployment.
- Uni-ControlNet integrates diverse local and global controls with only two adapters, reducing fine-tuning cost and model size.
- The framework supports simultaneous use of multiple conditions and facilitates composability.
- Extensive experiments validate improved controllability, generation fidelity, and composability.
- The paper identifies misuse risks and calls for responsible deployment, ethical regulation, and transparency flags on generated images.
A The Weight of the Global Condition
The global-control weight λ determines how strongly global conditions influence generated images when concatenated with projected global features and text. Increasing λ makes elements of the global conditions more visible.
- λ controls the influence of the global condition when its projection is concatenated with the text.
- Increasing λ makes elements of the global condition appear more prominently in generated images.
- Figure 9 compares generated images under increased λ values against the supplied textual prompts and global conditions.
B Condition Conflicts
The conflict study examines how Uni-ControlNet resolves incompatible conditions. HED dominates the resulting structure, while other conditions influence mainly non-overlapping regions and Openpose is weakest.
- The experiment uses incompatible local conditions, including two different dog-shaped controls, to analyze condition conflicts.
- HED is the strongest condition, and generated images closely follow its boundary when depicting the prompt.
- The condition-strength ordering is HED, Canny, sketch, depth, MLSD, segmentation map, then Openpose.
- Other conditions mainly affect regions without overlap, while Openpose is generally disregarded during conflicts.
C Hand-drawn Sketches
The supplementary studies examine hand-drawn sketches, extending a trained local adapter to new conditions, and composing multiple conditions of the same type. They describe the tested retraining strategies and the Uni-Channels approach.
- C Hand-drawn Sketches: The hand-drawn-sketch study evaluates model behavior despite distribution gaps between hand-drawn and model-generated sketches.
- Extending to New Conditions: The extension experiment adds Canny to a local adapter trained for six conditions and compares four retraining strategies.
- Extending to New Conditions: The retraining strategies range from retraining the entire feature extractor to random initialization without retraining.
- Extending to New Conditions: The extension process keeps the copied encoder weights fixed and uses 300k training samples.
- Extending to New Conditions: Figure 12 presents the extension from MLSD, HED, Sketch, OpenPose, Depth, and Segmentation to the same set plus Canny.
- Same-Type Composite Control: Uni-Channels addresses same-type composite control by adding three extra condition channels to the input.
- Same-Type Composite Control: Figure 13 presents visualization results for composite control using two conditions of the same type.
F Comparison with Stable Diffusion 2.1
Uni-ControlNet is compared with Stable Diffusion 2.1 variants for depth-map and reference-image control. The comparison highlights that the Stable Diffusion variants fine-tune the entire model rather than only adapters.
- The comparison evaluates Uni-ControlNet against SD2-depth and SD2-unclip for depth-map and reference-image inputs, respectively.
- Stable Diffusion 2.1 variants are evaluated using both FID and CLIP score.
- Whole-model fine-tuning, rather than adapter-only fine-tuning, is identified as a key factor behind SD2-depth and SD2-unclip’s strong performance.
- SD2-depth and SD2-unclip outperform ControlNet, GLIGEN, and T2I-Adapter in the cited comparisons.
G More Quantitative Results
Additional quantitative evaluation reports strong CLIP-score performance for Uni-ControlNet across most conditions and compares it with existing controllable diffusion models. A user study separately evaluates image quality, text matching, and condition alignment in single- and multi-condition settings.
- Uni-ControlNet achieves superior CLIP-score performance across most conditions compared with existing controllable diffusion models.
- Uni-ControlNet and Training-S2 consistently outperform other condition-injection and training strategies.
- Joint fine-tuning in Training-S2 does not produce obvious performance gains in most cases.
- The user study compares Uni-ControlNet with ControlNet, GLIGEN, and T2I-Adapter in single- and multi-condition settings.Each setting contains 20 cases.
- User-study evaluations measure generated-image quality, text matching, and alignment with the given condition.
H More Visualization Results
Additional visual results cover Uni-ControlNet under single- and multi-condition settings, including examples with four conditions. Supplementary figures also document the local and global adapter designs.
- More Visualization Results: Figure 17 presents Uni-ControlNet results for the single-condition setting, while Figure 18 presents multi-condition results.
- More Visualization Results: Rows 7–8 of Figure 18 show a more challenging case involving four conditions.
- Adapter Details: Figures 19 and 20 provide details of the local control adapter and its FDN-equipped ResBlock.
- More Visualization Results: The supplementary material includes visual examples with prompts describing a street scene and a flower on a plate with utensils.
- Adapter Details: Figure 21 provides details of the global control adapter, with d denoting text-token embedding dimension and K the number of global tokens.