Source-linked AI summary
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
TL;DR
Text-to-image diffusion models need finer spatial control than text prompts alone provide. ControlNet adds conditioning through a trainable copy connected to a frozen pretrained backbone with zero convolutions, and experiments show control across diverse inputs, settings, and dataset sizes. The results support its applicability to a wider range of image-diffusion conditions.
Problem
Text-to-image diffusion models provide limited spatial control, making complex layouts, poses, shapes, and forms difficult to specify through text prompts alone.
Method
ControlNet freezes a pretrained text-to-image model, reuses its encoding layers in a trainable copy, and connects the two with zero-initialized convolutions for spatial conditioning.
Results
ControlNet controls Stable Diffusion with diverse conditioning inputs, supports single or multiple conditions with or without prompts, and remains robust across dataset sizes.
Takeaways & Limitations
ControlNet’s results support applying the architecture to a wider range of conditions and related image-diffusion applications.
Abstract
from arXiv · showhide
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.
1. Introduction
Text-to-image diffusion models offer strong visual generation but limited spatial control, making complex compositions difficult to specify through prompts alone. ControlNet addresses this by adding spatial conditioning while preserving the pretrained model, and experiments show broad, robust control capabilities.
- Text prompts provide limited control over spatial composition, so matching complex layouts, poses, shapes, and forms often requires repeated prompt editing.
- Spatial conditioning images such as edges, pose skeletons, segmentation maps, depth, and normals can directly specify desired image composition.
- Specific conditioning datasets are often much smaller than general text-to-image training data, making end-to-end learning of conditional controls challenging.
- ControlNet freezes the large pretrained model, trains a copy of its encoding layers, and connects them with zero-initialized convolutions to learn diverse conditional controls without initially adding harmful feature noise.
- ControlNet controls Stable Diffusion using diverse conditions, supports single or multiple conditions with or without prompts, and remains robust across dataset sizes.
- The paper proposes ControlNet, pretrained controls for several spatial inputs, and validation through ablations and user studies against alternative architectures and baselines.
2. Related Work
Related work spans finetuning strategies, parameter-efficient adaptation, diffusion models, and image-to-image translation. These approaches motivate ControlNet’s use of frozen pretrained models, added modules, and zero-initialized connections for conditional control.
- Finetuning Neural Networks: Direct continued training can cause overfitting, mode collapse, and catastrophic forgetting, motivating finetuning strategies that avoid these issues.
- Finetuning Neural Networks: HyperNetworks train a small recurrent network to influence the weights of a larger network and have been applied to image-generation models.
- Finetuning Neural Networks: Adapter methods customize pretrained models by embedding new module layers, while related applications include incremental learning, domain adaptation, and backbone transfer.
- Finetuning Neural Networks: Additive Learning freezes original weights and adds parameters through masks, pruning, or hard attention, while Side-Tuning blends frozen-model and side-branch outputs.
- Finetuning Neural Networks: Low-Rank Adaptation learns parameter offsets with low-rank matrices to address catastrophic forgetting, based on low intrinsic dimensionality in over-parameterized models.
- ControlNet connections: ControlNet connects network blocks using zero-initialized layers, building on prior research into network-weight initialization and manipulation.
- Image Diffusion: Stable Diffusion is a large-scale latent diffusion implementation, while related image-diffusion methods support generation, editing, color variation, and inpainting.
- Image-to-Image Translation: Conditional GANs and transformers learn mappings between image domains, including models trained from scratch and pretrained image-to-image translation systems.
3. Method
ControlNet adds spatial conditioning to Stable Diffusion by freezing the pretrained model, cloning selected encoding blocks for training, and connecting them with zero convolutions. It supports multiple conditioning inputs and inference controls while preserving efficient fine-tuning and stable training behavior.
- Architecture: ControlNet freezes the original neural block and trains a cloned copy that receives an external conditioning vector, preserving the pretrained backbone.The locked parameters retain the capabilities of the large model trained with billions of images, while the trainable copy learns conditional controls.
- Architecture: Zero convolutions connect the trainable copy to the locked model, using 1 × 1 convolutions whose weights and biases are initialized to zero.At the first training step, both zero-convolution terms evaluate to zero, so the original block's behavior is initially preserved.
- Training: Zero convolutions prevent harmful noise from affecting the trainable copy at initialization while retaining the pretrained model’s capabilities as a strong learning backbone.The zero-initialized connections eliminate random noise as gradients in the initial training steps.
- Stable Diffusion integration: ControlNet replicates Stable Diffusion’s 12 encoding blocks and middle block across four resolutions, then adds their outputs to the U-net skip connections and middle block.The architecture uses the encoder levels of the U-net while leaving the original production model locked.
- Efficiency: Because the original encoder is frozen, ControlNet requires about 23% more GPU memory and 34% more time per iteration than optimizing Stable Diffusion without ControlNet.No gradient computation is required in the locked encoder during fine-tuning.
- Inference: ControlNet supports diverse conditioning inputs, multiple simultaneous conditions, and operation with or without text prompts.Multiple ControlNets can be composed by directly adding their outputs, while CFG Resolution Weighting adjusts condition connections by block resolution.
4. Experiments
Experiments evaluate ControlNet across diverse spatial conditions, prompt settings, architectures, baselines, and dataset sizes. Results show robust conditioning, competitive visual quality, scalable training, and transfer to community models.
- Experimental Setup: ControlNet supports Canny edges, depth, normals, lines, soft edges, segmentation, human key points, and user sketches, with or without prompts and with multiple conditions.The experiments include varied conditioning inputs and prompt settings, including composition of multiple conditions.
- Ablations: ControlNet succeeds across no-prompt, insufficient-prompt, conflicting-prompt, and perfect-prompt settings, whereas ControlNet-lite fails in no-prompt and insufficient-prompt cases.The ablation uses four prompt settings designed to represent possible real-world user behavior.
- Ablations: Replacing zero convolutions with Gaussian-initialized convolutions reduces performance to about the level of ControlNet-lite, indicating destruction of the trainable copy’s pretrained backbone during finetuning.The comparison evaluates both alternative convolution initialization and a lightweight single-layer architecture.
- User Study: Users compared ControlNet with PITI, two Sketch-Guided Diffusion settings, and ControlNet-lite on image quality and sketch fidelity across 20 unseen sketches.Twelve users ranked five methods for each sketch group.
- Industrial Comparison: 0.52 ± 0.17 average user precision indicates that ControlNet and SDv2-D2I produced almost indistinguishable depth-conditioned results, despite ControlNet using 200k samples, one RTX 3090Ti, and five days of training.SDv2-D2I used more than 12M images, thousands of GPU hours, and a large A100 cluster.
- Robustness and Transfer: Training does not collapse with 1k images, produces a recognizable lion, scales with additional data, and transfers directly to community models without retraining their neural networks.The experiments also examine content interpretation and transfer to Comic Diffusion and Protogen 3.4.
5. Conclusion
ControlNet reuses pretrained diffusion-model layers to learn conditional controls, with zero convolutions connecting the locked model and trainable copy. Experiments support its use across diverse conditions and broader applications.
- ControlNet learns conditional control for large pretrained text-to-image diffusion models.
- It reuses pretrained source-model layers as a deep encoder for learning specific conditions.
- Zero convolutions connect the original model and trainable copy while eliminating harmful training noise.
- Experiments verify effective control with single or multiple conditions, with or without prompts.
- Results across diverse conditioning datasets suggest that ControlNet can support a wider range of conditions and related applications.