Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models

• Text descriptions can be inefficient or insufficient for detailed controls such as fine-grained layouts, and models also struggle to understand complex prompts.The paper motivates adding control modes such as user-drawn sketches and semantic masks alongside text.Source: §1 · p. 2
• Composer [19] requires costly from-scratch training on billion-scale data, while ControlNet [20], GLIGEN [22], and T2I-Adapter [21] use one adapter per condition, increasing cost and model size as controls grow.The paper also identifies composability among conditions as a challenge for these approaches.Source: §1 · p. 2
• Uni-ControlNet handles different conditions in one model and supports composable control, unlike existing methods that lack this unified framework or perform poorly at composition.The paper points to Table 1 for the comparison and Section 4 for composability results.Source: §1 · p. 2
• It uses exactly two shared adapters regardless of control count: a local encoder modulates noise features, while a global encoder adds conditional tokens to text for cross-attention.The two adapters can be trained separately without joint training while still supporting composition of multiple controls.Source: §1 · p. 2
• After one epoch on 10 million text-image pairs, Uni-ControlNet reports promising fidelity and controllability, and its adapter designs outperform those of ControlNet [20], GLIGEN [22], and T2I-Adapter [21].Figure 1 shows examples using one or multiple conditions.Source: §1 · p. 2

Text-to-Image Generation

• Early GAN-based text-to-image works [23] [24] [25] faced training instability and poor generalization to open-domain scenarios; autoregressive models [30] [31] [32] [33] showed strong zero-shot capability there, while diffusion models [6, 7, 34? –40] better modeled fine-grained structure and texture than autoregressive models.Diffusion models were originally proposed by [1] [2].Source: §2 · pp. 2–3

Controllable Diffusion Models

• Controllable diffusion models extend text-to-image diffusion with additional user controls and have recently attracted increasing attention.Source: §2 · p. 3
• Uni-ControlNet fine-tunes two adapters for local and global controls, addressing the cost, model-size, and composability limitations of prior controllable diffusion methods.Composer [19] trains from scratch at substantial cost; ControlNet [20], GLIGEN [22], and T2I-Adapter [21] fine-tune modules on frozen Stable Diffusion but use independent adapters per condition. GLIGEN does not support composite control, Multi-ControlNet [20] adapters are isolated, and tested CoAdapter [21] shows inadequate composability; Uni-ControlNet is reported to surpass ControlNet, GLIGEN, and T2I-Adapter in controllability and quality.Source: §2 · p. 4

3.1 Preliminary

• Stable Diffusion denoises with a UNet encoder, middle block, and decoder linked by skip connections, while cross-attention uses noise features for queries and text embeddings for keys and values.The encoder and decoder each have 12 blocks.Source: §3.1 · p. 4

3.2 Control Adapter

• The global adapter projects a CLIP image embedding into four condition tokens by default and appends them to text tokens for cross-attention.The projected embeddings are aligned with Stable Diffusion text embeddings by a feedforward condition encoder.Source: §3.2 · pp. 5–6
• The method illustrates seven local control types—Canny, MLSD, HED, sketch, Openpose, Midas depth, and segmentation mask—and a global CLIP image embedding.The CLIP embedding represents a reference content image and is intended to convey semantic content beyond simple image features.Source: §3.2 · p. 5
• The local adapter freezes Stable Diffusion, copies its encoder and middle block, and adds their control features to decoder inputs through zero convolutions.The decoder input combines the copied middle-block output with the corresponding encoder skip feature, modified by the local adapter.Source: §3.2 · p. 5
• Local conditions are concatenated and injected at four resolutions through feature denormalization, which uses learned spatial scale and shift coefficients to modulate normalized noise features.The resolutions are 64 × 64, 32 × 32, 16 × 16, and 8 × 8.Source: §3.2 · p. 5

3.3 Training Strategy

• The two adapters are trained separately with condition dropout, then combined at inference to compose local and global controls without joint fine-tuning.Dropped conditions are represented by zero-valued input channels; training also randomly keeps or drops all conditions.Source: §3.3 · p. 6

4 Experiments

• The experiments fine-tune on 10 million LAION text-image pairs for one epoch and use seven local conditions plus CLIP image embeddings as a global condition.At inference, the adapters are merged; sampling uses 50 DDIM steps and classifier-free guidance scale 7.5.Source: §4 · pp. 6–7

4.1 Controllable Generation Results

• Single and composed controls produce faithful examples, including sketch-guided dogs, pose-and-sketch-guided scenes, and images combining depth maps with global image embeddings.In examples, global embeddings can change a dog’s background, while depth and global image conditions guide an elephant-shaped forest without a text prompt.Source: §4.1 · p. 7

4.2 Comparison with Existing Methods

• Uni-ControlNet achieves superior FID across most single-condition settings compared with existing approaches.Evaluation uses 5,000 generated images from COCO2017, with one randomly selected caption per image; Table 2 reports FID by condition.Source: §4.2 · p. 7, Table 2 · p. 8
• In the COCO2017 validation evaluation of single conditions, Uni-ControlNet outperforms baselines in 4 of 8 controllability metrics; ControlNet leads 3 and T2I-Adapter 1, while Uni-ControlNet uses one model.T2I-Adapter has no MLSD or HED results; GLIGEN has no MLSD or sketch results.Source: §4.2 · pp. 7–8
• In qualitative comparisons, Uni-ControlNet aligns slightly better with single conditions and fuses composed local and global controls more effectively than competing methods.Multi-ControlNet and CoAdapter sometimes omit objects or produce poorly integrated, unrealistic combinations; Uni-ControlNet generates a Minion-shaped bus from a Minion edge and bus image condition.Source: §4.2 · p. 8, Figure 6 · p. 9, Figure 5 · p. 9

4.3 Ablation Analysis

• The proposed multi-scale local-condition injection performs better under most settings than SPADE interpolation and input-layer-only injection.The alternatives show poor condition alignment or composite harmony; the proposed feature extractor projects conditions into layer-specific latent spaces to align them with noise features.Source: §4.3 · pp. 8–9, Table 4 · p. 9, Figure 7 · p. 10
• Adding the global condition only to the control adapter fails to convey that condition to the final image without an extended prompt in the main diffusion model.Source: §4.3 · p. 9, Figure 7 · p. 10
• The default separate-training strategy and Train-S2 outperform joint-from-scratch Train-S1, while further joint fine-tuning brings little gain in most cases.Train-S1 can underlearn the global adapter when local conditions provide richer guidance; for global-condition visual tests, no text prompts are used.Source: Table 4 · p. 9, §4.3 · p. 10, Figure 8 · p. 10

Conclusion and Social Impact

• Uni-ControlNet uses two adapters across conditions, unlike methods that often require an independent adapter per condition, reducing fine-tuning cost and model size while facilitating composability.Its design enables simultaneous use of multiple conditions.Source: §5 · p. 10
• Extensive experiments validate Uni-ControlNet’s effectiveness, showing improved controllability, generation fidelity, and composability.Source: §5 · p. 10
• The system empowers artists, designers, and content creators, but misuse risks call for responsible deployment, ethical regulations, and special image flags for transparency.Source: §5 · p. 10

A The Weight of the Global Condition

• Increasing λ makes global-condition elements more prominent while reducing text influence, as illustrated by shrinking forests and cities and expanding houses and desert coverage.The authors propose adjusting λ to generate desired results flexibly.Source: A The Weight of the Global Condition · p. 14, Figure 9 · p. 14

B Condition Conflicts

• The authors present conflicting-condition testing as an analysis of conflict resolution, noting that such incompatible controls are rare in real-world applications.They suggest the analysis may help design more robust condition integration for diverse and ambiguous situations.Source: B Condition Conflicts · p. 14
• HED is strongest, followed by Canny, sketch, depth, MLSD, and segmentation, while Openpose is weakest and usually ignored in conflicts.HED boundaries prevail in overlaps; Canny may preserve a small element from the other condition, while Openpose yields recognizable human elements only with segmentation.Source: B Condition Conflicts · p. 14, Figure 10 · p. 15
• The experiment combines two dog conditions and uses room detection for MLSD and a man for Openpose, with prompts matched to the condition pair.The prompts include “dog,” “dog and room,” and “dog and man.”Source: Figure 10 · p. 15

C Hand-drawn Sketches

• Despite distribution gaps from model-generated sketches, Uni-ControlNet handles hand-drawn sketches pretty well.Figure 11 presents visualizations of the hand-drawn-sketch results.Source: C Hand-drawn Sketches · p. 16, Figure 11 · p. 16

D Extension to New Conditions

• The extension experiment adds a condition by modifying the feature extractor’s first convolutional input and compares four retraining strategies.R1 retrains the entire feature extractor; R2 retrains only the pre-feature extractor that projects conditions from resolution 512 to 64.Source: D Extension to New Conditions · p. 16
• The extension study also tests retraining only the first convolutional layer (R3) or no retraining, with that layer randomly initialized (R4).Source: D Extension to New Conditions · p. 16
• For the extension experiment, the copied local-adapter encoder stays fixed during retraining on 300k samples, adding Canny to six existing local conditions.The tested change is from [MLSD + HED + Sketch + OpenPose + Depth + Seg] to [MLSD + HED + Sketch + OpenPose + Depth + Seg + Canny].Source: D Extension to New Conditions · p. 16

E Composite Control of Conditions with the Same Type

• Users can combine same-type conditions before model input, but the paper investigates how to achieve composite control of such conditions.Source: E Composite Control of Conditions with… · p. 16
• The proposed Uni-Channels strategy adds three condition channels, increasing a 21-channel input to 24 channels when representing seven local conditions.Figure 13 presents visualizations of composite control for two conditions of the same type.Source: E Composite Control of Conditions with… · p. 17, Figure 13 · p. 17

F Comparison with Stable Diffusion 2.1

• The comparison uses SD2-depth for depth-map inputs and SD2-unclip for reference-image inputs.Visualization results are shown in Figure 14.Source: F Comparison with Stable Diffusion 2.1 · p. 17
• The section reports FID and CLIP-score comparisons between Uni-ControlNet and Stable Diffusion 2.1.The quantitative results are presented in Tables 5 and 6; the supplied passages do not state their values.Source: Table 5 · p. 17, Table 6 · p. 17, F Comparison with Stable Diffusion 2.1 · p. 17
• SD2-depth and SD2-unclip outperform ControlNet, GLIGEN, and T2I-Adapter; whole-model fine-tuning is identified as a key factor in their performance.The SD2 models fine-tune the whole model to learn depth maps or reference images, rather than only fine-tuning adapters.Source: F Comparison with Stable Diffusion 2.1 · p. 17

CLIP Score

• Uni-ControlNet and Training-S2 consistently outperform other condition-injection and training strategies, while Training-S2 joint fine-tuning yields no obvious gains in most cases.These findings are reported as consistent with the main paper’s Sections 4.2 and 4.3.Source: G More Quantitative Results · p. 17
• Uni-ControlNet demonstrates superior CLIP-score performance across most conditions versus existing controllable diffusion models.The comparison follows the main paper’s Sections 4.2 and 4.3.Source: G More Quantitative Results · p. 17, Table 7 · p. 18

User Study

• The user study evaluates 20 cases per setting with 20 users, comparing Uni-ControlNet against ControlNet [20], GLIGEN [22], and T2I-Adapter [21] on image quality, text match, and condition alignment.Users select the best generated image for each metric; Figure 15 presents preference rates for the single-condition setting.Source: G More Quantitative Results · p. 18, Figure 15 · p. 18

H More Visualization Results

• Figure 17 presents additional visual results for the single-condition setting.These are among the additional qualitative results described in Section H.Source: H More Visualization Results · p. 18, Figure 17 · p. 20
• Figure 18 shows multi-condition results, including a four-condition case in rows 7–8.The paper identifies rows 7–8 as the more challenging four-condition example.Source: H More Visualization Results · p. 18, Figure 18 · p. 21

I Adapter Details

• Figure 19 details the local control adapter.Section I points to Figures 19–21 for details of the proposed local and global control adapters.Source: I Adapter Details · p. 19, Figure 19 · p. 22
• Figure 20 details the ResBlock with FDN in the local control adapter.It is presented alongside the local adapter diagram in Figure 19.Source: I Adapter Details · p. 19, Figure 20 · p. 22
• Figure 21 details the global control adapter: d is the dimension of text-token embedding, and K is the number of global tokens.Figures 19, 20 and 21 provide details of the proposed local and global control adapters.Source: I Adapter Details · p. 19, Figure 21 · p. 23