Source-linked AI summary
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie
TL;DR
Text prompts alone do not fully expose the controllable structure and color knowledge learned by large T2I models. The paper introduces lightweight T2I-Adapters that align external conditions with frozen pretrained models, providing flexible and composable guidance. The resulting adapters support color and structure control while preserving the original generation ability and generalizing to compatible custom models.
Problem
Text prompts provide limited flexible and accurate control over structures and semantics despite T2I models implicitly learning these capabilities.
Method
T2I-Adapter learns a low-cost alignment between external control signals and internal knowledge in frozen pretrained T2I models.
Results
T2I-Adapter supports spatial color and elaborate structure control, preserves original generation ability, enables multi-condition composition, and generalizes to compatible custom models.
Takeaways & Limitations
Lightweight adapters provide practical, condition-specific control for pretrained T2I models without requiring changes to their original generation networks.
Abstract
from arXiv · showhide
The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., color and structure) is needed. In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly. Specifically, we propose to learn simple and lightweight T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models. In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results. Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability. Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications.
1. Introduction
Text-to-image models learn rich visual information but text prompts alone provide limited, unstable control over precise structures and semantics. T2I-Adapter addresses this alignment problem with lightweight, plug-and-play adapters that add condition-specific guidance without changing the original model.
- Motivation: T2I models capture information across textures, edges, structures, and semantics, but text prompts do not always provide accurate structure guidance.This limits flexible control and can make results uncontrolled or unstable, especially for imaginative scenarios.
- Approach: T2I-Adapter learns an alignment between external control signals and internal knowledge in pretrained T2I models using a lightweight model and relatively little data.The pretrained model supplies generation ability while the adapter provides extra guidance.
- Properties: The adapters are plug-and-play: they do not affect the original network topology or generation ability of existing T2I diffusion models.They are inserted as additional guidance networks rather than requiring changes to the original model.
- Properties: Adapters support varied control conditions, including spatial color control and elaborate structure control.The paper presents these as distinct application areas for condition-specific adapters.
- Properties: More than one adapter can be easily composed to achieve multi-condition control.Composability allows multiple control signals to guide one generation process.
- Properties: Once trained, adapters can be used on custom models fine-tuned from the same T2I model.The contribution claims generalization beyond the model used during adapter training.
2. Related Work
Earlier image-synthesis methods included unconditional generation and conditional translation from modalities such as sketches, segmentation maps, keyposes, and text. Diffusion models substantially advanced text-to-image generation, while adapter-based transfer remained an open challenge for pretrained T2I models.
- Image Synthesis and Translation: Early image-synthesis methods commonly performed unconditional generation, whereas later conditional methods translated inputs such as sketches, segmentation maps, and keyposes into natural images.Text also became an important conditioning modality for image generation.
- Image Synthesis and Translation: Many conditional synthesis methods treated different conditions independently with task-specific training, although some work explored multimodal conditions.This establishes the prior context for methods supporting multiple controls.
- Diffusion Models: Diffusion models generate images from Gaussian noise through iterative denoising and achieved major success in image synthesis.The diffusion process adds noise over iterations, while the reverse process recovers the image.
- Diffusion Models: Recent diffusion systems improved text-to-image generation by injecting text conditions into denoising, commonly through cross-attention in feature space.Examples include Glide, DALL-E, CogView, Make-a-Scene, Stable Diffusion, and Imagen.
- Adapters: Adapters originated as compact, extensible alternatives to fine-tuning large pretrained models separately for each downstream task.Related work applied adapter ideas in NLP and computer vision.
- Adapters: Using low-cost adapters with pretrained text-to-image models remained an open challenge.The paper positions T2I-Adapter as addressing this gap.
3. Method
T2I-Adapter augments a frozen Stable Diffusion model with condition-specific guidance features, enabling more accurate structural and color control. Lightweight adapters can be trained efficiently, composed without extra training, and strengthened through early-stage-focused sampling.
- Stable Diffusion and the control gap: Stable Diffusion uses latent-space denoising with text conditioning, but text alone may provide insufficient structural guidance in complex scenarios.The model predicts noise at each denoising step and progressively reconstructs the clean latent representation.
- Adapter architecture: T2I-Adapter adds condition-derived guidance features to a frozen Stable Diffusion model while extracting control information through separate adapters.The adapters align external conditions with the model’s internal knowledge without changing the pretrained model parameters.
- Condition-specific control: Different structural conditions, including sketches, depth maps, semantic segmentation maps, and keyposes, are supported through task-specific adapters.The same framework also introduces a spatial color palette for controlling hue and color distribution.
- Composable control: Multiple adapters can be combined by adding their guidance features with adjustable weights, enabling multi-condition control without additional training.For example, sketch guidance can control structure while a spatial color palette controls appearance.
- Sampling strategy: Guidance is most effective early in DDIM sampling, while later-stage guidance has little effect because the main generation content is determined early.This observation motivates non-uniform training-time sampling that increases the probability of early time steps.
- Sampling strategy: Cubic time-step sampling strengthens adapter guidance compared with uniform sampling, particularly for color control.The reported weakness of uniform sampling is especially evident in color guidance, while cubic sampling rectifies it.
4.1. Implementation Details
The implementation trains lightweight adapters for multiple control conditions and evaluates them against existing methods using COCO-based settings.
- Training uses 10 epochs, batch size 8, Adam, and a 1×10−5 learning rate on 512 × 512 inputs.The SD-V1.4 backbone is adapted on 4 NVIDIA Tesla 32G-V100 GPUs, with training completed within 3 days.
- The study covers sketch, semantic segmentation, keypoints, color, and depth conditions.Sketch and segmentation use COCO-derived data, while keypoints, color, and depth use 600K LAION-AESTHETICS image-text pairs.
- The comparison evaluates SPADE, OASIS, PITI, and SD alongside the proposed method.The visualization compares methods using generation results conditioned on guidance maps.
- Table 1 reports FID and CLIP Score on the COCO validation set, with the best result highlighted.
4.2. Comparison
The comparison evaluates controlled generation against GAN-based, diffusion-based, and original SD baselines, reporting stronger visual quality and performance for T2I-Adapter.
- On 5,000 COCO validation images, T2I-Adapter produces results described as more vivid and more similar to the source images than competing methods.Each method performs one random inference per image for the visualization comparison.
- The results show promising performance, while T2I-Adapter improves the performance of SD and brings greater regularity.The quantitative evaluation is reported in Table 1 using FID and CLIP Score.
4.3. Applications
The adapters support independent control, image editing, composition, and transfer across related diffusion models, covering color and structural guidance.
- 4.3.1 Single-Adapter Controlling: Independent adapters control color, sketch, keypose, semantic segmentation, and depth, including imaginative scenarios and free-hand sketches.The paper describes the adapter as robust in the small-cat free-hand sketch example.
- 4.3.1 Single-Adapter Controlling: Sketch-adapter guidance can support local inpainting edits that pure SD struggles to achieve from ambiguous text guidance.Users erase an undesired region and inject adapter guidance into SD’s inpainting mode.
- 4.3.2 Composable Controlling: Different adapters compose without retraining, with depth+keypose and sketch+color combinations demonstrating complementary control.
- 4.3.3 Generalizable Ability: Adapters trained on SD-V1.4 can control SD-V1.5 and other custom models trained from the same T2I model.The paper states that this generalization broadens applications after a single training process.
4.4. Ablation Study
The ablation studies examine guidance injection and adapter compression, alongside evidence that adapters generalize from SD-V1.4 to other models.
- Generalization: Adapters trained on SD-V1.4 can perform control on SD-V1.5 and custom models, demonstrating cross-model generalization.
- Guidance Injection: Injecting guidance into the encoder is preferred because its longer encoder-decoder pathway can further refine adapter guidance features.The SD model uses encoder and decoder scales of 64×64, 32×32, 16×16, and 8×8.
- Guidance Injection: Multi-scale guidance improves generation, whereas injecting guidance into both encoder and decoder increases control strength but limits texture richness.
- Adapter Complexity: Adapter-small and adapter-tiny contain 18M and 5M parameters after ×4 and ×8 intermediate-feature compression.The compression targets sparse condition maps and compares three adapter sizes by generation quality.
5. Conclusion and Limitation
T2I-Adapter uses lightweight alignment models to expose implicit capabilities in pretrained T2I systems for accurate, controllable generation. It supports broad applications and composable or transferable control, but multi-adapter guidance still requires manual adjustment.
- Conclusion: T2I-Adapter aligns condition information with internal knowledge in pretrained T2I models rather than learning new generation abilities.The approach is lightweight and low-cost.
- Conclusion: The adapter supports spatial color control and elaborate structure control without affecting the pretrained T2I model’s original generation ability.
- Conclusion: Multiple adapters can be composed to achieve multi-condition control.
- Conclusion: A trained T2I-Adapter can be directly used with custom models fine-tuned from the same T2I model.
- Conclusion: Extensive experiments demonstrate excellent controlling and promising generation quality across a wide range of applications.
- Limitation: Multi-adapter control remains limited because combining guidance features requires manual adjustment.