Source-linked AI summary
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, Wei Yang
TL;DR
Text prompting can require complex engineering and may not express complex scenes, while existing image-prompt fine-tuning is computationally costly and incompatible with some controls. IP-Adapter uses decoupled cross-attention with a frozen pretrained diffusion model, achieving comparable or better performance with 22M parameters while supporting reuse, structural controls, and multimodal prompting.
Problem
Text prompts require complex engineering and are limited for expressing complex scenes, while existing image-prompt approaches can be costly, incompatible with text or structural controls, and prone to coarse image conditioning.
Method
IP-Adapter adds separate cross-attention layers for image features while freezing the pretrained text-to-image diffusion model.
Results
22M parameters achieve performance comparable to or better than fully fine-tuned image-prompt models and existing adapters.
Takeaways & Limitations
The adapter can be reused with custom models from the same base model, combined with structural control tools, and used alongside text prompts for multimodal generation.
Takeaways & Limitations
IP-Adapter resembles reference images in content and style but cannot produce images highly consistent with a given subject like Textual Inversion and DreamBooth.
Abstract
from arXiv · showhide
Recent years have witnessed the strong power of large text-to-image diffusion models for the impressive generative capability to create high-fidelity images. However, it is very tricky to generate desired images using only text prompt as it often involves complex prompt engineering. An alternative to text prompt is image prompt, as the saying goes: "an image is worth a thousand words". Although existing methods of direct fine-tuning from pretrained models are effective, they require large computing resources and are not compatible with other base models, text prompt, and structural controls. In this paper, we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Despite the simplicity of our method, an IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model. As we freeze the pretrained diffusion model, the proposed IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. With the benefit of the decoupled cross-attention strategy, the image prompt can also work well with the text prompt to achieve multimodal image generation. The project page is available at \url{https://ip-adapter.github.io}.
1 Introduction
IP-Adapter addresses the limitations of text-only and prior image-prompt methods with a lightweight, decoupled adapter for pretrained text-to-image diffusion models. It preserves text prompting while supporting multimodal and structurally controlled generation.
- Complex prompt engineering and text’s limited ability to express scenes motivate using images as richer prompts for image generation.
- Prior direct fine-tuning methods require substantial computation, eliminate original text generation, and are difficult to reuse with custom models.
- Merging image and text features in shared cross-attention can miss image-specific information and yield only coarse-grained control.
- IP-Adapter adds separate image-feature cross-attention layers while freezing the original UNet, enabling image prompting without modifying the pretrained model.
- 22M parameters make IP-Adapter comparable or better than fully fine-tuned image-prompt models in quantitative and qualitative experiments.
- IP-Adapter is reusable across custom models from the same base model and compatible with structural adapters such as ControlNet.
- Decoupled cross-attention keeps image prompting compatible with text prompting for multimodal image generation.
2 Related Work
Related work covers text-to-image diffusion models and lightweight adapters for structural, content, and style control. IP-Adapter introduces decoupled cross-attention to improve image-prompt adaptation beyond prior adapter methods.
- Text-to-image systems include autoregressive models that tokenize images and diffusion models that iteratively generate images from noise.
- Image-prompt diffusion models such as SD Image Variations and Stable unCLIP rely on fine-tuning, while DALL-E 2 conditions generation on image embeddings.
- Adapters reduce fine-tuning costs by adding trainable parameters while freezing the original model.
- ControlNet and T2I-adapter provide additional structural, color, and task-specific control for pretrained text-to-image diffusion models.
- Prior image-conditioned adapters support content or style control through feature injection, reference images, or appended image features.
- Prior lightweight adapters generally underperform fine-tuned image-prompt models, motivating IP-Adapter’s decoupled cross-attention mechanism.
3 Method
The method builds on text-conditioned diffusion models by adding a frozen-compatible image pathway through decoupled cross-attention, while retaining text conditioning and controllable inference.
- Preliminaries: Diffusion models generate images by iteratively denoising Gaussian noise, while conditioning can incorporate inputs such as text.The model predicts noise during training and uses iterative sampling during inference.
- Preliminaries: Stable Diffusion uses a UNet conditioned on CLIP text features and operates in latent space for greater efficiency than pixel-based diffusion models.The implementation uses Stable Diffusion as the example base model.
- Image Encoder: IP-Adapter extracts a global CLIP image embedding and projects it into a sequence of image features for insertion into the diffusion model.The projection network produces N = 4 features matching the text-feature dimension, while the CLIP image encoder remains frozen during training.
- Decoupled Cross-Attention: Decoupled cross-attention adds a separate image cross-attention layer for each original text cross-attention layer instead of concatenating image and text features.The image and text pathways share the query features but use separate key and value projections.
- Decoupled Cross-Attention: The final attention output adds image cross-attention to text cross-attention, with the new projection weights initialized from the original text projections.Only the added adapter parameters are trained while the original UNet remains frozen.
- Training and Inference: Randomly dropping image conditions during training enables classifier-free guidance, and inference can adjust image-condition strength with a weight factor.When the image-condition weight λ equals 0, the model becomes the original text-to-image diffusion model.
4.1 Experimental Setup
The experiments train IP-Adapter on a large multimodal dataset using SD v1.5, with a lightweight set of newly added image cross-attention modules.
- The training dataset contains about 10 million text-image pairs from LAION-2B and COYO-700M.
- The implementation uses SD v1.5 with OpenCLIP ViT-H/14 as the image encoder and adds one image cross-attention layer for each of 16 SD cross-attention layers.
- IP-Adapter has about 22M trainable parameters, including its projection network and adapted modules.
4.2 Comparison with Existing Methods
IP-Adapter is evaluated against methods trained from scratch, fine-tuned models, and adapters using COCO2017 image-prompt generation, with quantitative and qualitative comparisons.
- The comparison covers models trained from scratch, fine-tuned text-to-image models, and adapter-based methods.
- Quantitative evaluation uses 5,000 COCO2017 validation images, generating four images per image prompt for each method.
- CLIP-I measures similarity between generated-image and prompt-image embeddings, while CLIP-T measures similarity between generated images and the prompt captions.
- IP-Adapter is much better than other adapters and comparable or better than fine-tuned models with only 22M parameters.
- Qualitatively, IP-Adapter is mostly better than other adapters in image quality and alignment with the reference image.
4.3 More Results
Additional experiments show that IP-Adapter transfers across compatible models, combines with structural controls, supports image-to-image and inpainting, and enables multimodal prompting.
- Generalization and reuse: IP-Adapter can be reused with custom models fine-tuned from the same base diffusion model and works with text prompts and controllable tools.
- Generalization and reuse: IP-Adapter works well on Realistic Vision V4.0, Anything v4, and ReV Animated, and can directly apply to SD v1.4.
- Structure control: Combining IP-Adapter with ControlNet and T2I-Adapter produces controllable images using image prompts and additional structural conditions without fine-tuning.
- Structure control: With structural controls, IP-Adapter outperforms other methods in image quality and reference-image alignment.
- Image-to-image and inpainting: Replacing text prompts with image prompts enables image-guided image-to-image generation and inpainting.
- Multimodal prompts: Multimodal prompts combine image and text prompts, allowing text descriptions to edit attributes and change the subject’s scene.
- Multimodal prompts: Compared with other methods, IP-Adapter produces superior image quality and multimodal-prompt alignment.
4.4 Ablation Study
The ablation compares decoupled and simple adapters, then contrasts global and fine-grained image features. Decoupled attention improves image quality and consistency, while fine-grained features improve consistency but can reduce diversity.
- Importance of Decoupled Cross-Attention: The IP-Adapter generates higher-quality and more image-consistent results than the simple adapter without decoupled cross-attention.Both adapters were trained for 200,000 steps with the same configuration.
- Comparison of Fine-grained Features and Global Features: Global image embeddings may lose reference-image information, motivating a fine-grained adapter based on CLIP grid features and learned query tokens.The fine-grained design extracts penultimate-layer grid features and uses 16 learnable tokens with a lightweight transformer.
- Comparison of Fine-grained Features and Global Features: Fine-grained features generate more image-consistent results and capture spatial structure information, but may reduce the diversity of generated images.The comparison is shown in Figure 11.
- Comparison of Fine-grained Features and Global Features: Text prompts and structure maps can be combined with image prompts to produce more diverse images, including novel images guided by human poses.This provides additional conditions beyond the fine-grained image features.
5 Conclusions and Future Work
The paper concludes that IP-Adapter provides image prompting through decoupled cross-attention while retaining broad applicability and multimodal generation. Its main limitation is that it resembles reference content and style but does not achieve strong subject consistency.
- Conclusions: IP-Adapter uses separate cross-attention layers for image features and achieves comparable or better performance with only 22M parameters.The comparison is against some fully fine-tuned image prompt models and existing adapters.
- Conclusions: After one training run, IP-Adapter can integrate with custom models from the same base model and existing structural control tools, while combining image and text prompts.This supports multimodal image generation and expands the adapter's applicability.
- Future Work: IP-Adapter is limited to images resembling the reference in content and style, rather than highly consistent reproductions of a given subject.The authors identify stronger subject consistency as future work.