Source-linked AI summary
InstantID: Zero-shot Identity-Preserving Generation in Seconds
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, Yao Hu
TL;DR
Existing personalized-generation methods face efficiency, compatibility, and face-fidelity limitations, while ID-preserving generation requires strong facial semantics. InstantID introduces a plug-and-play IdentityNet and image adapter that use one facial image with strong semantic and weak spatial conditioning; experiments report robust, efficient, and compatible identity preservation across models and controls.
Problem
Existing personalized and ID-embedding methods can require costly fine-tuning, lack compatibility with community models, or fail to preserve high-fidelity facial identity.
Method
InstantID combines a lightweight image adapter with IdentityNet, using facial and landmark images plus text prompts to provide strong semantic and weak spatial identity conditions.
Results
InstantID preserves identity from one facial image while retaining text-editing control and compatibility with pre-trained models, LoRAs, ControlNet, and varied styles.
Takeaways & Limitations
The plug-and-play, tuning-free design supports efficient identity-preserving personalization and downstream applications without additional model-training costs.
Abstract
from arXiv · showhide
There has been significant progress in personalized image synthesis with methods such as Textual Inversion, DreamBooth, and LoRA. Yet, their real-world applicability is hindered by high storage demands, lengthy fine-tuning processes, and the need for multiple reference images. Conversely, existing ID embedding-based methods, while requiring only a single forward inference, face challenges: they either necessitate extensive fine-tuning across numerous model parameters, lack compatibility with community pre-trained models, or fail to maintain high face fidelity. Addressing these limitations, we introduce InstantID, a powerful diffusion model-based solution. Our plug-and-play module adeptly handles image personalization in various styles using just a single facial image, while ensuring high fidelity. To achieve this, we design a novel IdentityNet by imposing strong semantic and weak spatial conditions, integrating facial and landmark images with textual prompts to steer the image generation. InstantID demonstrates exceptional performance and efficiency, proving highly beneficial in real-world applications where identity preservation is paramount. Moreover, our work seamlessly integrates with popular pre-trained text-to-image diffusion models like SD1.5 and SDXL, serving as an adaptable plugin. Our codes and pre-trained checkpoints will be available at https://github.com/InstantID/InstantID.
1 Introduction
Personalized image generation must preserve nuanced human facial identity while remaining efficient and compatible with existing diffusion models. InstantID addresses this with a lightweight, tuning-free plug-in that uses one facial reference image and supports varied styles and applications.
- Personalized generation targets images consistent with a reference subject’s style, appearance, or character identity, with applications including advertising, portraits, animation, and virtual try-ons.
- Human facial identity is difficult to preserve because it requires fine-grained semantic detail and higher fidelity than general styles or objects.
- Fine-tuning methods such as DreamBooth, Textual Inversion, and LoRA can be accurate but are resource-intensive and time-consuming.
- InstantID uses a lightweight adapter rather than fine-tuning full UNet parameters, making it pluggable and compatible with community pre-trained models.
- InstantID performs inference with a single forward propagation and one facial reference image, eliminating test-time fine-tuning while targeting high-fidelity personalization.
- The method preserves text-editing control, supports integration with custom models, and is positioned for applications including novel views, identity interpolation, multi-ID, and multi-style synthesis.
2 Related Work
Related work develops text-to-image diffusion systems and subject-driven personalization, including methods specifically focused on preserving facial identity. These approaches differ in their use of fine-tuning, embeddings, and image-conditioning mechanisms.
- 2.1 Text-to-image Diffusion Models: Text-to-image diffusion models generate images from text conditions, with Stable Diffusion reducing computation by performing diffusion in latent image space.
- 2.2 Subject-driven Text-to-image Generation: Subject-driven generation uses a limited set of subject images to create customized images guided by text descriptions.
- 2.3 ID Preserving Image Generation: ID-preserving generation is a subject-driven special case focused on strongly semantic facial attributes and broad real-world applications.
- 2.3 ID Preserving Image Generation: LoRA requires individual training for each new character, whereas optimization-free methods avoid additional fine-tuning or inversion processes.
- 2.3 ID Preserving Image Generation: Embedding-based methods such as PhotoMaker enhance identity-centric representations through image-encoder processing and merged class-image embeddings.
3 Methods
InstantID combines an identity embedding, an image-prompt adapter, and IdentityNet to preserve facial identity while adapting pre-trained diffusion models. Its design uses strong semantic conditioning, weak spatial control, decoupled attention, and frozen base-model parameters.
- ID Embedding: InstantID extracts semantic face features with a face encoder, projects them into text-feature space, and uses the result as a face embedding.The face embedding provides the identity representation used by the added modules.
- Image Adapter: A lightweight image adapter uses decoupled cross-attention to inject ID embeddings as image prompts alongside text prompts without changing the original model.Text and image features enter separate cross-attention pathways.
- IdentityNet: IdentityNet encodes detailed reference-face features with weak spatial control, using five facial landmarks and ID embeddings instead of text conditions.The five landmarks cover the eyes, nose, and mouth, while ID embeddings focus cross-attention on identity-related representations.
- Training: Only the Image Adapter and IdentityNet are optimized, while the pre-trained text-to-image diffusion model remains frozen.Training uses human image-text pairs and a diffusion noise-prediction objective with a task-specific IdentityNet image condition.
- Architecture: The method separates the image adapter from IdentityNet, allowing their image-condition weights to be adjusted independently during training and inference.This decoupling supports more controlled and targeted conditioning.
4 Experiments
Experiments show that InstantID preserves identity with a single facial image, supports prompt and spatial editing, and remains competitive with training-based and other identity-preserving methods.
- Qualitative Results: InstantID maintains rich facial semantics, including expression, age, and identity, when generation uses only the reference image.
- Qualitative Results: Text prompts can change gender, clothes, and hair color without degrading text-control capabilities while maintaining identity consistency.
- Qualitative Results: InstantID integrates with pre-trained ControlNet models for flexible spatial control using canny and depth conditions.
- Ablative Study: IdentityNet alone achieves good ID retention, while adding the Image Adapter further enhances facial-detail restoration.
- Ablative Study: More reference images generally improve quality, but a single image still produces remarkable fidelity using averaged ID embeddings for multiple references.
- Comparison to Previous Methods: InstantID achieves competitive results with pre-trained character LoRAs without training and flexibly blends faces with backgrounds, especially in non-realistic styles.
5 Conclusions and Future Work
InstantID provides zero-shot, identity-preserving personalization from one facial image through a plug-and-play module, while maintaining high fidelity and compatibility with other models. Its remaining challenge is that coupled facial attributes make face editing difficult.
- InstantID enables zero-shot personalization in any style from one facial image while maintaining high fidelity.
- The method combines an Image Adapter for facial detail fidelity with an IdentityNet for strong identity control.
- InstantID preserves the original model’s text-editing capabilities and integrates with LoRAs and ControlNet without additional costs.
- Coupled facial attributes in the ID embedding, including gender and age information, make face editing challenging.
A.1 Implementation Detail
The implementation extracts spatial controls from a reference image to guide generation while retaining the reference pose information.
- Canny and depth images are extracted from the same reference pose image as the spatial control.
- Spatial controls are extracted from a reference image for use in customized image generation.
A.2 Supplementary Results
Supplementary results examine how IdentityNet and Image Adapter weighting affects generation and demonstrate novel-view, identity-interpolation, and multi-identity synthesis capabilities.
- Figure 9 evaluates generation across different Image Adapter and IdentityNet weight strengths.The x-axis represents Image Adapter weight strength, while the y-axis represents IdentityNet weight strength.
- Figure 10 presents novel-view synthesis under arbitrary poses.
- Figure 11 presents identity interpolation between different characters.
- Figure 12 presents multi-identity synthesis with regional control.