Source-linked AI summary
AnyDoor: Zero-shot Object-level Image Customization
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, Hengshuang Zhao
TL;DR
Object teleportation requires placing a target object accurately and seamlessly into a specified scene location, a capability previous methods support only incompletely. AnyDoor uses identity and frequency-aware detail features to guide a diffusion model, achieving high-fidelity zero-shot compositions with shape control and multi-subject editing.
Problem
Object teleportation requires accurate, seamless placement of a target object at a specified scene location, but prior methods lack identity consistency, location control, or practical zero-shot operation.
Method
AnyDoor extracts identity tokens and frequency-aware detail maps, then injects them into a pretrained text-to-image diffusion model to guide location-specific object composition.
Results
AnyDoor produces high-fidelity zero-shot compositions with shape control, multi-subject composition, and object moving or swapping capabilities.
Takeaways & Limitations
AnyDoor offers a general solution for image-input region-to-region mapping and supports applications including virtual try-on, object moving, and object swapping.
Takeaways & Limitations
Fine details such as small characters or logos remain difficult to preserve, while prior customization methods require multiple target images and lengthy tuning.
Abstract
from arXiv · showhide
This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations in a harmonious way. Instead of tuning parameters for each object, our model is trained only once and effortlessly generalizes to diverse object-scene combinations at the inference stage. Such a challenging zero-shot setting requires an adequate characterization of a certain object. To this end, we complement the commonly used identity feature with detail features, which are carefully designed to maintain texture details yet allow versatile local variations (e.g., lighting, orientation, posture, etc.), supporting the object in favorably blending with different surroundings. We further propose to borrow knowledge from video datasets, where we can observe various forms (i.e., along the time axis) of a single object, leading to stronger model generalizability and robustness. Extensive experiments demonstrate the superiority of our approach over existing alternatives as well as its great potential in real-world applications, such as virtual try-on and object moving. Project page is https://damo-vilab.github.io/AnyDoor-Page/.
1. Introduction
AnyDoor addresses zero-shot object teleportation by placing a target object into a specified scene location with identity-consistent, high-quality composition. It combines identity and frequency-aware detail features to support controllable customization and applications including composition, virtual try-on, object moving, and swapping.
- Problem: Object teleportation places a target object accurately and seamlessly into a desired location by regenerating a box- or mask-marked scene region.The target object serves as the template, supporting image composition, effect-image rendering, postermaking, and virtual try-on.
- Motivation: Prior methods either lacked identity-consistent content, especially for untrained categories, or could not specify a new concept’s location within a given scene.Paint-by-Example and Objectstitch edit regions from target-image templates, while customized synthesis methods generate new concepts without location specification.
- Method: AnyDoor generates high-quality, identity-consistent compositions zero-shot by combining discriminative identity tokens, frequency-aware detail maps, and background-scene interaction.An ID extractor produces identity tokens, while a detail extractor supplies detail maps as a supplement.
- Results: AnyDoor supports zero-shot new-concept synthesis with shape control and controllable editing of specific local scene regions.These capabilities are illustrated in the paper’s figure and enable multi-subject composition.
- Applications: Its generation fidelity and quality enable applications such as object moving and swapping, alongside broader image generation and editing tasks with image input.The paper positions AnyDoor as a foundation solution for these applications.
2. Related Work
Prior local image-editing methods primarily use text or image guidance to modify masked regions, replace objects, or preserve semantic consistency. Representative approaches include multi-step blending, segmentation-plus-inpainting, and CLIP-based image guidance.
- Local image editing: Local image-editing methods mainly focus on modifying masked image regions with text guidance.
- Local image editing: Blended Diffusion performs multi-step blending within the masked region to produce more harmonized outputs.
- Local image editing: Inpaint Anything combines SAM and Stable Diffusion to replace source-image objects with text-described targets.
- Local image editing: Paint-by-Example uses a CLIP image encoder to convert a target image into an embedding for guidance toward semantic consistency.
3. Method
AnyDoor represents target objects with complementary identity and detail features, injects them into a pre-trained diffusion model, and uses video and image data to learn appearance changes. Its method combines DINOv2-based identity extraction, collage and shape controls for detail generation, and modality-aware training data preparation.
- Identity feature: DINOv2 extracts global identity information, and a linear projector aligns concatenated tokens with the pre-trained text-to-image UNet embedding space.The identity extractor encodes the image as a global token T1×1536, producing projected ID tokens T257×1024.
- Detail feature: The detail extractor combines a background-removed object collage with shape and contour controls to preserve low-level appearance details and object gestures.It uses high-pass filters, RGB extraction, an eroded mask, and a ControlNet-style UNet encoder that produces hierarchical detail maps.
- Diffusion model: AnyDoor injects identity tokens and hierarchical detail maps into a pre-trained Stable Diffusion model to guide object-scene composition.Identity tokens replace text embeddings through cross-attention, while detail maps concatenate with decoder features at matching resolutions.
- Training data: Video-derived frame pairs provide realistic same-object variations across scenes, while segmentation, tracking, masking, and cropping construct training inputs and supervision.Two frames supply the target object and corresponding ground-truth instance, addressing the limitations of simple rotation, flip, and elastic augmentations.
- Training data: Adaptive timestep sampling lets video data contribute appearance variation and image data contribute high-quality details and versatile scenarios at different denoising stages.Video frames may suffer from low resolution or motion blur, whereas images lack appearance changes.
4. Experiments
Experiments show that AnyDoor preserves object identity and details while harmoniously adapting objects to new scenes, outperforming reference- and tuning-based alternatives in zero-shot settings. Ablations and applications further demonstrate the value of its design choices and broad control capabilities.
- Comparisons: AnyDoor produces highly faithful object details in zero-shot customization, whereas reference-based methods preserve only semantic consistency or coarse patterns for unseen concepts.Compared methods include Paint-by-Example, Graphit, IP-Adapter, and Stable Diffusion.
- Comparisons: AnyDoor avoids tuning-based methods’ multi-subject confusion while outperforming Paint-by-Example on new concepts and retaining strong fidelity.Paint-by-Example performs well on trained dog and cat categories but poorly on new concepts; DreamBooth, Custom Diffusion, and Cones improve fidelity yet mix subject attributes.
- Ablation studies: Ablations identify DINOv2 as the strongest ID-extractor backbone, while adaptive timestep sampling improves image quality and fidelity compared with even sampling.Even sampling provides better diversity but inferior quality and fidelity; the high-frequency detail map offers a satisfactory trade-off between detail preservation and pose or view variation.
- Applications: Without task-specific tuning, AnyDoor supports virtual try-on across benchmark and real-world scenarios, preserving garment appearance while enabling contour-controlled shape changes.The reported scenarios include people with different sexes, ages, and races, as well as complex backgrounds.
- Applications: AnyDoor extends to object moving, swapping, reshaping, and human pose transfer, preserving human face and garment identity with optional interactive controls.Object moving uses segmentation and inpainting, while pose transfer adds an extra ControlNet-copy for pose control.
5. Conclusion
AnyDoor enables object teleportation by combining discriminative identity and frequency-aware detail extraction. Trained on video and image data, it composites objects at specified scene locations with effective shape control and supports general region-to-region mapping.
- 5. Conclusion: AnyDoor characterizes target objects using discriminative identity and frequency-aware detail extractors.The identity extractor captures object identity, while the detail extractor is designed around frequency information.
- 5. Conclusion: Training on a large combination of video and image data supports object compositing at specified scene locations.The method places the target object into a scene image at a user-specified location.
- 5. Conclusion: AnyDoor provides a universal solution for general region-to-region mapping tasks with effective shape control.The conclusion identifies potential profitability across various applications.