Source-linked AI summary
Insert Anything: Image Insertion via In-Context Editing in DiT
Wensong Song, Hong Jiang, Zongxing Yang, Ruijie Quan, Yi Yang
TL;DR
Reference-based image insertion lacks a unified solution spanning diverse tasks and flexible control modes. Insert Anything trains a DiT-based framework with in-context editing on AnyInsertion, and experiments report consistent superiority across three benchmarks.
Problem
Existing insertion methods and datasets often focus on specific tasks or a single control mode, limiting comprehensive reference-based image editing.
Method
Insert Anything uses DiT multimodal attention and in-context diptych and triptych prompting to support mask- and text-guided person, object, and garment insertion.
Results
Insert Anything consistently achieves state-of-the-art performance across person, object, and garment insertion on AnyInsertion, DreamBooth, and VTON-HD.
Takeaways & Limitations
The framework provides a unified approach for preserving reference identity while integrating inserted elements with target-scene context.
Abstract
from arXiv · showhide
This work presents Insert Anything, a unified framework for reference-based image insertion that seamlessly integrates objects from reference images into target scenes under flexible, user-specified control guidance. Instead of training separate models for individual tasks, our approach is trained once on our new AnyInsertion dataset--comprising 120K prompt-image pairs covering diverse tasks such as person, object, and garment insertion--and effortlessly generalizes to a wide range of insertion scenarios. Such a challenging setting requires capturing both identity features and fine-grained details, while allowing versatile local adaptations in style, color, and texture. To this end, we propose to leverage the multimodal attention of the Diffusion Transformer (DiT) to support both mask- and text-guided editing. Furthermore, we introduce an in-context editing mechanism that treats the reference image as contextual information, employing two prompting strategies to harmonize the inserted elements with the target scene while faithfully preserving their distinctive features. Extensive experiments on AnyInsertion, DreamBooth, and VTON-HD benchmarks demonstrate that our method consistently outperforms existing alternatives, underscoring its great potential in real-world applications such as creative content generation, virtual try-on, and scene composition.
1. Introduction
Insert Anything addresses task and control limitations in reference-based image insertion with one unified framework trained on AnyInsertion. It supports person, object, and garment insertion under mask or text guidance and achieves state-of-the-art results across multiple benchmarks.
- Motivation: Existing insertion methods typically specialize in one task and one control mode, limiting broader applicability and creative flexibility.The cited challenges include task-specific datasets and either mask-guided or text-guided editing.
- Dataset: AnyInsertion covers person, object, and garment insertion with diverse categories and supports both mask- and text-guided editing.The dataset is designed to provide flexible training and evaluation across control modes.
- Method: Insert Anything uses DiT multimodal attention to jointly model text, masks, and image patches for unified control.The framework is intended to support either mask-guided or text-guided image editing.
- Method: In-context editing treats reference images as contextual content and uses diptych and triptych prompting to preserve identity while integrating elements into target scenes.The approach is designed to model interactions between inserted elements and surrounding context.
- Results: Insert Anything consistently achieves state-of-the-art performance on AnyInsertion, DreamBooth, and VTON-HD across human, object, and garment insertion.The evaluations cover the proposed dataset and two additional benchmarks.
2. Related Work
Prior image-insertion methods differ by task specificity and control strategy, while Insert Anything is positioned as a comprehensive framework spanning the main insertion types and both prompt modes.
- Task-specific methods: Prior work includes task-specific person and garment insertion methods alongside general object-editing systems.Examples include inpainting, pose-guided personalization, virtual try-on, and reference-feature extraction approaches.
- Control strategies: Existing general editing methods support partial combinations of insertion tasks or control modes rather than the full combination.The cited comparison distinguishes mask-guided systems, text-guided systems, and methods with only partial insertion support.
- Positioning: Insert Anything differs by targeting all three insertion types with both mask and text prompt support.The related-work discussion identifies this combination as the framework’s distinguishing scope.
3. AnyInsertion Dataset
AnyInsertion is constructed to address limited categories, restricted prompt types, and insufficient image quality in existing insertion datasets. It combines diverse sources and supports both mask- and text-prompted tasks.
- Dataset motivation: Existing datasets are limited by narrow categories, single prompt types, and low-quality video-derived images.Examples include garment-only or animal-and-plant-focused data, text-only or mask-only prompts, and motion blur or low resolution.
- Dataset comparison: AnyInsertion provides diverse categories, higher-resolution images, both mask- and text-prompts, and more samples than compared datasets.These properties are stated in the dataset comparison.
- Data construction: The dataset pairs reference images containing insertion elements with target images that provide the surrounding scene context.Object, accessory, person, and garment data are assembled from image or video sources using matching and filtering procedures.
- Dataset statistics: AnyInsertion contains 159,908 training samples: 58,188 mask-prompt pairs and 101,720 text-prompt pairs.The text-prompt records include reference images, reference masks, source images, target images, and texts.
- Dataset diversity: The dataset covers human subjects, daily necessities, garments, furniture, and various objects for multiple insertion tasks.The diversity is presented as supporting person, object, and garment insertion.
Image Encoder
The framework encodes concatenated reference, source, and mask inputs alongside image and text guidance before processing them with learnable DiT blocks for in-context insertion.
- Input encoding: Polyptych inputs combine reference, source, and masks, while a frozen VAE encoder preserves high-frequency details.Image and text encoders provide additional semantic guidance.
- Feature processing: The resulting image and text embeddings are combined and fed into learnable DiT transformer blocks for in-context learning.This design supports image insertion guided by either masks or text.
- Data context: The dataset includes a text-prompt subset containing 16 object-insertion pairs and 22 garment-insertion pairs.These counts are reported for the text-prompt subset.
4. Insert Anything Model
Insert Anything performs reference-based insertion through in-context editing, using diptych or triptych inputs and multimodal DiT attention to support mask- and text-guided control. Evaluations on mask- and text-prompt object insertion report performance exceeding existing methods across key metrics.
- Model overview: The framework takes a reference image, source image, and mask- or text-based control prompt to integrate a reference element into the source scene.Its goal is to preserve the reference element’s identity while editing the source context.
- In-context editing: Background removal isolates the reference element before in-context editing with either mask-prompt or text-prompt inputs.The reference background is removed using Grounding-DINO and SAM before constructing the editing inputs.
- In-context editing: Mask-prompt editing uses a diptych that concatenates the processed reference image with a partially masked source image and marks the insertion region in a binary mask.The reference panel receives zeros and the insertion region receives ones in the diptych mask.
- In-context editing: Text-prompt editing uses a triptych containing the processed reference image, unmodified source image, and a fully masked generation region.Its binary mask marks the reference and source regions with zeros and the generation region with ones.
- Evaluation: Insert Anything substantially surpasses AnyEdit on text-prompt object insertion and achieves state-of-the-art performance across mask-prompt object-insertion metrics against AnyDoor, MimicBrush, and ACE++.The reported comparisons cover AnyInsertion and DreamBooth for mask-prompt evaluation, and AnyInsertion for text-prompt evaluation.
- Multiple control modes: Multimodal DiT attention fuses image-branch features from reference images, source images, and masks with text-branch semantic guidance.The query, key, and value components concatenate text and image features for joint attention.
5. Experiments
Experiments evaluate Insert Anything across mask- and text-prompt insertion on multiple datasets, comparing quantitative and qualitative performance, ablations, and implementation settings. The method consistently outperforms existing alternatives across person, object, and garment insertion while preserving identity, detail, and contextual coherence.
- Experimental Setup: Evaluations cover AnyInsertion, DreamBooth, and VTON-HD, using 768×768 images, 50 denoising iterations, and separate mask- and text-prompt training batches.The model was trained for 5000 steps with batch sizes of 8 for mask prompts and 6 for text prompts.
- Quantitative Results: On mask-prompt garment insertion, Insert Anything surpasses OOTDiffusion and CatVTON across all metrics on AnyInsertion and VTON-HD.On VTON-HD, LPIPS decreases from 0.0513 to 0.0484, while PSNR rises from 25.64 to 26.10 and SSIM from 0.8903 to 0.9161.
- Qualitative Results: Qualitative comparisons show stronger identity preservation, fine-detail retention, and visual coherence across person, object, and garment insertion than existing methods.The comparisons include mask-prompt baselines and text-prompt comparisons with AnyEdit.
- Quantitative Results: Insert Anything consistently outperforms existing methods across mask- and text-prompt object insertion metrics.For mask prompts, SSIM improves from 0.7648 to 0.8791 on AnyInsertion and from 0.6039 to 0.7820 on DreamBooth; for text prompts, LPIPS decreases from 0.3473 to 0.2011.
- Ablation Study: Ablations show that removing polyptych in-context editing, semantic guidance, or AnyInsertion data degrades visual fidelity and sacrifices fine details or semantic cues.The ablation study uses weighted averages over person, object, and garment mask-prompt insertion results.
6. Conclusion
The conclusion presents Insert Anything as a unified reference-based image insertion framework supporting mask- and text-guided control across diverse tasks. It combines the AnyInsertion dataset, DiT-based processing, and in-context prompting, with experiments showing consistent superiority across three benchmarks.
- Conclusion: Insert Anything supports mask- and text-guided reference-based insertion across person, object, and garment tasks.The framework is designed to overcome the task and control limitations of specialized approaches.
- Conclusion: The framework uses AnyInsertion data, DiT capabilities, and diptych and triptych prompting to preserve identity while maintaining visual harmony.These components support insertion of reference elements into target scenes.
- Conclusion: Experiments on three benchmarks show consistent outperformance of state-of-the-art methods across person, object, and garment insertion.The conclusion frames the method as a versatile solution for reference-based image editing applications.
Supplementary Material
The supplementary material extends the paper with implementation details, dataset construction information, additional ablations, future-work directions, and societal-impact discussion.
- Supplementary Material: The supplementary material covers hybrid masking, adaptive cropping, dataset construction, additional ablation analyses, future work, and broader societal impacts.These topics are organized across supplementary sections A through E.
A. More Implementation Details
The implementation details adapt masking and cropping to insertion categories and editing-region size. The method combines varied mask types with adaptive zooming to balance precise guidance, pose information, fine-detail preservation, and contextual coherence.
- Hybrid Masking Approach: Training exclusively with segmentation masks limits free-form mask handling, while training solely with box masks can weaken pose guidance and adaptability to non-rectangular shapes.The hybrid strategy is motivated by these complementary limitations.
- Hybrid Masking Approach: Object and garment insertion use augmented box masks, while person insertion combines box, augmented box, and dilated segmentation masks.The category-specific design provides different forms of shape variation and pose guidance.
- Adaptive Crop Strategy: Adaptive crop-and-zoom preserves high-frequency details when an inserted element occupies a small target-image region.The editing region and surrounding pixels are cropped so the insertion receives more attention during processing.
- Adaptive Crop Strategy: Crop size must balance contextual integration against emphasis on the editing region: overly small crops lack context, while overly large crops dilute attention.The strategy adjusts crop size according to the editing region’s proportional area.
- Adaptive Crop Strategy: The crop factor uses the editing-region area ratio r and threshold T, with β defining the minimum crop-area fraction for vanishingly small regions.The experiments set β = 0.6 and T = 0.1.
B. More Dataset Information
The dataset combines e-commerce, video, and multi-view object sources to construct diverse reference-image pairs. A text-to-mask and inpainting pipeline handles shape changes while preserving unmodified content.
- Data Sources: E-commerce images provide diverse reference objects and human-containing images, including apparel and accessories.Human detection filters images containing people before further curation.
- Data Sources: LightGlue selects the most similar reference view when webpages contain multiple images of the same object.This matches source images with corresponding viewpoints during curation.
- Data Sources: HumanVid pairs frames with varied body poses but stable head orientations by selecting keyframes under pose constraints.The procedure chooses distant frames while limiting head-pose variation, reducing motion blur.
- Shape Changes: Shape-changing data are created through text-guided editing, mask extraction and combination, then inpainting refinement.The combined mask guides source-image generation while preserving regions that were not modified.
C. More Ablation Results
Ablations examine adaptive cropping and hybrid masking, while the discussion identifies physics constraints, expert specialization, and richer conditioning as future directions. The framework is also presented as useful for creative, accessible, personalized, and efficient image editing, alongside misuse and trust risks.
- Ablation Results: Removing adaptive cropping degrades small-object handling and high-frequency detail preservation.The ablation supports adaptive cropping during inference for retaining fine details.
- Ablation Results: Using only instance or box masks compromises the balance between mask precision and pose adaptability.The combined hybrid masking strategy is described as achieving balanced, high-quality insertion results.
- Future Directions: Physics-conformant constraints are proposed to make inserted elements more naturalistic when objects or scene compositions change.The proposed direction targets adherence to physical laws during editing.
- Future Directions: Mixture-of-Experts specialization is proposed to improve efficiency and accuracy across diverse editing scenarios.The discussion describes experts for challenges such as person insertion and object manipulation.
- Applications: The framework is described as improving creative expression, accessibility, virtual try-on, personalization, and digital-media workflow efficiency.These claimed applications include simplifying complex editing and reducing reliance on multiple specialized tools.
- Risks: Seamless image insertion raises risks involving deepfakes, privacy, intellectual property, and declining trust in visual media.The stated concerns include deceptive content, non-consensual manipulation, unauthorized copyrighted-material use, and difficulty distinguishing authentic images.