PaperlayerView exact revision on arXiv

Source-linked AI summary

TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition

Shilin Lu, Yanzhu Liu, Adams Wai-Kin Kong

arXiv:2307.12493v4cs.CVcs.AI

TL;DR

Image-guided composition must integrate user-provided objects across visual domains while preserving their appearance and the background, but existing methods can require costly optimization or retraining. TF-ICON uses an exceptional prompt for accurate real-image inversion and composite attention injection for training-free composition, and the authors report outperforming state-of-the-art inversion and composition baselines. [p03]

  • Problem

    Cross-domain image-guided composition requires seamless object integration while preserving object identity, illumination consistency, and background appearance, whereas existing approaches can require costly optimization or retraining.

  • Method

    TF-ICON uses an exceptional prompt to invert real images into latent codes and injects composite self-attention maps during denoising, without additional training, fine-tuning, data, or optimization.

  • Results

    The authors report that TF-ICON outperforms state-of-the-art baselines for both real-image inversion and cross-domain image composition.

  • Takeaways & Limitations

    TF-ICON provides a training-free interface for image-guided composition across diverse domains using pretrained attention-based text-to-image diffusion models.

  • Takeaways & Limitations

    TF-ICON cannot generate object views that critically differ from the reference, because self-attention maps constrain alternative-view development.

Abstract

from arXiv · show

Text-driven diffusion models have exhibited impressive generative capabilities, enabling various image editing tasks. In this paper, we propose TF-ICON, a novel Training-Free Image COmpositioN framework that harnesses the power of text-driven diffusion models for cross-domain image-guided composition. This task aims to seamlessly integrate user-provided objects into a specific visual context. Current diffusion-based methods often involve costly instance-based optimization or finetuning of pretrained models on customized datasets, which can potentially undermine their rich prior. In contrast, TF-ICON can leverage off-the-shelf diffusion models to perform cross-domain image-guided composition without requiring additional training, finetuning, or optimization. Moreover, we introduce the exceptional prompt, which contains no information, to facilitate text-driven diffusion models in accurately inverting real images into latent representations, forming the basis for compositing. Our experiments show that equipping Stable Diffusion with the exceptional prompt outperforms state-of-the-art inversion methods on various datasets (CelebA-HQ, COCO, and ImageNet), and that TF-ICON surpasses prior baselines in versatile visual domains. Code is available at https://github.com/Shilin-LU/TF-ICON

1. Introduction

Image-guided composition must preserve object identity and background while reconciling illumination across different visual domains. TF-ICON addresses this with a training-free diffusion framework built on accurate inversion and attention-map injection.

  • Cross-domain image composition requires preserving illumination consistency, identifying features, object appearance, and background appearance.
  • Existing approaches rely on costly instance-based optimization or retraining on tailored datasets, potentially damaging pretrained models’ rich prior.
  • TF-ICON performs image-guided composition with pretrained attention-based diffusion models without additional training, fine-tuning, data, or optimization.
  • The method inverts real images into latent codes with an exceptional prompt, then composes them through denoising with gradually injected composite self-attention maps.
  • The exceptional prompt supports accurate invertibility, while the proposed framework is reported to outperform prior inversion and composition baselines.

2. Related Work

Prior work separates text-guided and image-guided composition and explores several inversion strategies for generative models. TF-ICON’s framework combines reference-image inversion with training-free composition components.

  • Image composition: Image composition is applied to electronic commerce, entertainment, and data augmentation, and includes text-guided and image-guided settings.
  • Image composition: Text-guided composition can produce attribute leakage, attribute interchange, and missing objects when prompts specify multiple objects.
  • TF-ICON: Figure 2 depicts TF-ICON using the exceptional prompt to invert main and reference images into noises that form a composition starting point.
  • TF-ICON: The composition process injects self-attention maps from main and reference reconstructions together with cross-attention between the images.
  • Image inversion: GAN inversion methods include latent-based optimization, encoders, and fine-tuning, while diffusion editing commonly uses DDIM inversion.

3. Preliminary

Diffusion probabilistic models perturb data toward a tractable noise prior through a forward SDE and generate samples by numerically solving the reverse-time process. Stable Diffusion applies this VP-SDE formulation in latent space.

  • The forward diffusion process gradually injects noise into data until the resulting sample is typically distributed as a tractable prior without information about the original data.
  • The reverse diffusion process runs from T to 0 and depends on a score function estimated by a neural network trained with a denoising objective.
  • The forward SDE uses f(·, t) as the drift coefficient and g(t) as the diffusion coefficient.
  • The trained score model enables numerical solution of the reverse SDE, with VE, VP, and sub-VP SDEs offering alternative perturbation constructions.
  • This work uses a pretrained text-to-image Latent Diffusion Model, Stable Diffusion, which applies the VP SDE in latent space.

4. Method

TF-ICON performs training-free image-guided composition by combining accurate image inversion with attention-based denoising. It uses an exceptional prompt for inversion, noise incorporation, and composite self-attention injection to preserve image appearance across domains.

  • Framework overview: TF-ICON uses pretrained attention-based text-to-image models for image-guided composition without training, fine-tuning, extra data, or optimization.The framework consists of image inversion and composition generation, and can operate within 20 sampling steps.
  • Image inversion: Accurate inversion is obtained by solving diffusion ODEs with high-order solvers rather than relying only on commonly used DDIM inversion.The paper employs DPM-Solver++ because improved alignment between forward and backward ODE trajectories yields better latent representations.
  • Image inversion: The exceptional prompt removes prompt information by assigning a common value to all tokens and eliminating positional embeddings during inversion.It is used only for image inversion, while a normal prompt assists composition generation.
  • Composition generation: Composite self-attention injection combines main-image and reference-image self-attention maps with cross-attention, placing them according to patch indices.This design preserves foreground and background appearance while enabling domain switching during denoising.
  • Composition generation: The composition starts from merged inverted noises and Gaussian noise, with masks defining the user region, reference segmentation, and transition area.The Gaussian component helps the diffusion prior inpaint smooth transitions between masked and unmasked regions.
  • Composition generation: Background preservation must be applied gradually across noise levels because preserving it only at the final denoising step produces noticeable artifacts.The gradual implementation produces higher-quality results than the naïve final-step-only strategy.

5. Experiments

Experiments evaluate TF-ICON’s inversion, cross-domain composition, and design choices using benchmark comparisons, user studies, and ablations. The framework preserves foreground and background information while adapting compositions across visual domains.

  • Test benchmark: The 332-sample benchmark spans photorealism, pencil sketching, oil painting, and cartoon animation, with photorealistic reference images.Each sample includes a main image, reference image, user mask, and text prompt.
  • Qualitative comparisons: Across four visual domains, TF-ICON composes objects while maintaining their identities, whereas baselines show artifacts, prompt-driven deviations, or limited cross-domain adaptation.Paint by Example performs effectively in photorealism but struggles in other domains; SDEdit can preserve identifying features while changing the background.
  • Quantitative analysis: TF-ICON outperforms all evaluated baselines on photorealistic composition metrics, preserving background consistency, object correspondence, and text alignment.The evaluation uses LPIPS(BG), LPIPS(FG), CLIP(Image), and CLIP(Text).
  • User study: A user study with 50 participants and 40 ranking questions evaluates foreground preservation, background consistency, seamless composition, and text alignment across domains.Each question presents five options generated by distinct methods.
  • Ablation study: The complete algorithm outperforms ablated variants on all metrics except CLIP(Text), while cross-attention interactions benefit foreground and background preservation and domain switching.Background preservation at different noise levels affects both background and foreground, unlike preservation only at the final step.

6. Limitations and Future Work

TF-ICON is limited in generating object views that critically differ from the reference, and it inherits Stable Diffusion’s shortcomings and biases.

  • Scope boundary: TF-ICON may require restricted reference-image choices because self-attention maps constrain object layout and appearance, limiting substantially different views.Looser self-attention injection can produce different views but often compromises appearance preservation.
  • Future work: Stable Diffusion’s inherited shortcomings and biases may produce artifacts in certain scenarios.The paper suggests personalized concept learning or NeRF-related techniques as possible directions, with NeRF methods potentially requiring expensive training.

7. Conclusion

The paper concludes that high-order ODE solvers with the exceptional prompt improve real-image inversion and provide a foundation for training-free cross-domain composition. TF-ICON outperforms SOTA baselines for inversion and composition, while its attention design transfers contextual information into incorporated objects.

  • 7. Conclusion: The method is presented as a training-free framework for attention-based text-to-image diffusion models and is reported to outperform SOTA baselines for image inversion and composition.The paper positions image composition as potentially useful for content creators and downstream industry applications.
  • 7. Conclusion: High-order DPM-Solver++ produces better forward–backward trajectory alignment than DDIM, with no additional alignment improvement from increasing solver order.The comparison averages L1 and L2 distances over 5000 COCO2017 validation images using 50 inversion and decoding steps.
  • 7. Conclusion: High-order DPM solvers produce less blurred compositions than DDIM inversion under the same 20-step sampling process, while increasing solver order adds no noteworthy visual enhancement.The experiments use second-order DPM-Solver++ for TF-ICON.
  • 7. Conclusion: The exceptional prompt uses a common token value without positional embeddings so text-driven diffusion models can more accurately invert real images.Identical token embeddings create consistent directional movement across patch embeddings, unlike normal and null prompts.

A.4. Token Value Analysis

Token-value analysis finds that the exceptional prompt’s inversion performance is stable across sampled common token values. The appendix also documents preprocessing and gradual background preservation choices used by the composition pipeline.

  • Token Value Analysis: Across 100 uniformly sampled token values, MAE, SSIM, and LPIPS show negligible inversion variation on COCO images.The analysis uses Stable Diffusion with second-order DPM-Solver++ over 50 steps.
  • Token Value Analysis: For 150 COCO images, the low average standard deviations across reconstructions indicate that token selection does not significantly affect inversion performance.Means and standard deviations are computed over reconstructions from 100 tokens.
  • Reference preprocessing: The reference-image preprocessing segments the foreground, resizes and repositions it to the user mask, and zero-pads it to the main-image dimensions.A pretrained segmentation model extracts the object before geometric adjustment.
  • Implementation: TF-ICON synthesizes a single image in approximately 8 seconds on one A100 GPU, depending on the user-mask and reference-image sizes.The runtime is stated for the proposed training-free composition framework.
  • Background preservation: Gradual background preservation during denoising avoids artifacts that can occur when preservation is applied only at the final step.Blending noisy images at intermediate noise levels can be progressively corrected by subsequent diffusion steps.

B.4. Experimental Settings and Hyperparameters

The experiments evaluate image reconstruction and composition using Stable Diffusion, with specified datasets, prompts, solvers, and comparison baselines. The framework uses a 20-step diffusion procedure and an exceptional prompt with common token value 7788 in inversion settings.

  • Image Reconstruction: The reconstruction experiments use CelebA-HQ, COCO2017, and ImageNet, with MAE, LPIPS, and SSIM evaluating reconstruction quality.The CelebA-HQ evaluation uses the first 1500 images; COCO2017 uses all 5000 validation images, and ImageNet uses 3000 randomly sampled validation images.
  • Composition Procedure: Algorithm 1 takes normal and exceptional prompt embeddings, main and reference images, user and segmentation masks, and thresholds as inputs, returning a composition result.The procedure encodes the main and reference images, performs starting-point incorporation and image composition, then decodes the final latent.
  • Image Composition: The composition comparison uses Deep Image Blending, Blended Diffusion, Paint by Example, and SDEdit as baselines.Quantitative comparisons are conducted in photorealism, while other domains use user studies and qualitative comparisons.
  • Hyperparameters: The diffusion ODEs are solved with second-order DPM-Solver++ in 20 steps, using the exceptional prompt for the first two inversion ODEs and the normal prompt with CFG scale 2.5 for the last.The thresholds are τA = 0.4 and τB = 0; reconstruction settings use CFG scale 5 and exceptional-prompt token value 7788.

C. Ablation of Value Injection

The ablation examines whether injecting attention-map values improves composition and how attention-based design choices affect cross-domain transitions. Injecting only self-attention maps performs better than injecting both attention maps and corresponding values.

  • Value Injection: LPIPS(BG) = 0.10, LPIPS(FG) = 0.63, CLIP(Image) = 81.37, and CLIP(Text) = 27.68 when attention maps and value information are injected.These metrics are lower than those obtained by injecting only attention maps.
  • Value Injection: Injecting all information can make generation more rigid and hinder transitions across visual domains, especially in the sketchy domain.The stated explanation is direct replacement of all information from the guiding images.
  • Value Injection: Self-attention-map injection preserves semantic layouts while allowing values to derive from inherent composition features.This design is presented as an alternative to directly replacing all guiding-image information.
  • Self-Attention Visualization: Self-attention maps preserve semantic information, and unfolding their rows or columns reveals underlying image semantics.The visualization includes self-attention maps, averaged row and column unfoldings, and top-three PCA components.

G. Test Benchmark

The paper introduces a 332-sample benchmark for unified evaluation of cross-domain image-guided composition. It spans four main-image domains and uses photorealistic reference images with masks and prompts.

  • Benchmark Design: The benchmark contains 332 samples, each pairing a main background image, reference foreground image, user mask, and text prompt.The benchmark is designed for cross-domain image-guided composition evaluation.
  • Visual Domains: Main images cover photorealism, pencil sketching, oil painting, and cartoon animation, while all reference images are photorealistic.Reference images remain photorealistic because segmentation models are generally more effective in that domain.
  • Data Distribution: The benchmark includes 237 photorealism, 37 oil painting, 31 pencil sketching, and 27 cartoon animation samples.It is intended to be publicly available for evaluating cross-domain image-guided composition methods.

H. Additional Qualitative Results

Additional results provide qualitative reconstruction comparisons across COCO, ImageNet, and CelebA-HQ, alongside ablations and composition comparisons across multiple visual domains. The societal-impact discussion notes benefits and risks associated with accessible image composition.

  • Image Reconstruction: Figures 16–18 compare Stable Diffusion reconstruction outputs on COCO, ImageNet, and CelebA-HQ, respectively.The comparisons include classifier-free guidance, conditional, unconditional, and exceptional-prompt outputs.
  • Image Composition: Figure 19 presents additional ablation results, while Figures 20–25 show qualitative composition comparisons across sketching, oil painting, cartoon animation, and photorealism.The qualitative comparisons are made against state-of-the-art baselines.
  • Societal Impacts: Image-guided composition may empower people without professional artistic skills, but it can also be misused for harassment, fake news, bias amplification, and copyright infringement.The discussion specifically identifies risks from web-scraped training data and cross-artistic-domain composition.
  • Ablation Study: The framework combines a baseline with self-attention injection, background input, cross-attention injection, and exceptional-prompt variants in its ablation presentation.SA denotes self-attention and CA denotes cross-attention.
Loading 2307.12493v4…