PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Image Attention for Zero-Shot Appearance Transfer

Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, Daniel Cohen-Or

arXiv:2311.03335v1cs.CVcs.GR

TL;DR

Appearance transfer must preserve a target structure while applying another image’s appearance, even when the objects differ in shape, viewpoint, or category. The paper introduces Cross-Image Attention, which combines structure-image queries with appearance-image keys and values during diffusion denoising, alongside three quality-enhancement mechanisms. Results show effective zero-shot transfer across diverse object domains while preserving source structure and capturing target appearance.

  • Problem

    Appearance transfer requires matching semantically related regions across images while preserving the target structure despite differences in shape, viewpoint, and category.

  • Method

    The method inverts both images into a pretrained diffusion model and mixes structure-image queries with appearance-image keys and values through Cross-Image Attention during denoising.

  • Results

    The method performs zero-shot appearance transfer across diverse object domains and variations in shape, viewpoint, and instance count, while capturing target appearance and preserving source structure.

  • Takeaways & Limitations

    Semantic correspondences can support appearance transfer between objects with differing shape, size, or viewpoint without model training or user-provided conditioning.

  • Takeaways & Limitations

    The method depends on accurate semantic correspondences and successful latent inversion; mismatched semantics or poorly editable inversions can produce artifacts.

Abstract

from arXiv · show

Recent advancements in text-to-image generative models have demonstrated a remarkable ability to capture a deep semantic understanding of images. In this work, we leverage this semantic knowledge to transfer the visual appearance between objects that share similar semantics but may differ significantly in shape. To achieve this, we build upon the self-attention layers of these generative models and introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. Specifically, given a pair of images -- one depicting the target structure and the other specifying the desired appearance -- our cross-image attention combines the queries corresponding to the structure image with the keys and values of the appearance image. This operation, when applied during the denoising process, leverages the established semantic correspondences to generate an image combining the desired structure and appearance. In addition, to improve the output image quality, we harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Importantly, our approach is zero-shot, requiring no optimization or training. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint between the two input images.

1. Introduction

The paper introduces Cross-Image Attention for zero-shot appearance transfer, preserving one image’s structure while transferring another image’s appearance through semantic correspondences. Three additional mechanisms improve transfer quality, and experiments show effectiveness across varied objects, shapes, viewpoints, and instance counts.

  • Motivation: Appearance transfer requires associating semantically similar regions across images before transferring appearance without altering the target structure.The challenge includes objects from different categories, with varying shapes and viewpoints.
  • Core idea: Cross-Image Attention applies self-attention across images, forming associations between semantically similar regions despite distinct subjects, shapes, and structures.It builds on semantic information encoded in diffusion models’ cross- and self-attention mechanisms.
  • Core idea: The method combines structure-image queries with appearance-image keys and values during denoising to establish correspondences and transfer appearance to matching regions.The inputs are first inverted into the latent space of a pretrained text-to-image diffusion model.
  • Quality improvements: Three mechanisms improve transfer quality by focusing attention, strengthening cross-image guidance, and aligning appearance and output image statistics.These mechanisms respectively amplify attention-map variance, adapt classifier-free guidance, and use AdaIN to better preserve color.
  • Results: The approach supports zero-shot appearance transfer across diverse object domains, including image pairs with substantial variation in shape, viewpoint, and instance count.Quantitative comparisons indicate better capture of target appearance while preserving source structure.

2. Related Works

Prior appearance-transfer methods include trained generative models, single-exemplar optimization, and diffusion-based approaches using global appearance guidance. In contrast, this method uses semantic correspondences across images without training or per-image optimization, enabling diverse cross-domain transfers.

  • Appearance transfer: Appearance transfer differs from neural style transfer by transferring appearance between semantically related regions rather than applying a global artistic style.The paper frames appearance transfer as a specialized form of image-to-image translation.
  • Learned approaches: Earlier generative approaches trained GANs or autoencoders to separate and recombine structure and appearance representations.These methods used paired or unpaired collections, including architectures that separately encoded structure and appearance.
  • Learned approaches: Single-exemplar methods reduced supervision but could require training a dedicated generator for each image pair, taking dozens of minutes per input.One approach used pretrained DINO-ViT features to guide the transformation during training.
  • Diffusion approaches: Diffusion-based appearance-transfer methods avoided additional training but commonly relied on global appearance descriptors, limiting transfer to coarse results or same-category objects.These approaches guided denoising through losses applied to noisy latent codes.
  • This work: Unlike prior approaches, this method uses semantic correspondences without training or per-image optimization, requires one forward pass, and supports diverse cross-domain image pairs.Its self-attention modification injects keys and values between different images rather than between an image and its output edit.

3. Method

The method uses pretrained diffusion-model self-attention to establish semantic correspondences between a structure image and an appearance image, then transfers appearance during denoising. Three extensions improve focus, guidance, and image-statistics alignment while addressing artifacts from cross-image attention.

  • 3.2. Cross-Image Attention: Cross-Image Attention replaces output-image keys and values with appearance-image keys and values, enabling implicit appearance transfer between semantically similar objects.The structure image supplies queries, while the appearance image supplies keys and values.
  • 3.2. Cross-Image Attention: Queries from the structure image attend to semantically corresponding regions in the appearance image even when the objects differ substantially in shape.Giraffe queries attend to corresponding zebra regions, such as the head and leg.
  • 3.3. Appearance Transfer: The pipeline inverts both inputs into latent space, initializes the output latent from the structure latent, and applies cross-image attention during parallel denoising.The modified attention replaces standard self-attention in decoder layers at 32×32 and 64×64 resolutions.
  • 3.3. Appearance Transfer: The method also adds appearance guidance and AdaIN-based statistic alignment to strengthen cross-image influence and better preserve appearance-image colors.These mechanisms operate during denoising alongside the cross-image attention operation.
  • 3.3. Appearance Transfer: Attention-map contrasting focuses queries on fewer semantically similar appearance-image keys, mitigating inaccurate transfers caused by diffuse cross-image aggregation.Without contrasting, a query may attend broadly across many image regions, producing unwanted artifacts.

4. Experiments

Experiments evaluate cross-image attention for appearance transfer through qualitative, quantitative, user-study, comparison, and ablation results. The method transfers appearance across domains and substantial variations while balancing target-structure preservation with appearance fidelity.

  • Evaluation Scope: The evaluation covers zero-shot appearance transfer across multiple object domains, including cross-domain pairs and variations in shape, viewpoint, and instance count.The experiments compare the method with existing techniques and assess structure preservation, appearance fidelity, and overall image quality.
  • Qualitative Evaluation: The method transfers prominent semantic features between objects, including building domes and columns and car headlights and front grills.Qualitative results include challenging pairs with differing object shapes.
  • Comparison to Existing Methods: Against Swapping Autoencoder, the method preserves general structure while capturing more precise appearance characteristics and semantic details.Swapping Autoencoder often transfers only general color schemes and misses distinctive appearance features.
  • Comparison to Existing Methods: Compared with SpliceViT and DiffuseIT, the method handles differences in object number, shape, and viewpoint while operating zero-shot without external disentanglement models.SpliceViT can produce artifacts for objects with substantially different visual characteristics, whereas DiffuseIT also struggles with high-quality natural-image transfer.
  • Quantitative Comparison: Across six domains, the method achieves comparable structure preservation and appearance fidelity to alternative methods while balancing geometry with prominent target-appearance semantics.The evaluation uses mean IoU for structure preservation and Gram-matrix distances for appearance fidelity.
  • User Study: In the user study, the method consistently outperforms SpliceViT and DiffuseIT in appearance preservation and quality across four domains, with better or comparable structure preservation.Swapping Autoencoder performs best for buildings, likely reflecting its per-domain training; the method remains comparable in structure preservation there.
  • Ablation Study: Ablations show that attention-map contrasting reduces artifacts, AdaIN refines color distribution, and appearance guidance improves fine-level image details.The complete method adds these components incrementally to a baseline using cross-image attention alone.

5. Limitations and Discussion

The method is limited by the generative model’s ability to establish semantic correspondences and by the quality and editability of image inversion. These constraints make transfers between weakly related subjects more challenging and can introduce artifacts.

  • Semantic Correspondences: Transfers between subjects that do not share strong semantics can be challenging because the method relies on accurate correspondences between the input images.This limitation is illustrated for cross-domain subjects.
  • Inversion Dependence: Transfer quality depends on inversion reconstructing editable latent codes, and failures or random-seed sensitivity can introduce unwanted artifacts.The paper notes that output details such as a leg may vary across inversion seeds.

6. Conclusions

The method enables zero-shot semantic appearance transfer between natural-image objects, even when their shape, size, or viewpoint differs. Cross-Image Attention establishes correspondences between images, while denoising-time extensions improve realism and preserve structure.

  • 6. Conclusions: Cross-Image Attention enables semantic-based appearance transfer between objects in natural images without model training or user-provided conditioning.It mixes queries, keys, and values from two images to establish implicit semantic correspondences.
  • 6. Conclusions: The method supports appearance transfer when objects differ in shape, size, or viewpoint.The conclusion explicitly describes transfer across these variations.
  • 6. Conclusions: Three extensions reduce the domain gap from cross-image mixing by manipulating noisy latent codes and denoising-model representations.These extensions support gradual appearance transfer and more realistic, high-quality images.
  • 6. Conclusions: The work motivates further study of generative-model internal semantics for diverse zero-shot tasks with minimal user intervention.This is presented as a direction enabled by deeper understanding of internal representations.

A. Additional Details

The implementation uses Stable Diffusion with scheduled cross-image-attention injection, appearance-guidance mechanisms, and optional structure injection. Additional experiments include user-study comparisons and appearance-transfer result sets across domains.

  • Implementation Details: The method operates on Stable Diffusion v1.5 using DDPM inversion and 100-step DDIM denoising.Input images are inverted with the prompt “A photo of a domain” and default DDPM-inversion hyperparameters.
  • Implementation Details: Cross-image attention replaces decoder self-attention at 32×32 and 64×64 resolutions during selected denoising intervals.Keys and values are injected between timesteps 10–70 at 32×32 and 10–90 at 64×64.
  • Implementation Details: Appearance transfer uses contrast amplification, guidance scaling, and AdaIN to focus correspondences, strengthen transfer, and align image statistics.The implementation sets β = 1.67, α = 3.5, and applies AdaIN between timesteps 20 and 100.
  • Structure Injection: Structure injection replaces output keys and values with structure-derived keys and values at selected timesteps to preserve fine structural details.For categories such as animal ears, injection every five timesteps balances appearance transfer with structural fidelity.
  • User Study: The user study contains 16 structure–appearance pairs across buildings, animals, cars, cakes, and birds.Eight pairs come from buildings, while two pairs come from each of the four other domains; Swapping Autoencoder comparisons are limited to buildings.
  • Additional Results: Additional figures show transfer results for multiple appearance images paired with one structure image and enlarged outputs.Figure 12 uses one structure image with three appearance images; Figures 13–15 provide further or enlarged results.
Loading 2311.03335v1…