PaperlayerView exact revision on arXiv

Source-linked AI summary

Splicing ViT Features for Semantic Appearance Transfer

Narek Tumanyan, Omer Bar-Tal, Shai Bagon, Tali Dekel

arXiv:2201.00424v1cs.CV

TL;DR

Semantic appearance transfer must map visual appearance between semantically related regions of natural images while preserving source structure, even when only one image pair is available. Splice uses fixed DINO-ViT features, with separate appearance and structure representations, to train a generator that combines them without adversarial training or extra semantic inputs. It produces high-quality transfers across diverse in-the-wild pairs and substantial variations in objects, pose, and appearance, but depends on DINO-ViT capturing the relevant semantic associations.

  • Problem

    Semantic appearance transfer requires associating related regions between natural images while preserving source structure, often with only a single image pair available.

  • Method

    Splice trains a generator with fixed DINO-ViT features, representing appearance by the global [CLS] token and structure by key self-similarity.

  • Results

    The method produces semantically meaningful appearance transfer across diverse natural image pairs with substantial variations in object count, pose, and appearance, including high-resolution HD results.

  • Takeaways & Limitations

    The framework enables unguided semantic appearance transfer from a single image pair without segmentation, correspondences, or adversarial training.

  • Takeaways & Limitations

    Performance depends on DINO-ViT’s internal representation capturing semantic associations across the two images; mismatches can produce failed or incorrect mappings.

Abstract

from arXiv · show

We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are "painted" with the visual appearance of their semantically related objects in a target appearance image. Our method works by training a generator given only a single structure/appearance image pair as input. To integrate semantic information into our framework - a pivotal component in tackling this task - our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model which serves as an external semantic prior. Specifically, we derive novel representations of structure and appearance extracted from deep ViT features, untwisting them from the learned self-attention modules. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Our framework, which we term "Splice", does not involve adversarial training, nor does it require any additional input information such as semantic segmentation or correspondences, and can generate high-resolution results, e.g., work in HD. We demonstrate high quality results on a variety of in-the-wild image pairs, under significant variations in the number of objects, their pose and appearance.

1. Introduction

The paper targets semantic appearance transfer between natural images while preserving source structure, using only one image pair. It introduces ViT-based representations and a generator objective to combine structure and appearance without adversarial training.

  • Semantic appearance transfer must match related regions across images, realistically repaint the source while preserving its structure.The task remains challenging under pose, appearance, and shape differences, making semantic information necessary.
  • Unlike domain translation, the method generates the appearance of a particular target image from a single structure/appearance pair.This setting addresses scenes and objects lacking convenient image collections.
  • DINO-ViT features provide semantic information at fine spatial granularity, while the [CLS] token captures global appearance including object parts.Feature inversion shows that images can be reconstructed from these features despite their semantic content.
  • The method represents appearance with the deepest-layer [CLS] token and structure with self-similarity of keys from the last attention module.These representations are extracted by untwisting information from learned self-attention modules.
  • The generator splices desired structure and appearance in ViT feature space using a single image pair, without segmentation, correspondences, or adversarial training.The framework also supports high-resolution image generation in HD.
  • Experiments cover natural image pairs with substantial variation in object count, pose, and appearance.

2. Related Work

The paper addresses unguided semantic appearance transfer between arbitrary in-the-wild natural image pairs, contrasting with domain translation, style transfer, and methods requiring extra semantic inputs. It uses fixed DINO-ViT features to inject semantic information into single-pair generation.

  • The problem is semantic visual appearance transfer between two in-the-wild natural images without user guidance.The paper states that no existing method specifically addresses this challenge.
  • Domain translation typically learns source-to-target mappings with GANs trained on paired or unpaired image collections.The paper instead targets the appearance of a particular target image while preserving source structure.
  • Single-example image-to-image methods use low-level information and lack semantic understanding, whereas this method adds a pretrained ViT semantic prior.
  • Neural Style Transfer and related methods generally transfer artistic style globally, while this work maps appearance between semantically related regions in natural images.
  • Semantic style transfer methods may be limited to color transformation, require annotations or correspondences, or operate in controlled domains.This paper aims to handle arbitrary in-the-wild input pairs.
  • DINO-ViT is a self-supervised Vision Transformer whose learned representations support tasks including image retrieval, segmentation, and semantic correspondence.Deep features capture semantic object parts and representations shared across related object classes.
  • The pipeline trains a generator so fixed DINO-ViT features of its output match source structure and target appearance representations.The figure specifies appearance matching through [CLS] features and structure matching through key self-similarity.

3. Method

Splice uses fixed DINO-ViT features to represent appearance globally with the deepest [CLS] token and structure through self-similarity of deep spatial keys. A generator is trained on one structure/appearance pair to produce an output matching both representations.

  • Given source structure image Is and target appearance image It, the generator produces Io with source structure and target-object appearance.
  • Appearance representation: The deepest [CLS] token represents appearance because it captures global information while remaining spatially flexible.Across layers, [CLS] features progress from local texture toward higher-level object parts; multiple inversions retain global information despite structural variation.
  • Structure representation: Structure is represented by self-similarity among deepest-layer spatial keys, preserving layout, shape, and semantic object components while discarding appearance.PCA visualization shows self-similarity captures object structure and semantic parts, such as distinctive polar-bear legs and body despite shared texture.
  • Feature analysis: Deepest-layer keys remain invertible enough to reconstruct the original image, while their self-similarity emphasizes semantic structure.
  • Generator training: The splice objective combines appearance, structure, and identity losses, with α = 0.1 and β = 0.1 used for all experiments.Appearance matches the target [CLS] token; structure matches source key self-similarity; identity regularizes the generator using the target keys.
  • Generator training: Training uses augmented examples derived from a single input pair, requiring the generator to learn a mapping rather than optimize one test instance.

4. Results

Splice transfers appearance across semantically related regions in diverse image pairs while preserving source structure, outperforming several comparison methods qualitatively and in human preference evaluation.

  • The Wild-Pairs dataset contains 25 high-resolution image pairs spanning animals, fruits, and other objects with changes in object count, pose, and appearance.
  • Splice transfers appearance across objects and within objects under substantial variations in pose, object count, and appearance.
  • The method demonstrates semantic appearance transfer across diverse image pairs, unlike GAN-based methods restricted to their training datasets.
  • Comparisons to Prior Work: STROTSS and WCT2 preserve source structure but often produce color bleeding or global color artifacts, showing that color transfer alone is insufficient.
  • Comparisons to Prior Work: Compared with SA, Splice better preserves fidelity to both inputs while training only on the single input pair and without a large domain-specific image collection.
  • Comparisons to Prior Work: SinCUT handles mostly low-level translation but fails when higher-level reasoning is required, whereas Splice transfers appearance across semantic regions without adversarial training.
  • Ablation: Removing individual loss terms causes failures in target-appearance mapping, source-structure preservation, or fine-detail maintenance.
  • Human Perceptual Evaluation: The AMT evaluation used 65 images and collected 7000 judgments comparing Splice with existing baselines.

A B A-to-B B-to-A

The method is evaluated on semantic appearance transfer in challenging image pairs, including cross-domain and mismatched-category cases. It outperforms baselines while preserving semantic layout, though failures arise when DINO-ViT misses the relevant correspondence.

  • Evaluation settings: Appearance-transfer evaluations include semantically related objects across domains and pairs with unrelated object categories.Figure 10 distinguishes realistic versus non-realistic inputs and semantically related versus unrelated object categories.
  • Perceptual evaluation: The method outperforms GAN-based and style-transfer baselines in AMT perceptual evaluations.Table 1 compares the method with SA, STROTSS, and WCT2 across datasets.
  • Perceptual evaluation: The method outperforms all baselines across image collections, especially on the challenging In-the-Wild-Pairs.It remains competitive with SA despite SA being trained on 500K mountain images.
  • Semantic layout preservation: The method better preserves scene layout than SA and STROTSS and is closest to WCT2, whose color-only changes yield the highest IoU.Layout preservation is measured by mean IoU between output and input-structure segmentation maps.
  • Ablation and limitations: Removing the appearance loss prevents target-appearance mapping and produces only slight color changes, while DINO-ViT correspondence failures cause incorrect transfers.The limitation appears when objects are out of distribution or semantically unrelated in the ViT representation.

5. Conclusions

The paper introduces semantic appearance transfer between natural images without user guidance. It uses DINO-ViT representations to disentangle and splice structure and appearance, producing high-quality results across challenging image pairs without adversarial training.

  • Problem: The paper tackles semantic appearance transfer between related objects in two in-the-wild natural images without user guidance.This defines the paper’s new problem setting within style and appearance transfer.
  • Approach: DINO-ViT provides an external semantic prior for disentangling structure and appearance and splicing them in ViT-feature space.The generation process combines the desired structure and appearance representations.
  • Results: The method produces high-quality results across domains, poses, and varying numbers of objects without adversarial training.The conclusion reports applicability to diverse and challenging input pairs.

A.1. Generator Network Architecture

The generator is a U-Net with a five-layer encoder and symmetric decoder. Skip-connected feature transformations are concatenated into decoder levels before the final output convolution.

  • Network design: The generator Gθ uses a U-Net architecture with a five-layer encoder and a symmetrical decoder.The encoder and decoder use mirrored channel dimensions.
  • Layer blocks: Each layer uses 3×3 convolutions followed by BatchNorm and LeakyReLU activation.These operations define the layer block throughout the encoder and decoder.
  • Channel dimensions: Encoder channel dimensions are [3 →16 →32 →64 →128 →128], with the decoder following the reversed order.The architecture expands and then contracts feature channels across corresponding levels.
  • Skip connections: Additional 1×1 convolutions transform encoder features, which are concatenated with the corresponding decoder levels.A final 1×1 convolution produces the network output.

A.2. ViT Feature Extractor Architecture

The feature extractor is a fixed, self-supervised DINO-ViT model used to provide the paper’s ViT features. The implementation uses ViT-B/8 with twelve layers and 8×8 patches.

  • Feature extractor: The method leverages a pre-trained ViT model trained through self-supervision as a feature extractor.The model is used as a fixed external feature source.
  • Model configuration: The selected model is the twelve-layer ViT-B/8 configuration.This identifies the specific DINO-ViT architecture used in the experiments.
  • Patch configuration: ViT-B/8 processes images using an 8×8 patch configuration and was downloaded from the official implementation.The passage specifies the patch setup and model source.

A.3. Training Details

The framework is optimized with a full objective using specified relative weights, Adam, a constant learning rate, and batches of augmented image views.

  • A.3. Training Details: The full objective is optimized with relative weights α = 0.1 and β = 0.1 using Adam at a constant learning rate of 2 · 10−3.Each batch contains augmented views of the source structure and target appearance images; unaugmented images are added every 75 iterations.

A.4. Data Augmentations

Data augmentations create multiple internal examples from the source structure and target appearance images through cropping, flipping, color jittering, and Gaussian blurring.

  • A.4. Data Augmentations: At each training step, the input image pair is augmented to create multiple internal examples.The listed transformations are applied separately to source and target images.
  • A.4. Data Augmentations: Source structure images receive random crops covering 95%–100% of image height and random horizontal flips with probability 0.5.These augmentations are specified for the source image Is.
  • A.4. Data Augmentations: Both source and target images use random horizontal flipping with probability 0.5.
  • A.4. Data Augmentations: Source structure images receive color jittering with probability 0.5 across brightness, contrast, saturation, and hue.
  • A.4. Data Augmentations: Source structure images receive Gaussian blurring with probability 0.5 using a 3x3 filter and σ uniformly sampled from 0.1–2.0.
  • A.4. Data Augmentations: Target appearance images receive random crops covering 95%–100% of image height.The target-image augmentation passage explicitly specifies cropping.
Loading 2201.00424v1…