PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning for Unpaired Image-to-Image Translation

Taesung Park, Alexei A. Efros, Richard Zhang, Jun-Yan Zhu

arXiv:2007.15651v3cs.CVcs.LG

TL;DR

Unpaired image translation must preserve input content while changing domain-specific appearance, yet cycle-consistency imposes a restrictive bijective assumption. The paper uses multilayer patchwise contrastive learning to maximize mutual information between corresponding input and output patches, with internal negatives. Experiments report a faster, lighter model that outperforms prior one-sided and state-of-the-art alternatives, and the method can train with single-image domains.

  • Problem

    Unpaired translation needs correspondence-preserving content changes without assuming a bijection between domains, because cycle-consistency can be too restrictive.

  • Method

    The method maximizes mutual information between corresponding input and output patches with a multilayer, patchwise InfoNCE framework that uses internal input-image negatives.

  • Results

    The faster, lighter model outperforms prior one-sided translation methods and state-of-the-art models using auxiliary networks and multiple loss functions.

  • Takeaways & Limitations

    The framework enables one-sided unpaired translation and can be trained when each domain contains only a single image.

  • Takeaways & Limitations

    The paper identifies a scope boundary in cycle-consistency: perfect reconstruction is difficult when one domain contains additional information compared with the other.

Abstract

from arXiv · show

In image-to-image translation, each patch in the output should reflect the content of the corresponding patch in the input, independent of domain. We propose a straightforward method for doing so -- maximizing mutual information between the two, using a framework based on contrastive learning. The method encourages two elements (corresponding patches) to map to a similar point in a learned feature space, relative to other elements (other patches) in the dataset, referred to as negatives. We explore several critical design choices for making contrastive learning effective in the image synthesis setting. Notably, we use a multilayer, patch-based approach, rather than operate on entire images. Furthermore, we draw negatives from within the input image itself, rather than from the rest of the dataset. We demonstrate that our framework enables one-sided translation in the unpaired image-to-image translation setting, while improving quality and reducing training time. In addition, our method can even be extended to the training setting where each "domain" is only a single image.

1 Introduction

The paper replaces restrictive cycle-consistency with patchwise mutual-information maximization to preserve input content while changing domain-specific appearance. Its contrastive framework uses multilayer patches and internal negatives, enabling effective one-sided unpaired translation with a faster, lighter model.

  • Motivation: Cycle-consistency can be too restrictive because it assumes a bijection between domains, whereas the proposed method preserves content without preserving appearance.The motivating task is translating a horse into a zebra while retaining the horse's structure.
  • Method: The framework maximizes mutual information between corresponding input and output patches using a multilayer, patchwise contrastive loss.Figure 1 describes the generated output patch as closer to its corresponding input patch than to random patches.
  • Method: InfoNCE associates corresponding input and output patches while disassociating noncorresponding patches, encouraging domain-invariant features such as object parts and shapes.The learned representation is intended to ignore domain-specific textures.
  • Design choices: Internal negatives drawn from within the input image better preserve input content than negatives drawn externally from other dataset images.The method avoids requiring a memory bank or specialized architecture.
  • Results: Extensive experiments report that the faster, lighter model outperforms prior one-sided translation methods and state-of-the-art models using auxiliary networks and multiple losses.The framework is presented as enabling one-sided translation in the unpaired setting.

2 Related Work

Prior unpaired translation methods commonly use cycle-consistency, but its bijective assumption can be overly restrictive. This paper instead applies mutual-information-based contrastive learning to conditional image synthesis and studies design choices specific to that setting.

  • Image translation and cycle-consistency: Cycle-consistency is widely used to enforce correspondence in unpaired translation, but its bijective assumption can be too restrictive when domains contain unequal information.Perfect reconstruction is especially difficult when one domain has additional information.
  • Relationship preservation: Relationship-preservation methods encourage input similarities, distances, vector relations, or geometric structure to remain analogous in translated outputs.These approaches use varied constraints, including perceptual similarity, predefined distances, triplet losses, and equivariance.
  • Perceptual similarity: Per-pixel reconstruction metrics such as ℓ1 do not reflect human perceptual preferences and can produce blurry results, motivating learned perceptual representations.Deep networks such as VGG have been repurposed as perceptual losses.
  • Contrastive representation learning: The method uses data-derived negatives so its cross-domain similarity function can adapt to the particular input and output domains without a predefined similarity function.This contrasts with fixed similarity constraints used in some relationship-preservation approaches.
  • Contrastive representation learning: Mutual-information methods use noise contrastive estimation to bring associated signals together in an embedding while separating them from other samples.Associated signals may include images, representations, neighboring patches, or transformed views.
  • Contrastive representation learning: The paper is the first described use of InfoNCE for conditional image synthesis and studies negative sampling, multilayer patchwise representations, and other synthesis-specific factors.The design choices include the number and source of negatives, hyperparameters, and augmentations.

3 Methods

CUT translates in one direction by combining adversarial realism with multilayer, patchwise contrastive correspondence between input and output images. It uses internal input patches as negatives to preserve content while changing domain-specific appearance.

  • Patchwise contrastive learning: Contrastive learning treats each output patch as a query, its same-location input patch as the positive, and other input-image patches as negatives.The negatives are sampled from different locations within the same input image.
  • Multilayer, patchwise contrastive learning: The method uses multiple encoder layers and spatial patches, with deeper layers representing larger input regions and MLPs projecting features into a shared embedding space.This multilayer design aligns both global and local content correspondence.
  • Negative sampling: Internal negatives outperform external patches because internal statistics avoid modeling unnecessary intra-class variation.The authors hypothesize this helps the encoder capture domain-invariant concepts while the decoder synthesizes domain-specific features.
  • Final objective: The generator maps input images to the target domain with an adversarial loss while PatchNCE matches corresponding input-output patches.The objective may also apply PatchNCE to target-domain images as an identity-preserving regularizer.
  • Model variants: CUT uses identity regularization, whereas FastCUT removes it and increases the input-domain PatchNCE weight to λX = 10.FastCUT is presented as a faster, lighter version of the one-sided approach.
  • Motivation and formulation: The approach maximizes mutual information between high-dimensional input and output image spaces through contrastive learning rather than relying on cycle-consistency and inverse translation networks.The one-direction mapping avoids inverse auxiliary generators and discriminators.

4 Experiments

Experiments evaluate image quality, correspondence, efficiency, and design choices across unpaired translation datasets. CUT and FastCUT outperform or compete strongly with baselines while reducing computational requirements, with performance depending on PatchNCE configuration.

  • Evaluation: CUT and FastCUT outperform baselines on image-quality metrics across datasets, while Cityscapes segmentation scores indicate improved discovered correspondence.The evaluation includes FID for image quality and semantic segmentation metrics for correspondence.
  • Datasets and protocol: The experiments cover Cat→Dog, Horse→Zebra, and Cityscapes at 256×256 resolution, using established image-quality and correspondence protocols.Cityscapes additionally provides corresponding labels for semantic evaluation.
  • Ablations: Removing identity regularization improves Horse→Zebra results but decreases Cityscapes performance, where training can collapse before recovering.The regularizer stabilizes training on Cityscapes but is unnecessary for FastCUT in some settings.
  • Ablations: Reducing the number of encoder layers or sampling external patches hurts performance, whereas fewer negatives can still perform strongly.The default configuration uses internal negatives and multilayer PatchNCE.

Appendix C.2 for more evaluation details.

The evaluation studies contrastive design choices, efficiency, learned correspondences, and extensions to single-image translation. Results show that internal negatives, multilayer supervision, identity regularization, and the proposed contrastive objective support quality, stability, and practical one-sided translation.

  • Speed and memory: 40% faster and 31% more memory-efficient than CycleGAN, the identity-loss method preserves competitive translation quality with lower training cost.FastCUT is 63% faster and 53% lighter while achieving superior metrics to CycleGAN.
  • Contrastive design choices: Internal negatives outperform external negatives, whose use causes quality loss and repeated textures in Cityscapes.Sampling negatives from the same image provides a stronger content-preservation signal than adding or replacing them with negatives from other images.
  • Contrastive design choices: Using only the encoder’s last layer drastically reduces performance, indicating that multilayer features provide important dense supervision for image synthesis.The method samples every four layers from pixels through the 16th layer, rather than using a single whole-image representation.
  • Contrastive design choices: The identity-loss regularizer makes training more stable, although removing it can improve final FID on some tasks and can cause collapse during training.Without the regularizer, performance decreases on Cityscapes despite improving on horse→zebra.
  • Learned similarity: Encoder visualizations suggest that corresponding patches from different domains occupy similar embedding locations, supporting implicitly learned cross-domain correspondences.The analysis compares patch similarities with exp(v · v−/τ) and visualizes the top three PCA components before and after translation.
  • Conclusion: The method maximizes mutual information by bringing corresponding input-output patches together and pushing away noncorresponding patches, enabling one-sided and single-image unpaired translation.The approach avoids cycle-consistency and does not rely on a predefined similarity function such as ℓ1 or perceptual loss.

A.2 Additional datasets

Additional experiments evaluate the method on Apple→Orange, Summer→Winter, and GTA→Cityscapes, where it provides better or comparable results to CycleGAN across varied datasets.

  • A.2 Additional datasets: The method is evaluated on Apple→Orange, Summer→Winter, and GTA→Cityscapes in comparison with CycleGAN.The datasets span natural objects, landscapes, and street scenes.
  • A.2 Additional datasets: It provides better or comparable results to CycleGAN, demonstrating flexibility across a variety of datasets.
  • A.2 Additional datasets: Apple→Orange uses 996 apple and 1,020 orange images from ImageNet.
  • A.2 Additional datasets: Summer→Winter uses 1,273 summer and 854 winter Yosemite images scraped using the FlickAPI.
  • A.2 Additional datasets: GTA→Cityscapes uses 24,966 GTA images and 19,998 Cityscapes street-scene images.

Appendix B Additional Single Image Translation Results

Additional single-image translation experiments compare the method with style-transfer and CycleGAN baselines at high resolution. The results retain painting structure while reproducing reference-photo texture and appear more photorealistic than several baselines.

  • Appendix B Additional Single Image Translation Results: During training, images are randomly scaled to widths between 384 and 1024, with 16 randomly sampled 128 × 128 crops per iteration.
  • Appendix B Additional Single Image Translation Results: At 1–2 Megapixels, the results look more photorealistic than CycleGAN, Gatys et al., and WCT2.
  • Appendix B Additional Single Image Translation Results: The quality of the results is on par with STROTSS.
  • Appendix B Additional Single Image Translation Results: Training the painting-to-photo translation requires only a single image from each domain.
  • Appendix B Additional Single Image Translation Results: The comparisons include Gatys et al., WCT2, STROTSS, and a modified patch-based CycleGAN.
  • Appendix B Additional Single Image Translation Results: The method reproduces reference-photo texture while retaining the structure of the input paintings.

C.1 Training details

The training details specify a controlled CycleGAN-based setup for isolating the patch-based contrastive loss and a multi-layer feature extraction procedure for computing it.

  • C.1 Training details: The contrastive-loss study matches CycleGAN’s architecture and hyperparameters except for the loss function.The setup uses a ResNet generator, PatchGAN discriminator, Least Square GAN loss, batch size 1, and Adam with learning rate 0.002.
  • C.1 Training details: CUT is trained for up to 400 epochs, while FastCUT is trained for up to 200 epochs.
  • C.1 Training details: PatchNCE extracts features from five layers with receptive fields ranging from 1×1 to 99×99.
  • C.1 Training details: At each layer, 256 random locations are sampled and passed through a 2-layer MLP to obtain 256-dimensional final features.

C.2 Evaluation details

The evaluation details define how visual quality, segmentation performance, and PatchNCE are measured and implemented, including FID preprocessing and feature-sampling operations.

  • C.2 Evaluation details: The evaluation protocol includes FID for visual quality and semantic segmentation metrics for Cityscapes.
  • C.2 Evaluation details: FID resizes images to 299-by-299 and uses the last average-pooling activations of pretrained Inception V3.
  • C.2 Evaluation details: Cityscapes segmentation evaluates 500 validation outputs after resizing them to 256x128 and passing them through a trained DRN-D-22 network.
  • C.2 Evaluation details: PatchNCE takes sampled input and generated-image features with temperature tau and returns a contrastive loss.
  • C.2 Evaluation details: Positive logits are computed from corresponding feature products, while negative logits use pairwise sampled-feature products with diagonal entries removed.
  • C.2 Evaluation details: The loss applies cross-entropy to flattened logits with zero-valued targets.

C.4 Distribution matching

The horse→zebra training set is biased toward larger zebras, and CUT can match this distribution by enlarging horses, whereas FastCUT behaves more conservatively. This flexibility can introduce unwanted background changes but can also support dramatic geometric transformations.

  • 37% vs. 18%: zebras occupy about twice as many pixels as horses on average in the horse→zebra training set.The estimate uses an off-the-shelf DeepLab semantic segmentation model.
  • CUT can enlarge horses to better match the training statistics than CycleGAN.The method's distribution-matching capacity follows from the imbalance in object size between the domains.
  • FastCUT behaves more conservatively and more similarly to CycleGAN when trained with λX = 10 and flip-equivariance augmentation.These design choices increase the PatchNCE weight and add horizontal-flip consistency.
  • The strong distribution-matching capacity can introduce undesired zebra patterns on the background in horse→zebra translation.
  • The same capacity can enable dramatic geometric changes for applications such as Cat→Dog.

C.5 Additional Ablation studies

Additional ablations show that encoder and projection-network weight sharing improves FID and reduces parameters, while allowing PatchNCE to update the decoder is crucial for avoiding severe artifacts.

  • 50.5 FID: removing weight sharing for the encoder and MLP projection network worsens performance relative to the original model's 46.6 FID.The separate input and generated-image networks also increase the model's parameter count.
  • 444.2 FID: preventing PatchNCE from updating the decoder produces severe artifacts compared with the original model's 46.6 FID.The decoder then receives updates only through the adversarial loss.
  • PatchNCE jointly helps learn a better encoder and decoder together with the GAN loss.Artifacts in generated images make input-output patch correspondences difficult for the encoder to identify, increasing PatchNCE loss.
Loading 2007.15651v3…