PaperlayerView exact revision on arXiv

Source-linked AI summary

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros

arXiv:1703.10593v7cs.CV

TL;DR

Unpaired image-to-image translation lacks aligned input-output examples, limiting standard supervised approaches. CycleGAN learns bidirectional mappings with adversarial and cycle-consistency losses, outperforming prior approaches across applications including style, object, season, and photo translation.

  • Problem

    Image-to-image translation often lacks paired examples linking source images to corresponding target images.

  • Method

    CycleGAN combines adversarial distribution matching with forward and backward cycle-consistency losses between mappings G: X → Y and F: Y → X.

  • Results

    The method outperforms prior approaches across style transfer, object transfiguration, season transfer, and photo enhancement applications.

  • Takeaways & Limitations

    Completely unpaired data can support image translation across diverse applications.

  • Takeaways & Limitations

    The method often struggles with transformations requiring geometric changes, such as dog-to-cat transfiguration.

Abstract

from arXiv · show

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

1. Introduction

The paper addresses image-to-image translation without paired examples by learning mappings between image domains from set-level supervision. It combines adversarial distribution matching with cycle consistency and demonstrates the method across several applications, outperforming prior baselines.

  • Problem setting: Unpaired training data provides separate source and target image sets without information about which source image matches which target image.This contrasts with paired training data, where corresponding examples are available.
  • Problem motivation: Paired training data is difficult and expensive to obtain, especially for small semantic-segmentation datasets and artist-authored stylization outputs.The paper motivates learning domain translation without paired input-output examples.
  • Adversarial translation: The method learns G: X →Y so generated outputs are indistinguishable from target-domain images under an adversarial loss.This objective aims to match the generated output distribution with the empirical target distribution.
  • Cycle consistency: Because distribution matching alone does not ensure correct individual input-output relationships, the method adds an inverse mapping F: Y →X and cycle consistency.The translators should act as inverses, translating an image forward and back to recover the original.
  • Applications and evaluation: Applications include collection style transfer, object transfiguration, season transfer, and photo enhancement, with performance exceeding compared prior approaches.The baselines use hand-defined style-content factorizations or shared embedding functions.

2. Related work

Prior work spans adversarial image generation, paired image-to-image translation, unpaired cross-domain learning, cycle consistency, and neural style transfer. The paper distinguishes its general-purpose formulation from these approaches by avoiding task-specific similarity functions and shared low-dimensional embeddings.

  • Generative Adversarial Networks: GANs use adversarial loss to make generated images indistinguishable from images in the target domain, supporting image generation, editing, and representation learning.Conditional GANs extend this idea to applications including text-to-image, inpainting, future prediction, video, and 3D data.
  • Image-to-Image Translation: Image-to-image translation evolved from Image Analogies to CNN-based parametric mappings learned from input-output examples, including the pix2pix framework.The cited approaches use paired training examples to learn a translation function.
  • Unpaired Image-to-Image Translation: Unpaired approaches relate domains X and Y through Bayesian patch-based models, weight sharing, common representations, and other domain-relationship strategies.The passage identifies Rosales et al., CoGAN, and cross-modal scene networks as examples of prior unpaired methods.
  • Unpaired Image-to-Image Translation: Unlike prior approaches, the formulation requires neither a task-specific predefined similarity function nor shared low-dimensional embedding spaces, making it general-purpose for vision and graphics tasks.The paper states that it compares directly against several prior and contemporary approaches in Section 5.1.
  • Cycle Consistency: Cycle consistency regularizes structured mappings through forward-backward consistency, back translation and reconciliation, and related higher-order consistency ideas.The paper places this strategy in a longer history spanning visual tracking and language translation.
  • Neural Style Transfer: Neural style transfer combines one image’s content with another’s style using Gram-matrix statistics, whereas this work learns mappings between image collections and targets tasks such as painting-to-photo and object transfiguration.The passage contrasts collection-level translation with transfer between two specific images.

3. Formulation

The formulation learns two inverse mappings between unpaired domains, using adversarial losses to match translated and real-image distributions. Cycle consistency constrains these mappings to reconstruct inputs, and the full objective combines both requirements.

  • 3. Formulation: The model learns mappings G: X → Y and F: Y → X from samples in two domains, with discriminators D_Y and D_X distinguishing translated from real images.D_Y compares G(x) with y, while D_X compares F(y) with x.
  • 3.1. Adversarial Loss: Adversarial losses train G to produce images resembling domain Y and F to produce images resembling domain X.Each mapping minimizes its adversarial objective against a discriminator maximizing it.
  • 3.2. Cycle Consistency Loss: Adversarial losses alone permit mappings that match target distributions while arbitrarily permuting input images, so they do not determine meaningful correspondences.This ambiguity arises because sufficiently capable networks can map the same inputs to random target-domain permutations.
  • 3.2. Cycle Consistency Loss: Cycle consistency requires F(G(x)) ≈ x and G(F(y)) ≈ y, enforcing forward and backward reconstruction across the two translation directions.The loss incentivizes each image to return to its original domain after a round trip.
  • 3.2. Cycle Consistency Loss: Replacing the L1 cycle-consistency norm with an adversarial loss in preliminary experiments did not improve performance.The comparison involved adversarial losses between reconstructed and original images in both directions.
  • 3.2. Cycle Consistency Loss: Reconstructed images F(G(x)) closely match their input images x, demonstrating the behavior induced by cycle consistency.The paper reports this behavior across experiments illustrated in Figure 4.
  • 3.3. Full Objective: The full objective combines adversarial losses for both mappings with cycle consistency, weighted by λ to control the objectives’ relative importance.Optimization jointly trains G, F, D_X, and D_Y.
  • 3.3. Full Objective: The model can be viewed as two translation-structured autoencoders, F ◦ G: X → X and G ◦ F: Y → Y, with adversarially trained intermediate translations.Ablation experiments report that both adversarial and cycle objectives are critical, while a single cycle is insufficient.

4. Implementation

The implementation uses residual-block generators with instance normalization and 70 × 70 PatchGAN discriminators. Training is stabilized with least-squares adversarial loss and buffered generated images, alongside fixed optimization and learning-rate schedules.

  • Network Architecture: Generators use three convolutions, residual blocks, two fractionally-strided convolutions with stride 2, and an RGB-mapping convolution.They use 6 residual blocks for 128 × 128 images and 9 blocks for 256×256 and higher-resolution images, with instance normalization.
  • Network Architecture: Discriminators are 70 × 70 PatchGANs that classify overlapping image patches as real or fake.This patch-level design has fewer parameters than a full-image discriminator and can operate on arbitrarily sized images.
  • Training details: Least-squares loss replaces the negative log likelihood objective to stabilize training and generate higher-quality results.The generator minimizes Ex∼pdata(x)[(D(G(x)) −1)2], while the discriminator minimizes Ey∼pdata(y)[(D(y) −1)2] + Ex∼pdata(x)[D(G(x))2].
  • Training details: A buffer of 50 previously generated images updates the discriminators instead of using images from only the latest generators, reducing model oscillation.This follows Shrivastava et al.’s strategy.
  • Training details: All experiments use λ = 10, Adam with batch size 1, learning rate 0.0002, and training from scratch.The learning rate remains constant for 100 epochs, then linearly decays to zero over the next 100 epochs.

5. Results

The Results section evaluates CycleGAN against recent unpaired translation methods, analyzes the contributions of adversarial and cycle consistency losses, and demonstrates the algorithm across applications without paired data.

  • Comparisons: CycleGAN is compared with recent unpaired image-to-image translation methods on paired datasets using available ground-truth input-output pairs for evaluation.The paired datasets enable evaluation despite the methods being designed for unpaired translation.
  • Ablations: Ablation studies examine the importance of the adversarial loss and cycle consistency loss by comparing the full method with several variants.This isolates the contributions of the two losses to the overall approach.
  • Applications: The method’s generality is demonstrated across a wide range of applications where paired training data does not exist.The paper uses “CycleGAN” as the abbreviated name for its method.

5.1. Evaluation

The evaluation compares CycleGAN with several baselines on unpaired Cityscapes labels↔photos and Google Maps maps↔aerial photos, using perceptual and task-specific quantitative metrics. CycleGAN generally outperforms the baselines, approaches paired pix2pix quality, and depends on both adversarial and cycle-consistency losses.

  • Qualitative comparison: CycleGAN often produces translations similar in quality to fully supervised pix2pix, while the baselines fail to achieve compelling results.The comparison is intended to assess how closely unpaired training can approach the paired-data reference.
  • Ablation study: CycleGAN achieves 0.58 Per-pixel acc., 0.22 Per-class acc., and 0.16 Class IOU, compared with lower scores for the ablated losses.Removing either GAN or cycle-consistency loss substantially degrades results, while one-direction cycle losses often cause instability and mode collapse.
  • Perceptual realism: CycleGAN fools AMT participants on around a quarter of trials in both maps→aerial photos and aerial photos→maps, while baselines almost never do.At 512 × 512, maps→aerial photos scores were CycleGAN: 37.5% ± 3.6% and pix2pix: 33.9% ± 3.1%; aerial photos→maps scores were CycleGAN: 16.5% ± 4.1% and pix2pix: 8.5% ± 2.6%.
  • Quantitative evaluation: CycleGAN outperforms the baselines on both Cityscapes labels→photo and photos→labels evaluations.The study uses the FCN score for labels→photo and per-pixel accuracy, per-class accuracy, and Class IOU for photo→labels.

5.2. Applications

The method is demonstrated across unpaired translation applications including collection style transfer, object transfiguration, season transfer, painting-to-photo generation, and photo enhancement. An identity-mapping loss preserves input color composition, while comparisons show more natural-looking results than neural style transfer.

  • Collection style transfer: Collection style transfer learns the style of an entire artwork collection, enabling landscape photos in the style of artists such as Van Gogh rather than one selected painting.The model is trained on landscape photographs from Flickr and WikiArt.
  • Photo generation from paintings: The identity-mapping loss helps preserve the color of paintings during painting-to-photo translation by discouraging unnecessary tint changes.Without it, Monet paintings may be mapped to sunset photographs because that remains compatible with adversarial and cycle-consistency losses.
  • Photo enhancement: Photo enhancement generates shallower-depth-of-field flower photos from smartphone images by translating them toward DSLR photos captured with larger apertures.The source smartphone photos usually have deep depth of field because of their small aperture.
  • Comparison with Gatys et al.: Compared with neural style transfer, the method produces natural-looking results similar to the target domain, whereas Gatys et al. often fails to produce photorealistic results.Neural style transfer requires finding target style images that closely match the desired output.

6. Limitations and Discussion

The method achieves compelling results in many unpaired translation tasks but remains unreliable for geometric changes, dataset mismatches, and semantic ambiguities. Despite a persistent gap versus paired training, plentiful unpaired data makes the approach valuable and motivates weak supervision.

  • Failure cases: CycleGAN often succeeds on color and texture changes but struggles with geometric transformations, such as dog→cat transfiguration, where outputs make minimal input changes.The paper identifies these as typical failure cases rather than uniformly positive results.
  • Failure cases: Training-distribution limitations can cause failures when examples omit relevant contexts, as in horse→zebra translation trained without horseback-riding images.The model became confused because the ImageNet wild horse and zebra synsets lacked people riding either animal.
  • Supervision gap: Unpaired translation still trails paired training and can permute semantically ambiguous labels, such as tree and building in photos→labels translation.The authors suggest weak or semi-supervised data may resolve such ambiguities and yield more powerful translators.
  • Discussion: Although limitations remain, plentiful completely unpaired data should be used, and the paper extends what is possible in the unsupervised setting.This conclusion frames the method’s contribution despite its unresolved failure modes.

7. Appendix

The appendix specifies the training procedure, network architectures, and dataset construction used across the experiments. It also documents task-specific resolutions, dataset sizes, and identity-loss settings.

  • Training procedure: Networks were trained from scratch with learning rate 0.0002, held constant for 100 epochs, then linearly decayed to zero over 100 more epochs.Weights were initialized from N(0, 0.02), and the discriminator objective was divided by 2 during optimization.
  • Datasets: Experiments used datasets ranging from 400 facade-label/photo training images to around 50, 000 edges-to-shoes training images, with task-specific resolutions and splits.Cityscapes used 2975 training images at 128 × 128, Maps↔aerial used 1096 images at 256×256, and Edges→shoes used around 50, 000 images for 5 epochs.
  • Task-specific settings: For Monet’s paintings→photos and flower photo enhancement, images were processed at high resolution or 360 pixels by width, with identity-loss weight 0.5λ and λ = 10.Monet training used random square crops and 512-pixel-wide inputs with preserved aspect ratio; flower datasets contained 1813 smartphone and 3326 DSLR images.
  • Generator architectures: Generators used 6 residual blocks for 128 × 128 images and 9 residual blocks for 256 × 256 or higher-resolution images.The architectures were adopted from Johnson et al. and implemented in both PyTorch and Torch.
  • Discriminator architectures: Discriminators used a 70 × 70 PatchGAN with the C64-C128-C256-C512 architecture, leaky-ReLU slope 0.2, and no InstanceNorm in the first C64 layer.A final convolution produced a 1-dimensional output.
Loading 1703.10593v7…