PaperlayerView exact revision on arXiv

Source-linked AI summary

DualGAN: Unsupervised Dual Learning for Image-to-Image Translation

Zili Yi, Hao Zhang, Ping Tan, Minglun Gong

arXiv:1704.02510v4cs.CV

TL;DR

Labeled image pairs for image translation are expensive, scarce, or misaligned. DualGAN addresses this gap by jointly training adversarial translators in opposite directions with reconstruction feedback from two unlabeled domains, achieving comparable or better results than conditional GAN on some tasks while remaining weaker on certain semantics-based tasks.

  • Problem

    Image-to-image translation commonly requires large labeled and matching image-pair datasets, but obtaining such data can be expensive, scarce, or impractical.

  • Method

    DualGAN jointly trains primal and inverse GAN translators between two unlabeled image domains, using adversarial domain feedback and reconstruction error from the closed loop.

  • Results

    With unlabeled data, DualGAN produces outputs comparable to or better than conditional GAN trained with labeled image and pixel-level correspondences, while outperforming GAN across evaluated tasks.

  • Takeaways & Limitations

    DualGAN enables general-purpose image-to-image translation from unlabeled domains across multiple real-world applications.

  • Takeaways & Limitations

    DualGAN is outperformed by conditional GAN on some semantics-based tasks because target distributions alone cannot provide pixel and label correspondence information.

Abstract

from arXiv · show

Conditional Generative Adversarial Networks (GANs) for cross-domain image-to-image translation have made much progress recently. Depending on the task complexity, thousands to millions of labeled image pairs are needed to train a conditional GAN. However, human labeling is expensive, even impractical, and large quantities of data may not always be available. Inspired by dual learning from natural language translation, we develop a novel dual-GAN mechanism, which enables image translators to be trained from two sets of unlabeled images from two domains. In our architecture, the primal GAN learns to translate images from domain U to those in domain V, while the dual GAN learns to invert the task. The closed loop made by the primal and dual tasks allows images from either domain to be translated and then reconstructed. Hence a loss function that accounts for the reconstruction error of images can be used to train the translators. Experiments on multiple image translation tasks with unlabeled data show considerable performance gain of DualGAN over a single GAN. For some tasks, DualGAN can even achieve comparable or slightly better results than conditional GAN trained on fully labeled data.

1. Introduction

The paper targets general-purpose image-to-image translation without labeled pairs, using dual learning to train translators from unlabeled domains. DualGAN combines adversarial domain matching with paired primal-dual translation and reports strong comparisons with GAN and conditional GAN.

  • Motivation: Labeled image pairs are costly, scarce, or misaligned across tasks such as photo-sketch and daylight-to-night translation.Pixelwise or patchwise labeling can be unrealistic, while moving objects create content discrepancies even when paired images are captured.
  • Motivation: The goal is an unsupervised, general-purpose framework that learns image translators using only two sets of unlabeled images.The central challenge is training a translator without data specifying correct translations.
  • Approach: DualGAN adapts dual learning by training primal and dual image translators whose nested application forms a reconstruction loop.The approach is inspired by simultaneously training opposite language translators with reconstruction loss.
  • Approach: Unlike the NLP approach, DualGAN uses adversarially trained GAN discriminators to capture target-domain distributions without requiring pretrained domain models.This design is intended for general-purpose image translation, where task-specific pretrained models may be difficult to obtain.
  • Contribution: DualGAN takes two unlabeled image domains as input and simultaneously learns translators in both directions, enabling diverse image-to-image translation tasks.The paper compares it with GAN and conditional GAN, reporting that DualGAN can outperform supervised methods on some applications.

2. Related work

Prior work includes supervised and application-specific conditional GANs, NLP dual learning, and unpaired image translation methods. DualGAN extends dual learning to general-purpose image translation with domain-adaptive discriminators and reconstruction feedback, without requiring a shared cross-domain representation.

  • Conditional GANs: Conditional GAN image-to-image methods include application-specific systems and a general-purpose approach that requires many labeled image pairs.Examples include super-resolution, texture synthesis, style transfer, video prediction, and general-purpose translation.
  • Dual learning: NLP dual learning trains opposite translators side-by-side from monolingual data using target-language membership scores and reconstruction error.Pretrained English and French language models provide the domain-specific feedback signals.
  • DualGAN: DualGAN replaces pretrained domain representations with domain-adaptive GAN discriminators and measures reconstruction error using mean absolute image differences.This supports a general-purpose image-to-image conversion framework without domain-specific knowledge.
  • Unpaired translation: CycleGAN shares the unpaired translation and cycle-consistency idea, while CoGAN instead learns a joint distribution through shared high-level weights.The paper characterizes CoGAN's weight-sharing assumption as task-dependent for general-purpose translation.
  • DualGAN: DualGAN trains primal and dual GANs jointly, allowing reconstruction error to provide feedback without a joint representation bridging the image domains.This distinguishes its mechanism from approaches that connect domains through shared representations.

3. Method

DualGAN uses two coupled GANs to translate between unlabeled image domains in both directions and reconstruct inputs through a closed loop. Its training combines adversarial domain matching with reconstruction, using architectures and optimization choices designed to preserve image information and stabilize learning.

  • Dual translation: DualGAN trains a primal generator GA from domain U to V and a dual generator GB from V to U using two GANs.Each GAN pairs its generator with a discriminator that distinguishes generated samples from real images in the target domain.
  • Dual translation: Images are translated across domains and then back to their originals, enabling reconstruction error to train both translators.For u, the loop is GB(GA(u, z), z′); for v, it is GA(GB(v, z′), z).
  • Objective and optimization: The discriminators use WGAN losses instead of sigmoid cross-entropy to improve convergence, sample quality, and optimization stability.Generators are updated only after the discriminators have trained for ncritic steps, providing more reliable gradient information.
  • Objective and optimization: L1 recovery loss is added to the GAN objective to measure reconstruction error and reduce the blurriness associated with L2 distance.The reconstruction weights λU and λV are typically set between 100.0 and 1,000.0, with smaller λU sometimes preferred when U contains natural images and V does not.
  • Network architecture: Generators are U-shaped fully convolutional networks with skip connections, while Patch-GAN discriminators emphasize local texture and style.Recovery loss encourages global and low-frequency preservation, complementing the discriminators’ local high-frequency modeling.
  • Training procedure: Training alternates ncritic discriminator updates with one generator update using mini-batch SGD and RMSProp.The procedure clips discriminator parameters and avoids momentum-based methods because Adam occasionally caused instability.

4. Experimental results and evaluation

The evaluation tests DualGAN on several general-purpose image-to-image translation tasks using both labeled benchmark datasets and unlabeled, unpaired material images. The reported visual comparisons emphasize structure and detail preservation, including under imperfect alignment.

  • Tasks and datasets: Experiments cover photo-sketch conversion, label-image translation, and artistic stylization across varied image-to-image translation tasks.The broader comparison includes day↔night, labels↔facade, face photo↔sketch, and map↔aerial photo translation.
  • Tasks and datasets: Four labeled datasets provide corresponding images for comparing DualGAN with GAN and supervised cGAN, although pixel-level feature alignment is not guaranteed.The datasets are PHOTO-SKETCH, DAY-NIGHT, LABEL-FACADES, and AERIAL-MAPS.
  • Qualitative results: In label→facade translation, DualGAN preserves label-image structure despite finer-detail misalignment, whereas GAN and cGAN produce artifacts and cGAN often becomes blurry.Examples include the roof in the second row and the entrance in the third row.
  • Tasks and datasets: DualGAN is also evaluated on MATERIAL, an unlabeled and unpaired dataset assembled from Flickr images spanning materials, illumination, composition, color, texture, and sub-types.This demonstrates evaluation with abundant web-sourced images rather than only corresponding image pairs.

5. Qualitative evaluation

Across paired and unpaired translation tasks, DualGAN generally produces sharper, less blurry, less artifact-prone outputs that preserve input content better than GAN and often compare favorably with supervised cGAN. User studies support these qualitative gains, while semantic-label translations remain a weakness relative to cGAN.

  • Experimental setup: DualGAN and GAN were trained without paired images, whereas cGAN used labeled image correspondences across day→night, labels↔facade, face photo↔sketch, and map↔aerial tasks.All models used the same training datasets and were tested on novel data.
  • Qualitative comparisons: Compared with GAN, DualGAN generally produces less blurry outputs, fewer artifacts, better input-structure preservation, and stronger target-domain texture, color, or style capture.The paper attributes these improvements to reconstruction loss and the feedback it provides through the dual generator.
  • Qualitative comparisons: DualGAN often compares favorably with supervised cGAN in output sharpness and faithfulness, including photo→sketch and sketch→photo face translation.The face results show sharper DualGAN photo→sketch outputs and more artifacts and blurriness for GAN and cGAN in sketch→photo.
  • Unpaired-domain tasks: DualGAN generates visually convincing material and painting translations from unpaired domains, with generally fewer artifacts than GAN.Evaluated tasks include oil↔Chinese, plastic→metal, metal→stone, leather→fabric, and wood↔plastic.
  • Quantitative evaluation: In AMT studies, DualGAN outperforms GAN on material perceptual success rates and realness scores for all tested tasks, while surpassing cGAN on two realness tasks.The material study evaluated 176 output images with ten Turkers each; the realness score ranged from 0 to 4.
  • Quantitative evaluation: cGAN outperforms DualGAN on facade→label and aerial→map segmentation because unpaired training makes proper label correspondences difficult to infer.The paper notes that additional pixel-level correspondence helps cGAN map labels to colors and textures correctly.

6. Conclusion

DualGAN uses unlabeled data and can improve on GAN, sometimes matching or surpassing supervised cGAN, but correspondence-dependent semantic labeling remains a limitation.

  • Conclusion: DualGAN produces sharper map→aerial results than GAN, despite occasionally mapping interstate highways to building roofs.The translation is trained without image correspondences.
  • Conclusion: For aerial photo→map translation, DualGAN outperforms GAN but remains below cGAN.cGAN labels local roads better but still misses interstate highways.
  • Conclusion: DualGAN improves GAN outputs and can generate results comparable to or better than labeled-data cGAN.This finding holds across various image-to-image translation tasks in the reported experiments.
  • Conclusion: DualGAN is outperformed by cGAN on some semantics-based labeling tasks because pixel and label correspondences cannot be inferred from target distributions alone.The authors propose investigating a small amount of labeled data as a warm start.
  • Conclusion: DualGAN outperforms GAN but is less accurate than cGAN on facades→label segmentation.Without correspondence, correctly segmented regions may receive incorrect labels.

Appendix

The appendix points readers to additional DualGAN results and the released source code.

  • Appendix: Additional results appear in Figures 11, 13, 15, 14, 12, 16, and 17, with source code released on GitHub.The cited repository is duxingren14/DualGAN.
Loading 1704.02510v4…