Source-linked AI summary
Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
TL;DR
Image-to-image translation has traditionally required separate machinery for different input-output representations, leaving the general-purpose effectiveness of image-conditional GANs unclear. The paper presents conditional adversarial networks as a generic framework and shows reasonable results across many translation tasks, including aerial photos that fooled participants in 18.9% of trials.
Problem
Image-to-image translation tasks have traditionally used separate special-purpose methods, and the effectiveness of image-conditional GANs as a general-purpose solution remained unclear.
Method
The paper uses conditional adversarial networks with a U-Net generator and PatchGAN discriminator to learn task-adapted image translations and losses.
Results
Across a wide variety of image-to-image translation problems, conditional GANs produce reasonable results; aerial photos fooled participants on 18.9% of trials.
Takeaways & Limitations
Conditional adversarial networks are a promising approach for many image-to-image translation tasks, especially those involving highly structured graphical outputs.
Takeaways & Limitations
The final models produce only minor stochasticity despite dropout noise, limiting their ability to generate highly stochastic outputs.
Abstract
from arXiv · showhide
We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.
1. Introduction
Image-to-image translation converts one representation of a scene into another, such as an RGB image, gradient field, edge map, or semantic label map. The paper explores conditional GANs as a general-purpose solution, addressing manual loss design and demonstrating reasonable results across varied problems.
- Motivation: Image-to-image translation translates one possible representation of a scene into another corresponding representation.Examples include RGB images, gradient fields, edge maps, and semantic label maps.
- Problem: CNNs require manually designed losses, while naive Euclidean pixel loss tends to produce blurry outputs by averaging plausible results.The passage identifies loss design as substantial manual effort and explains blurring as a consequence of Euclidean-distance minimization.
- Approach: GANs learn a loss that distinguishes real from fake images while training a generative model to minimize that loss.This enables specifying a high-level goal such as making outputs indistinguishable from reality.
- Approach: Conditional GANs learn a conditional generative model by conditioning on an input image and generating a corresponding output image.This makes cGANs suitable for image-to-image translation tasks.
- Contribution: The paper demonstrates that conditional GANs produce reasonable results across a wide variety of image-to-image translation problems.It frames this as the primary contribution, while noting that earlier work focused on specific applications.
2. Related work
Prior image-to-image methods often use per-pixel losses or hand-designed structured losses, whereas conditional GANs learn a loss over joint output configurations. The method also builds on conditional GAN work while differing in its U-Net generator and PatchGAN discriminator.
- Structured losses for image modeling: Per-pixel classification and regression treat output pixels as conditionally independent given the input image.These formulations regard the output space as unstructured.
- Structured losses for image modeling: Conditional GANs learn a structured loss that penalizes joint output configurations rather than relying on predefined losses.Prior structured-loss methods include conditional random fields, SSIM, feature matching, nonparametric losses, convolutional pseudo-priors, and covariance-statistics matching.
- Conditional GANs: Earlier conditional GANs used discrete labels, text, or images for tasks including normal-map prediction, future-frame prediction, product-photo generation, and sparse-annotation image generation.An autoregressive approach also addressed image generation from sparse annotations.
- Conditional GANs: The method uses a U-Net generator and a convolutional PatchGAN discriminator that penalizes structure at the scale of image patches.A similar PatchGAN architecture had previously been proposed to capture local style statistics, while this work applies the approach across a wider range of problems.
3. Method
The method uses conditional GANs to map an observed image and noise to an output, with an adversarial discriminator distinguishing generated from real images. It combines L1-based low-frequency correctness with a U-Net generator and PatchGAN discriminator to preserve aligned structure and model local high-frequency detail.
- Conditional adversarial formulation: Conditional GANs learn G: {x, z} → y, while discriminator D distinguishes generated outputs from real images.The generator and discriminator are trained adversarially, with G minimizing and D maximizing the conditional GAN objective.
- L1 and PatchGAN losses: The method combines an L1 term for low-frequency correctness with adversarial training focused on high-frequency structure.L1 is preferred to L2 because it encourages less blurring, while the discriminator models local image-patch structure.
- Noise input: Adding Gaussian noise was ineffective in initial experiments because the generator learned to ignore it, motivating the method’s final noise-handling design.Without noise, the mapping would be deterministic and unable to match distributions beyond a delta function.
- U-Net generator: The generator uses U-Net skip connections to bypass the bottleneck and directly transfer low-level information shared by aligned input and output structures.Each connection concatenates all channels at layer i with those at layer n −i.
- PatchGAN discriminator: PatchGAN classifies each N ×N image patch as real or fake, penalizing local structure rather than requiring a full-image discriminator.The architecture can use patches much smaller than the full image while producing high-quality results, with fewer parameters and faster execution.
4. Experiments
Experiments across diverse graphics and vision tasks show that conditional GANs support broadly applicable image-to-image translation, including realistic outputs from small datasets. Ablations and evaluations indicate that combining cGAN and L1 losses, using U-Net and PatchGAN designs, and testing perceptual realism provide strong practical results.
- Experimental Tasks: The method is evaluated on diverse tasks including label-map↔photo, map↔aerial photo, colorization, edges→photo, sketch→photo, day→night, and thermal→color translation.The experiments use datasets including Cityscapes, CMP Facades, scraped Google Maps data, and other referenced collections.
- Data Requirements and Speed: 400 facade images and 91 unique webcams suffice for decent results, with facade training taking less than two hours on a single Pascal Titan X GPU.At test time, all models run in well under a second.
- Losses: With λ = 100, combining cGAN and L1 reduces artifacts relative to cGAN alone, while L1 alone produces reasonable but blurry results.The cGAN alone produces sharper images but introduces visual artifacts on certain applications.
- Architectures: The U-Net generates realistic cityscape images whereas the encoder-decoder without skip connections is unable to learn realistic image generation in these experiments.The comparison tests whether low-level information can shortcut across the network.
- Discriminators: The 70 × 70 PatchGAN alleviates the tiling artifacts of 16×16 PatchGAN and achieves slightly better scores, whereas the full 286 × 286 ImageGAN has a considerably lower FCN-score.Scaling beyond 70 × 70 does not appear to improve visual quality.
5. Conclusion
Conditional adversarial networks are presented as a promising general approach to image-to-image translation, especially for highly structured graphical outputs. Their task- and data-adapted learned loss supports application across varied settings.
- Conditional adversarial networks are promising for many image-to-image translation tasks, particularly those with highly structured graphical outputs.
- The networks learn a loss adapted to the task and data at hand, enabling applicability across a wide variety of settings.
- The method is illustrated across labels→photo, day→night, edges→handbags, edges→shoes, photo inpainting, and thermal-to-RGB translation.
6. Appendix
The appendix specifies the encoder–decoder, U-Net, and discriminator architectures, along with training augmentation, dataset protocols, and a batch-normalization caveat affecting batch-size-1 experiments.
- Architectures: Convolutional layers use 4 × 4 filters with stride 2; encoders and discriminators downsample, while decoders upsample by 2.Ck denotes Convolution-BatchNorm-ReLU, and CDk adds 50% dropout.
- Architectures: The U-Net adds concatenative skip connections between encoder layer i and decoder layer n−i, increasing decoder channel counts.Its decoder is CD512-CD1024-CD1024-C1024-C1024-C512-C256-C128.
- Architectures: The 70 × 70 discriminator uses C64-C128-C256-C512, while other discriminators vary depth to change receptive-field size.The appendix also defines 1 × 1 and 16 × 16 variants.
- Training protocols: Random jitter resizes 256×256 inputs to 286 × 286 before randomly cropping them back to 256 × 256.Training protocols vary across tasks, including mirroring, batch sizes, epochs, and dataset-specific train/test splits.
- Caveat: For batch-size-1 experiments, batch normalization zeros bottleneck activations, effectively skipping the innermost layer, although removing it produces little observed difference.The public code removes batch normalization from this layer, but the paper’s experiments retain the original setup.