Image-to-Image Translation with Conditional Adversarial Networks

1. Introduction

• Image-to-image translation maps one representation of a scene to another, unifying tasks traditionally handled with separate, special-purpose machinery [39] [18].Given sufficient training data, the common task is to predict pixels from pixels, whether the representations are RGB images, gradients, edges, or semantic labels.Source: §1 · p. 1
• CNNs automate image prediction, but effective losses still require manual design; minimizing Euclidean pixel distance tends to produce blurry results [43].Euclidean distance favors averaging plausible outputs, causing blur; designing losses that encourage sharp, realistic images generally requires expert knowledge.Source: §1 · pp. 1–2
• Conditional GANs generate an output conditioned on an input image, making them suitable for image-to-image translation.GANs [13] [44] learn a real-versus-fake loss while training a generator to minimize it. This data-adaptive loss supports tasks that traditionally require different loss functions.Source: §1 · p. 2
• Conditional GANs produce reasonable results across varied image-to-image translation problems using a simple framework.The paper also analyzes the effects of important architectural choices.Source: §1 · p. 2

Structured losses for image modeling

• Conditional GANs learn losses over joint output configurations, unlike per-pixel classification or regression formulations that assume conditional independence [39] [35].Earlier structured losses include conditional random fields [10], SSIM, and feature matching [15]. The learned conditional GAN loss can theoretically penalize any structural difference between output and target.Source: §2 · p. 2

Conditional GANs

• Unlike application-specific predecessors, this framework contains no application-specific components, making its setup simpler than most earlier approaches.Prior conditional GANs used discrete labels [41] [13], text [46], and images, including normal-map inputs and sparse annotations [31] [48].Source: §2 · p. 2
• Earlier inpainting [43] and superresolution [36] systems used unconditional GANs, relying on additional terms such as L2 regression to enforce input dependence.These approaches differ from conditioning the GAN itself on the input image.Source: §2 · p. 2
• The generator uses a U-Net-based architecture, while the convolutional PatchGAN discriminator penalizes structure only at image-patch scale.A similar PatchGAN previously captured local style statistics [38]; this work demonstrates broader applicability and investigates the effect of changing patch size.Source: §2 · p. 2

3.1. Objective

• The generator combines conditional adversarial training with L1 reconstruction loss, using L1 rather than L2 to encourage less blurring.The generator minimizes the adversarial objective while the discriminator maximizes it. An unconditional discriminator that does not observe the input image is also compared.Source: §3.1 · p. 3

3.3. Optimization and inference

• Dropout replaces ignored Gaussian input noise during training and inference, but outputs exhibit only minor stochasticity, leaving full conditional-distribution entropy uncaptured.Optimization alternates discriminator and generator updates using Adam [32]. Inference retains dropout and uses test-batch normalization statistics rather than aggregated training statistics.Source: §3.1 · p. 3, §3.3 · p. 4

3.2.1 Generator with skips

• The U-Net generator concatenates mirrored encoder and decoder layers, bypassing the bottleneck to preserve low-level information shared by aligned inputs and outputs.The generator and discriminator architectures are adapted from [44], using convolution-BatchNorm-ReLu modules [29].Source: §3.2.1 · p. 3, §3.2 · p. 3

3.2.2 Markovian discriminator (PatchGAN)

• PatchGAN classifies each N × N patch as real or fake, averaging convolutional responses to enforce high-frequency structure while L1 handles low-frequency correctness.It assumes independence between pixels separated by more than a patch diameter, making it interpretable as a texture/style loss.Source: §3.2.2 · p. 4
• Patches much smaller than the full image can still produce high-quality results, allowing smaller PatchGANs with fewer parameters, faster execution, and arbitrarily large inputs.The passage attributes the supporting demonstration to Section 4.4.Source: §3.2.2 · p. 4

4. Experiments

• 400 facade training images and 91 unique day-to-night webcams illustrate small-data feasibility; facade training took under two hours on one Pascal Titan X GPU.Tests span semantic labels↔photos, architectural labels→photos, maps↔aerial imagery, colorization, edges/sketches→photos, day→night, thermal→color, and inpainting. All models run in well under a second on that GPU.Source: §4 · p. 4

4.1. Evaluation metrics

• Evaluation combines human real-versus-fake judgments with semantic recognition, since per-pixel error misses the joint image statistics targeted by structured losses.AMT studies evaluate maps, aerial photos, and colorization; Cityscapes-trained FCN-8s [39] scores synthesized photos by classification accuracy against their conditioning labels.Source: §4.1 · pp. 4–5

4.2. Analysis of the objective function

• Combining L1 and cGAN sharpens outputs while reducing artifacts; conditioning or L1 preserves input correspondence, and cGAN better matches ground-truth color distributions.GAN-based objectives achieve higher FCN-scores; unconditional GAN collapses toward input-independent outputs, whereas L1+GAN performs similarly to L1+cGAN. L1 alone favors blurry outputs and conditional-median, grayish colors; adding it to cGAN with λ = 100 reduces artifacts.Source: Figure 4 · p. 5, Table 1 · p. 6, §4.2 · p. 6

4.3. Analysis of the generator architecture

• U-Net skip connections outperform an otherwise identical encoder-decoder for Cityscapes synthesis under both conditional adversarial and L1-only training.Skip connections carry low-level information across the network; removing them prevented realistic image synthesis in these experiments.Source: Table 2 · p. 6, Figure 5 · p. 6, §4.3 · p. 6

4.4. From PixelGANs to PatchGANs to ImageGANs

• 70×70 PatchGAN reduces 16×16 tiling artifacts, slightly improves FCN-scores, and exceeds the full 286×286 ImageGAN’s FCN-score despite similar visual quality.PixelGAN increases colorfulness without spatial sharpening, whereas 16×16 patches produce sharp outputs. Fixed-size patch discrimination supports arbitrarily large images, and convolutional generators trained at 256×256 successfully translate 512×512 inputs.Source: Table 3 · p. 6, §4.4 · p. 7, Figure 8 · p. 8

4.5. Perceptual validation

• 18.9% of trials fooled participants for generated aerial photos, significantly above L1; photo→map achieved 6.1%, not significantly different from L1.L1+cGAN colorization fooled participants on 22.5% of trials, statistically similar to the L2 variant of but below ’s full method at 27.8%. The authors suggest rigid map geometry makes structural errors more visible than in aerial photographs.Source: Table 4 · p. 7, Table 5 · p. 7, §4.5 · pp. 7–8

4.6. Semantic segmentation

• L1 regression outperforms cGAN on Cityscapes photo→labels, although cGAN alone can generate label maps with reasonable accuracy.The authors suggest less ambiguous vision targets often need only reconstruction losses. The label maps were not fully discrete because of bilinear resizing and JPEG artifacts.Source: Table 6 · p. 9, §4.6 · p. 9, §4.7 · p. 9

4.7. Community-driven Research

• Community projects extend pix2pix to new translation tasks, suggesting broad applicability, but are uncontrolled creative demonstrations that may modify the released code.Examples include background removal, palette generation, sketch-to-portrait translation, and pose transfer.Source: §4.7 · p. 9

• The results suggest conditional adversarial networks are promising for many image-to-image translation tasks.Source: §5 · p. 9
• These networks learn a loss adapted to the task and data, making them applicable across a wide variety of settings.Source: §5 · p. 9
• The approach is especially promising for tasks involving highly structured graphical outputs.Source: §5 · p. 9

6.1. Network architectures

• 4 × 4 stride-2 convolutions downsample encoder and discriminator features and upsample decoder features, each by a factor of 2.Ck denotes Convolution-BatchNorm-ReLU with k filters; CDk additionally applies dropout at a rate of 50%.Source: §6.1 · p. 16

6.1.1 Generator architectures

• U-Net augments the encoder-decoder with skip connections that concatenate corresponding encoder and decoder activations.Connections link encoder layer i to decoder layer n−i, increasing decoder channel counts. The output convolution produces three channels generally, or two for colorization, followed by Tanh.Source: §6.1.1 · p. 16

6.1.2 Discriminator architectures

• 70 × 70 discriminators use C64-C128-C256-C512, while other receptive-field sizes use variants of the same basic architecture.Both 1 × 1 and 16 × 16 discriminators use C64-C128; the former uniquely uses 1 × 1 filters throughout. A final convolution and Sigmoid produce a one-dimensional output; the first layer omits BatchNorm, and leaky ReLUs use slope 0.2.Source: §6.1.2 · p. 16

6.2. Training details

• 256×256 inputs receive random jitter by resizing to 286 × 286 and randomly cropping back to 256 × 256.All networks are trained from scratch, with Gaussian weight initialization using mean 0 and standard deviation 0.02.Source: §6.2 · p. 16
• 200 epochs train Cityscapes labels→photo, architectural labels→photo, and maps↔aerial photograph models, using random jitter and mirroring.Training uses 2975 Cityscapes images [12], 400 architectural images, and 1096 Google Maps images; Cityscapes architecture comparisons use batch size 10, while its objective experiments and the other two tasks use batch size 1. Testing uses Cityscapes validation data, a random architectural split, and a latitude-based map split with a buffer preventing shared training and test pixels.Source: §6.2 · p. 16
• 4-image batches train colorization, edge reconstruction, day→night, thermal→color, and inpainting models, with task-specific training durations and test splits.Colorization uses 1.2 million ImageNet images [51] for approximately 6 epochs with mirroring but no jitter, following evaluation protocols and [35]; shoes use 50k images and handbags 137K images, both for 15 epochs with random train/test splits. Day→night uses 17823 images from 91 training webcams for 17 epochs with jitter and mirroring and 10 test webcams; thermal→color uses 36609 images from sets 00–05 for 10 epochs and tests on sets 06–11; inpainting uses 14900 images for 25 epochs and 100 held-out images following [43].Source: §6.2 · pp. 16–17

6.3. Errata

• Batch size 1 zeros bottleneck activations through batch normalization; removing bottleneck normalization fixes this, with little observed difference in the reported experiments.U-Net can bypass the zeroed bottleneck, whereas the encoder-decoder requires batch size greater than 1 unless bottleneck normalization is removed. Public code implements the correction; the paper retains the original experiments.Source: §6.2 · p. 16, §6.3 · p. 17

6.4. Change log

• arXiv v2 reran generator comparisons with batch size 10 and recomputed FCN-scores after evaluation cleanup; later versions expanded reported examples.FCN evaluation switched to lossless PNG results and removed unnecessary downsampling; v2 also added thermal→color, inpainting, and community results. Version v3 added further community contributions and corrected minor typos.Source: §6.4 · p. 17