PaperlayerView exact revision on arXiv

Source-linked AI summary

A U-Net Based Discriminator for Generative Adversarial Networks

Edgar Schönfeld, Bernt Schiele, Anna Khoreva

arXiv:2002.12655v2cs.CVcs.LGeess.IV

TL;DR

GANs remain challenged by the need for globally coherent images with locally accurate structures and textures. The paper introduces a U-Net discriminator that supplies global and per-pixel feedback, plus CutMix-based consistency regularization, and reports improved FID-based results over BigGAN across three datasets.

  • Problem

    GANs still struggle to synthesize images with global semantic coherence, long-range structure, and exact local detail.

  • Method

    The paper combines a U-Net discriminator with global and per-pixel decisions and applies CutMix-based consistency regularization to its decoder output.

  • Results

    The method improves generated-image quality over BigGAN in FID and IS across FFHQ, CelebA, and COCO-Animals, including a 2.7-point average FID improvement.

  • Takeaways & Limitations

    U-Net GAN enables generators to produce images with varying structure, appearance, and detail while maintaining global and local realism.

  • Takeaways & Limitations

    The framework is evaluated with BigGAN and may require exploration with the StyleGAN family.

Abstract

from arXiv · show

Among the major remaining challenges for generative adversarial networks (GANs) is the capacity to synthesize globally and locally coherent images with object shapes and textures indistinguishable from real images. To target this issue we propose an alternative U-Net based discriminator architecture, borrowing the insights from the segmentation literature. The proposed U-Net based architecture allows to provide detailed per-pixel feedback to the generator while maintaining the global coherence of synthesized images, by providing the global image feedback as well. Empowered by the per-pixel response of the discriminator, we further propose a per-pixel consistency regularization technique based on the CutMix data augmentation, encouraging the U-Net discriminator to focus more on semantic and structural changes between real and fake images. This improves the U-Net discriminator training, further enhancing the quality of generated samples. The novel discriminator improves over the state of the art in terms of the standard distribution and image quality metrics, enabling the generator to synthesize images with varying structure, appearance and levels of detail, maintaining global and local realism. Compared to the BigGAN baseline, we achieve an average improvement of 2.7 FID points across FFHQ, CelebA, and the newly introduced COCO-Animals dataset. The code is available at https://github.com/boschresearch/unetgan.

1. Introduction

The paper addresses GANs’ difficulty synthesizing images with both global coherence and exact local detail by redesigning the discriminator as a U-Net with global and per-pixel outputs, supplemented by CutMix consistency regularization.

  • GANs still struggle to synthesize images with global semantic coherence, long-range structure, and exact detail.
  • Standard classification discriminators often emphasize either global structure or local details rather than maintaining representations of both.
  • The proposed U-Net discriminator simultaneously provides whole-image and per-pixel real/fake decisions, combining classification with segmentation.
  • CutMix consistency regularization spatially combines real and synthetic patches and their labels for the decoder while assigning the mixed image a fake global label.
  • The model is called U-Net GAN.
  • 4 FID points on FFHQ, 1.6 FID points on CelebA, and a reduction from 16.37 to 13.73 on COCO-Animals were reported against BigGAN.

2. Related work

Related work has improved GAN performance through objective, regularization, and architectural changes, including data-mixing augmentations that construct synthetic training images.

  • GAN research has pursued better performance through objective reformulations, regularization techniques, and architectural changes.
  • Prior work explored connections between representations without trading off local over global realism.
  • Mix&Cut regularizations: MixUp interpolates input images and target labels using the same random factor, while related methods also interpolate in intermediate layers.

3. U-Net GAN Model

U-Net GAN replaces the standard discriminator with an encoder-decoder that jointly provides global image classification and local per-pixel decisions. CutMix consistency regularization further trains these local predictions to remain consistent under mixed real/fake inputs.

  • 3.3. Implementation: The U-Net architecture and consistency regularization can be combined with other adversarial losses, while the generator architecture is left unchanged.The implementation builds the decoder from generator-network building blocks and retains the original discriminator encoder structure.
  • 3.1. U-Net Based Discriminator: Skip-connections combine low-level encoder features with high-level decoder features to support precise localization alongside global image context.This follows the encoder-decoder design used for dense prediction and image segmentation.
  • 3.1. U-Net Based Discriminator: The discriminator is extended into a U-Net whose encoder preserves global classification while its decoder segments images into real and fake regions per pixel.The decoder is connected to the encoder through a bottleneck and skip-connections, combining high- and low-level information.
  • 3.1. U-Net Based Discriminator: The discriminator loss combines scalar encoder decisions with decoder decisions averaged across all pixels.The encoder loss follows the scalar GAN discriminator output, while the decoder loss uses the mean per-pixel decision.
  • 3.2. Consistency Regularization: Consistency regularization penalizes differences between predictions on a CutMix image and the corresponding CutMix of predictions on its real and fake source images.The regularization term is added to the discriminator objective with weighting hyper-parameter λ, while the generator objective remains unchanged.
  • 3.2. Consistency Regularization: CutMix mixes patches from real and generated images using a binary mask, assigning spatial real/fake labels to the decoder and a global fake label to the encoder.The mixed image is globally labeled fake so the generator does not learn to reproduce CutMix artifacts.

4. Experiments

Experiments evaluate U-Net GAN against BigGAN across FFHQ, CelebA, and COCO-Animals using FID and IS, including ablations and training-dynamics analyses. The proposed model improves reported image-quality metrics and provides complementary global and local discriminator responses.

  • Experimental setup: Experiments use FFHQ and CelebA for unconditional synthesis and COCO-Animals for class-conditional synthesis at 256×256 or 128×128 resolution.COCO-Animals contains approximately 38k training images across 10 animal classes.
  • Experimental setup: FID is the primary metric, complemented by IS; reported values use 50k synthetic images and five independent runs.FID is described as more consistent with human evaluation of realism and variation than IS.
  • Results: The ablation study reports an average median-FID improvement of 3.7 points over BigGAN across the proposed components.The U-Net discriminator alone improves median FID from 12.42 to 10.86 on FFHQ and from 16.55 to 15.86 on COCO-Animals; CutMix consistency further lowers it to 7.63 and 13.87.
  • Results: 4.0 FID points separate U-Net GAN from BigGAN on FFHQ, where U-Net GAN reaches 7.48 FID.For COCO-Animals, U-Net GAN reaches 13.73 FID, improving 2.64 points over BigGAN; IS also increases from 11.77 to 12.29.
  • Discriminator responses: Encoder and decoder predictions are complementary: samples can be locally plausible but globally incoherent, or globally coherent but locally inconsistent.The decoder score is averaged over pixels for visualization, and the differing predictions provide more pronounced generator feedback.
  • Training dynamics: Approximately 60% of runs succeed for both models, while U-Net GAN collapses around 30k iterations versus more than 200k for BigGAN.Among successful runs, U-Net GAN achieves lower standard deviation in FID scores than BigGAN.
  • Training dynamics: U-Net GAN’s discriminator loss decays slowly, whereas BigGAN’s approaches zero quickly, which the authors link to the observed FID gains and potential for longer-training improvement.Encoder and decoder loss components show similar trends but operate at different scales because pixel-level decisions must be uniform across the image.

5. Conclusion

The paper proposes a U-Net discriminator that supplies global and local feedback, alongside CutMix-based consistency regularization. These changes improve generated-image realism across three datasets.

  • The U-Net discriminator provides both global and local feedback to the generator.
  • CutMix-based consistency regularization strengthens the U-Net discriminator.
  • The resulting generator synthesizes images with varying detail while maintaining global and local realism.
  • U-Net GAN improves over BigGAN in FID across FFHQ, CelebA, and COCO-Animals.

Supplementary Material

The supplementary material extends the main paper with additional evaluations, synthetic examples, and implementation details. It covers quantitative results, exemplar images, network architectures, and hyperparameter settings.

  • Section A reports additional quantitative results.
  • Sections B and C provide exemplar synthetic images on FFHQ and COCO-Animals.
  • Section E documents network architectures and hyperparameter settings.

A. Additional Evaluations

Additional evaluations compare U-Net GAN with BigGAN and other state-of-the-art models using FID and IS across FFHQ, COCO-Animals, and CelebA. Results show lower FID and more consistent performance across runs.

  • Evaluation uses FID and IS on FFHQ, COCO-Animals, and CelebA with both PyTorch and TensorFlow implementations.
  • 7.48 FID on FFHQ improves 4.0 points over the BigGAN discriminator in the unconditional setting.
  • 13.73 FID on COCO-Animals improves 2.64 points over BigGAN for conditional image generation.
  • 2.95 FID on CelebA outperforms COCO-GAN, PG-GAN, and the BigGAN baseline.
  • U-Net GAN outperforms BigGAN on best, mean, median, and standard-deviation FID across five independent runs.Standard deviation drops from 0.24 to 0.11 on COCO-Animals and from 0.16 to 0.04 on CelebA.

B. Qualitative Results on FFHQ

Qualitative FFHQ results show diverse, realistic images, smooth latent interpolations, detailed per-pixel discriminator responses, and stronger local realism than BigGAN. CutMix visualizations indicate that local predictions incorporate contextual information.

  • Qualitative samples: U-Net GAN generates diverse 256 × 256 FFHQ faces that maintain local and global realism.
  • Per-pixel discriminator feedback: The U-Net discriminator produces detailed, spatially coherent per-pixel feedback for generated images.
  • Latent-space interpolations: Latent interpolations produce semantically smooth changes in facial expressions, hair, beards, glasses, and hair color.
  • BigGAN comparison: U-Net GAN images exhibit finer details and better local realism than uncurated BigGAN images.
  • CutMix predictions: Decoder predictions for identical local patches vary across mixing contexts, indicating that local decisions use contextual information.

C. Qualitative Results on COCO-Animals

On COCO-Animals, U-Net GAN produces varied, high-quality class-conditional samples, object-focused discriminator maps, and semantically smooth latent interpolations.

  • Generated COCO-Animals samples: U-Net GAN generates good-quality COCO-Animals images with high variation within classes.Class-conditional projection in the decoder’s pixel output space does not introduce class leakage or otherwise affect class separation.
  • Per-pixel U-Net discriminator feedback: Per-pixel discriminator prediction maps often exhibit a bias toward objects.
  • Interpolations in the latent space: Latent-space interpolations are semantically smooth across animal classes and scenes.Backgrounds, instance counts, object shapes, and colors change smoothly between interpolated samples.

D. Details on the COCO-Animals Dataset

COCO-Animals is a medium-sized, 10-class animal dataset designed for high-resolution image-generation experiments, combining selected images from COCO and OpenImages.

  • Dataset composition: COCO-Animals contains approximately 38k images across 10 animal classes and targets experiments requiring a high-resolution CIFAR10 equivalent.The classes are bird, cat, dog, horse, cow, sheep, giraffe, zebra, elephant, and monkey.
  • Data sources and selection: The dataset combines COCO images with a human-verified OpenImages subset containing semantic label maps and binary masks.The selected samples come from OpenImages because of their better visual quality.
  • Data sources and selection: Bounding-box filters exclude images where the target animal is too small or occupies too much of the image.The class-specific thresholds were chosen so the number of appropriate images is approximately equal.

E. Architectures and Training Details

The U-Net discriminator combines BigGAN-style encoder and decoder components with skip-connected features and produces per-pixel outputs; training follows BigGAN hyperparameter settings and uses class-matched CutMix.

  • Architecture: The U-Net discriminator’s encoder and decoder follow the original BigGAN discriminator and generator setups, respectively.Encoder features are concatenated to decoder inputs, doubling the decoder’s number of input channels.
  • Architecture: The class-conditional 128 × 128 and unconditional 256 × 256 U-Net discriminators output one-channel spatial predictions from a 64-channel feature map.A 1 × 1 convolution reduces the decoder feature map to one channel.
  • Training: Training hyperparameters follow BigGAN settings and are summarized in Table S5.
  • Training: CutMix regularization mixes real and generated samples only within the same class for class-conditional generation.For example, real and fake zebras are mixed, but real zebras are not mixed with fake elephants.
  • Architecture: The supplementary architecture tables cover BigGAN and U-Net discriminator configurations for class-conditional 128 × 128 and unconditional 256 × 256 generation.
Loading 2002.12655v2…