PaperlayerView exact revision on arXiv

Source-linked AI summary

Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation

Runfa Chen, Wenbing Huang, Binghui Huang, Fuchun Sun, Bin Fang

arXiv:2003.00273v6cs.CV

TL;DR

Unsupervised image-to-image translation needs effective models without paired data, while conventional frameworks abandon the discriminator after training. NICE-GAN reuses discriminator layers as a target-domain encoder and decouples its training, achieving generally superior performance over state-of-the-art methods on several benchmarks.

  • Problem

    Unsupervised image-to-image translation uses unpaired data, but conventional frameworks leave the discriminator unused after training and separately encode images.

  • Method

    NICE-GAN reuses early target-domain discriminator layers as an encoder and trains that encoder only with the discriminator through a decoupled paradigm.

  • Results

    NICE-GAN generally achieves superior performance over state-of-the-art methods on several popular benchmarks, with ablations validating its proposed components.

  • Takeaways & Limitations

    Reusing discriminators can produce a more compact and effective translation architecture while retaining the discriminator's encoding component for inference.

Abstract

from arXiv · show

Unsupervised image-to-image translation is a central task in computer vision. Current translation frameworks will abandon the discriminator once the training process is completed. This paper contends a novel role of the discriminator by reusing it for encoding the images of the target domain. The proposed architecture, termed as NICE-GAN, exhibits two advantageous patterns over previous approaches: First, it is more compact since no independent encoding component is required; Second, this plug-in encoder is directly trained by the adversary loss, making it more informative and trained more effectively if a multi-scale discriminator is applied. The main issue in NICE-GAN is the coupling of translation with discrimination along the encoder, which could incur training inconsistency when we play the min-max game via GAN. To tackle this issue, we develop a decoupled training strategy by which the encoder is only trained when maximizing the adversary loss while keeping frozen otherwise. Extensive experiments on four popular benchmarks demonstrate the superior performance of NICE-GAN over state-of-the-art methods in terms of FID, KID, and also human preference. Comprehensive ablation studies are also carried out to isolate the validity of each proposed component. Our codes are available at https://github.com/alpc91/NICE-GAN-pytorch.

1. Introduction

NICE-GAN reuses discriminator layers as the target-domain encoder, yielding a more compact architecture and direct adversarial training. It addresses the resulting training inconsistency with a decoupled paradigm and reports superior benchmark performance.

  • Motivation: Unsupervised translation avoids the need for paired images but faces a non-identifiability problem addressed through additional constraints.Proposed regulations include weight-coupling, cycle-consistency, identity mapping, or combinations of these mechanisms.
  • Motivation: Conventional translation frameworks use separate encoders, generators, and discriminators for each domain, motivating a more compact formulation.The encoder embeds input images, the generator translates hidden vectors, and the discriminator aligns domains through GAN training.
  • Motivation: Discriminators perform semantic encoding before classifying images as real or false, revealing overlap between encoding and discrimination.Prior observations found strong responses to input images in early learned discriminator features.
  • NICE-GAN: NICE-GAN reuses early discriminator layers as the target-domain encoder, eliminating an independent encoder and retaining the encoding component for inference.The reused encoder is directly trained through discriminative loss and can inherit multi-scale discriminator expressiveness.
  • Decoupled training: Because the reused encoder belongs to both translation and discrimination, conventional min-max training can assign it conflicting optimization objectives.NICE-GAN decouples training by associating encoder updates only with the discriminator and not the generator.
  • Contributions: The paper reports a compact NICE-GAN architecture, a decoupled training paradigm, superior benchmark performance, and ablations validating its components.It identifies discriminator reuse for encoding as the central architectural contribution.

2. Related Work

Related work situates NICE-GAN among paired and unpaired image-to-image translation methods and distinguishes its discriminator reuse from related embedding and introspective generation mechanisms.

  • Image-to-Image Translation: Paired conditional GAN frameworks support image-to-image translation but require paired training data, limiting practical usage.Related extensions address tasks such as super-resolution and video generation.
  • Related discriminator reuse: NICE-GAN differs from INN by generating a novel image with one forward pass rather than requiring sequential training during inference.Both approaches reuse a discriminator for embedding images into hidden vectors.
  • Related discriminator reuse: NICE-GAN uses one domain's discriminator to generate a false sample of the other domain, forming a mutual introspective mechanism unlike IAN's pure-generation setting.IAN reuses one discriminator to generate self-false samples.

3. Our NICE-GAN

NICE-GAN reuses discriminator encoding layers as domain encoders, combining compactness with multi-scale discrimination and decoupled optimization. Its architecture includes residual attention and trains encoders only under discriminator objectives while freezing them during generator-related losses.

  • No Independent Component for Encoding: NICE-GAN divides each discriminator into an encoding part and a classification part, using the encoding part to replace the corresponding independent translator encoder.For example, E^D_y replaces E_y in f_y→x, while E^D_x replaces E_x in f_x→y.
  • No Independent Component for Encoding: Reusing discriminator encoders removes independent encoding components and retains the discriminator’s encoding function for inference, producing a more compact architecture.
  • Multi-Scale Discriminators: Multi-scale classifiers operate at local, middle, and global receptive fields, with feature maps progressively downsampled and connected to separate classification branches.The receptive fields are 10 × 10, 70 × 70, and 286 × 286 for the local, middle, and global scales, respectively.
  • Residual Attention: Residual attention combines weighted feature maps with the original encoder features through a trainable γ, recovering unmodified features when γ = 0.This lets the model adjust feature-map importance while preserving the original representation as a residual path.
  • Training Objectives: NICE-GAN uses adversarial, identity reconstruction, and cycle-consistency losses, with reconstruction and cycle losses addressing the non-identifiability of unsupervised translation.The generators use residual blocks and sub-pixel convolutional layers, while spectral normalization and cycle consistency are used to prevent mode collapse.
  • Decoupled Training: The decoupled strategy trains encoders under discriminator objectives but freezes them during adversarial minimization, reconstruction, and cycle-consistency losses.This prevents the shared encoder from simultaneously receiving conflicting translation and discrimination optimization directions.

4. Experiments

NICE-GAN is evaluated against established unsupervised translation methods on four unpaired benchmarks using FID, KID, and human preference. Results and ablations examine performance, model complexity, multi-scale discrimination, latent representations, and decoupled training.

  • Experimental setup: NICE-GAN is compared with CycleGAN, UNIT, MUNIT, DRIT, and U-GAT-IT on four unpaired image-translation benchmarks.The benchmarks are horse↔zebra, summer↔winter Yosemite, vangogh↔photo, and cat↔dog.
  • Quantitative comparison: FID and KID generally rank NICE-GAN lowest across tasks except zebra→horse, indicating strong translation performance across varying domains.The paper reports consistency between the two metrics, while noting that competing methods tend to perform well only on particular datasets.
  • Human evaluation: Human preference results are obtained by asking volunteers to choose the most realistic output among NICE-GAN and two selected baselines.Participants first see a source-target reference pair and then evaluate outputs in randomized order with unlimited time.
  • Ablation study: NICE-GAN’s performance improves when its key components are combined; ablations show that NICE and residual attention each contribute, with NICE having greater importance.The study evaluates NICE, multi-scale discrimination, and residual attention on cat↔dog, and reports the full model outperforming the variants.
  • Ablation study: NICE training produces latent distributions that are more clustered and closer across domains while remaining separable, shortening the latent transition path.The paper uses t-SNE and MMD on cat↔dog to analyze latent vectors with and without NICE.
  • Ablation study: Sharing more or fewer discriminator-encoder layers generally hurts performance, making the default choice of sharing the whole encoder preferable.The ablation varies the number of shared layers and reports degraded performance when the sharing depth is changed.
  • Ablation study: Removing the largest-scale discriminator causes serious detriment, while all three scales are generally necessary and the multi-scale design is more robust.The importance of the middle scale is task-dependent, and adding the smallest scale shows no clear enhancement on this task.
  • Decoupled training analysis: The standard coupled training variant is unstable, whereas a classifier-based variant is more stable but remains inferior to NICE-GAN.The training-curve comparison supports the paper’s analysis of inconsistency when encoder responsibilities overlap between translation and discrimination.

5. Conclusion

NICE-GAN reuses discriminators for encoding and uses decoupled training for unsupervised image-to-image translation. Experiments report generally superior performance over state-of-the-art methods, supported by ablation studies.

  • NICE-GAN reuses discriminators for encoding in unsupervised image-to-image translation.
  • The framework develops a decoupled training paradigm for efficient training.
  • NICE-GAN generally achieves superior performance over state-of-the-art methods on several popular benchmarks.

A.1. Introduction of state-of-the-art models

The reviewed state-of-the-art models address unsupervised image-to-image translation with different architectural mechanisms, including shared latent representations, style-content decomposition, multi-scale discrimination, and adaptive normalization.

  • CycleGAN: CycleGAN learns mappings between domains with adversarial and cycle-consistency losses using convolutional, residual, up-sampling, and discriminator layers.
  • UNIT: UNIT uses two VAE-GANs with a shared latent space, multi-scale discriminators, and shared high-level encoder-decoder weights.
  • MUNIT: MUNIT decomposes representations into content and style codes, uses AdaIN in the decoder and a multi-scale discriminator, and can generate varied outputs.
  • DRIT: DRIT decomposes images into content and style codes, uses a multi-scale discriminator, and shares content codes across domains.
  • U-GAT-IT: U-GAT-IT combines a CAM module with AdaLIN in an end-to-end unsupervised translation framework.

A.2. Network Architecture

NICE-GAN uses AdaLIN-based generators and spectrally normalized discriminators with Leaky-ReLU activations; its implementation also includes named normalization, attention, and convolutional notation.

  • Generator: NICE-GAN generators use AdaLIN in decoder layers except the final output layer.
  • Discriminator: NICE-GAN discriminators apply Leaky-ReLU with negative slope 0.2 and spectral normalization in all layers.
  • Normalization: Softmax constrains ρ to [0, 1] in AdaLIN, replacing clipping.
  • Notation: The architecture notation defines N as output channels, K as kernel size, S as side size, and P as padding size.
  • Notation: The notation also identifies LIN, SN, and RA as layer-instance normalization, spectral normalization, and residual connection in the CAM attention module.

A.3.1 Discussing γ

The residual-attention parameter γ is task-specific and controls the balance between attended and original features. For global-content translation tasks, γ is close to 0.

  • γ is task-specific in the Residual Attention module.
  • For photo→vangogh and summer→winter, γ is close to 0, emphasizing global features over local details.

A.3.2 More analysis on the multi-scale discriminator.

The multi-scale analysis indicates that all three discriminator scales are generally necessary, while NICE improves semantic, texture, and local-part encoding. Additional results show meaningful hidden-space interpolation and generally strong performance under extended training.

  • Multi-Scale Analysis: All three discriminator scales are generally necessary across translation tasks.Removing the local scale raises horse→zebra FID from 66 to 90, while omitting the medium scale raises summer↔winter FID from 76/76 to 88/96.
  • Hidden-vector analysis: Adversarial, identity reconstruction, and cycle-consistency losses jointly support domain transfer, reconstruction, and meaningful hidden interpolations.The interpolation results are consistent with a shared latent space enforced by the NICE framework and these three losses.
  • Hidden-vector analysis: NICE produces more concise and distinguishable semantic encodings than an independent encoder.Heat-map visualizations indicate clearer capture of object textures and local parts when NICE is used.
  • Additional comparisons: NICE-GAN generally performs superiorly to state-of-the-art methods when methods are compared at 200K discriminator iterations.The comparison uses FID and KID, for which lower values are better.
  • Qualitative results: Qualitative results vary by translation direction, with NICE-GAN showing stronger texture addition for horse→zebra but subtle color differences remaining.For zebra→horse, U-GAT-IT is reported to obtain the best results, while several other directions produce relatively ideal results.
Loading 2003.00273v6…