PaperlayerView exact revision on arXiv

Source-linked AI summary

U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation

Junho Kim, Minjae Kim, Hyeonwoo Kang, Kwanghee Lee

arXiv:1907.10830v4cs.CVeess.IV

TL;DR

Unsupervised image-to-image translation methods can struggle with differing amounts of shape and texture change and may require dataset-specific configurations. U-GAT-IT combines auxiliary-classifier attention with AdaLIN to focus translation and adapt normalization. Experiments report superior results to existing state-of-the-art GAN-based models while using a fixed architecture and hyper-parameters.

  • Problem

    Existing methods struggle across image translations with different amounts of shape and texture change and may require dataset-specific network or hyper-parameter settings.

  • Method

    U-GAT-IT uses an auxiliary-classifier attention module and Adaptive Layer-Instance Normalization to guide important regions and flexibly control transformation.

  • Results

    Experiments report superior performance compared with existing state-of-the-art GAN-based models using a fixed network architecture and hyper-parameters.

  • Takeaways & Limitations

    The method supports translations involving varied geometry and style changes without dataset-specific architecture or hyper-parameter modifications.

Abstract

from arXiv · show

We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model to focus on more important regions distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adaptive Layer-Instance Normalization) function helps our attention-guided model to flexibly control the amount of change in shape and texture by learned parameters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with a fixed network architecture and hyper-parameters. Our code and datasets are available at https://github.com/taki0112/UGATIT or https://github.com/znxlwm/UGATIT-pytorch.

1 INTRODUCTION

Existing unsupervised image-to-image translation methods struggle when domains differ substantially in shape and texture, often requiring dataset-specific preprocessing or tuning. U-GAT-IT addresses this with attention-guided translation and AdaLIN to support varied transformations using a fixed architecture and hyper-parameters.

  • Motivation: Prior methods often succeed on local texture changes but struggle with large shape changes in wild image translation tasks.Examples include photo2vangogh and photo2portrait versus selfie2anime and cat2dog.
  • Motivation: Pre-processing such as cropping and alignment is often used to limit difficult data distributions.
  • Motivation: Existing methods may require network structures or hyper-parameters adjusted for specific datasets.This limits their ability to handle both shape-preserving and shape-changing translations with one fixed configuration.
  • Proposed approach: U-GAT-IT combines a new attention module with a learnable normalization function in an end-to-end unsupervised translation method.
  • Proposed approach: The attention module identifies regions distinguishing source and target domains, helping the model transform important areas while ignoring minor regions.
  • Proposed approach: AdaLIN lets the attention-guided model flexibly control shape and texture changes without modifying architecture or hyper-parameters.

2 UNSUPERVISED GENERATIVE ATTENTIONAL NETWORKS WITH ADAPTIVE LAYER-INSTANCE NORMALIZATION

U-GAT-IT performs unsupervised translation with attention modules in both generator and discriminator, while AdaLIN adaptively combines instance and layer normalization. Its objective jointly uses adversarial, cycle, identity, and CAM losses to guide realistic translation and preserve relevant image properties.

  • Framework: U-GAT-IT uses two generators, two discriminators, and attention modules in both generator and discriminator for unpaired source-to-target translation.Discriminator attention guides the generator toward realistic regions, while generator attention emphasizes regions distinguishing the domains.
  • Attention module: The auxiliary classifier produces domain-specific attention maps that focus translation on semantically important regions and help the model determine where to improve.These maps are computed from encoded feature maps and embedded into the generator and discriminator.
  • AdaLIN: AdaLIN combines instance and layer normalization through a learned gate ρ constrained to [0, 1], with γ and β generated from the attention map.Values of ρ near 1 favor instance normalization, whereas values near 0 favor layer normalization; initialization differs between decoder residual and up-sampling blocks.
  • AdaLIN: AdaLIN combines AdaIN and layer normalization to selectively preserve or change content information across a wide range of image-to-image translation problems.The method addresses AdaIN's assumptions about channel correlations while retaining a faster normalization-based approach than WCT.
  • Loss function: The full objective jointly trains the model with adversarial, cycle, identity, and CAM losses, using Least Squares GAN for stable adversarial training.Cycle consistency addresses mode collapse, identity consistency preserves input-output color distributions, and CAM losses use auxiliary classifiers to identify regions requiring improvement.
  • Loss function: Figure 2 compares source images, generator and discriminator attention maps, and ablation results with and without CAM.The discriminator visualizations include local and global attention maps.

3 EXPERIMENTS

Experiments evaluate U-GAT-IT through ablations, visual comparisons, user preferences, and KID across five unpaired image-translation datasets. The results indicate benefits from combining attention modules with adaptive normalization, including strong qualitative performance and low KID across most tasks.

  • Experimental setup: The evaluation compares U-GAT-IT with CycleGAN, UNIT, MUNIT, DRIT, AGGAN, and CartoonGAN on five unpaired datasets.All baseline methods use the authors’ implementations; images are resized to 256 x 256 for training.
  • CAM analysis: Attention maps help the generator focus on discriminative source regions while discriminator maps capture both global structure and local regions.The generator can fine-tune areas emphasized by discriminators with different receptive-field sizes.
  • AdaLIN analysis: AdaLIN balances source-feature preservation and target-style transfer by learning whether decoder layers should rely more on instance or layer normalization.Instance normalization preserves source features but may limit global style transfer, whereas layer normalization transfers style but can lose source features.
  • AdaLIN analysis: The combined attention-and-AdaLIN model achieves the lowest KID in the selfie2anime ablation, outperforming either component used separately.The cited evaluation reports that combining both components improves performance beyond using attention or AdaLIN alone.
  • Qualitative evaluation: In a perceptual study with 135 participants, U-GAT-IT scores significantly higher except on photo2vangogh, where it remains comparable.Visual comparisons also report undistorted outputs, preserved source semantics, and stronger results than competing methods.
  • Quantitative evaluation: U-GAT-IT obtains the lowest KID scores except on photo2vangogh and photo2portrait, where the difference from the lowest score is small.The method also produces stable source-to-target and target-to-source translations and outperforms AGGAN on reported shape-changing transformations.

4 CONCLUSIONS

U-GAT-IT combines an attention module with AdaLIN for unsupervised image-to-image translation, producing visually pleasing results across datasets with fixed architecture and hyper-parameters. Experiments support attention-guided focus on distinct regions and AdaLIN’s role in handling varying geometry and style changes.

  • 4 CONCLUSIONS: U-GAT-IT combines attention guidance and AdaLIN to translate images across various datasets with a fixed network architecture and hyper-parameter.The method is reported to produce more visually pleasing results than existing state-of-the-art GAN-based models.
  • 4 CONCLUSIONS: Attention maps obtained by an auxiliary classifier guide the generator toward regions that are distinct between source and target domains.The experiments support the assumption that these maps help the generator focus on semantically important areas.
  • 4 CONCLUSIONS: AdaLIN is essential for translating datasets with different amounts of geometry and style changes.It adaptively supports the model across datasets containing varied transformation requirements.

A RELATED WORKS

Prior work established GAN-based image translation for paired and unpaired data, while CAM-based attention identifies discriminative regions for domain-guided transformation. Cycle consistency was introduced to support unpaired translation.

  • A RELATED WORKS: GANs have produced strong results in image generation, inpainting, and image translation.The cited applications span multiple generative vision tasks.
  • A RELATED WORKS: Pix2pix introduced a conditional GAN framework for image-to-image translation, while later work extended translation to unpaired datasets.CycleGAN introduced cyclic consistency loss for unpaired image translation.
  • A RELATED WORKS: Class Activation Maps identify discriminative image regions, which this work uses to intensively change regions distinguishing two domains.The approach combines global average pooling with global max pooling to improve results.

A.4 NORMALIZATION

Style-transfer research uses CNN feature statistics as style descriptors, and Instance Normalization removes style variation by normalizing those statistics. AdaLIN adaptively selects between Instance and Layer Normalization to control shape and texture changes.

  • A.4 NORMALIZATION: CNN feature statistics, including Gram matrices, means, and variances, can describe image styles.These statistics motivate normalization-based approaches to style transfer.
  • A.4 NORMALIZATION: Instance Normalization removes style variation by directly normalizing image feature statistics and is commonly used in style transfer.It is used more often than Batch Normalization or Layer Normalization in this setting.
  • A.4 NORMALIZATION: AdaLIN adaptively selects a ratio between Instance Normalization and Layer Normalization.This lets the attention-guided model flexibly control the amount of shape and texture change.

B.1 NETWORK ARCHITECTURE

U-GAT-IT’s generator uses a downsampling encoder, residual blocks, and an upsampling decoder, with Instance Normalization in the encoder and AdaLIN in the decoder.

  • B.1 NETWORK ARCHITECTURE: The generator encoder contains two stride-two convolution layers for down-sampling and four residual blocks.The encoder performs spatial reduction before decoding.
  • B.1 NETWORK ARCHITECTURE: The decoder contains four residual blocks and two stride-one up-sampling convolution layers.This structure reconstructs the translated image after the residual processing stages.
  • B.1 NETWORK ARCHITECTURE: Instance Normalization is used in the encoder, whereas AdaLIN is used in the decoder.The architecture assigns different normalization functions to the two generator stages.

B.2 TRAINING

Training uses a fixed optimization schedule, augmentation pipeline, and batch size across experiments.

  • Models use Adam with β1=0.5 and β2=0.999, weight decay of 0.0001, and batch size one.
  • Images are horizontally flipped with probability 0.5, resized to 286 x 286, and randomly cropped to 256 x 256.
  • Training runs for 500,000 iterations at learning rate 0.0001, followed by linear decay through 1,000,000 iterations.

C DATASET DETAILS

The experiments use several established unpaired translation datasets plus the newly created selfie2anime dataset, with dataset-specific training and test counts.

  • selfie2anime: Selfie2anime contains 3,400 female selfie images and 3,400 female anime-face images for training, plus 100 images per domain for testing.Selfie images come from a 46,836-image annotated selfie dataset; anime images were filtered from retrieved animation-character images and resized to 256 x 256.
  • horse2zebra and photo2vangogh: Horse2zebra uses 1,067 horse and 1,334 zebra training images, with test sets of 120 horse and 140 zebra images.
  • horse2zebra and photo2vangogh: Photo2vangogh uses 6,287 photo and 400 vangogh training images, with 751 photo and 400 vangogh test images.The vangogh training and test data are the same.
  • cat2dog and photo2portrait: The cat2dog and photo2portrait dataset passage reports class counts and randomly selected test sets for the corresponding experiments.The passage labels the counts as 871 cat, 1,364 zebra, 6,452 photo, and 1,811 vangogh, and lists test sets of 120 horse, 140 zebra, 751 photo, and 400 vangogh images.

D ADDITIONAL EXPERIMENTAL RESULTS

Additional results provide visual comparisons with attention feature maps across multiple translation tasks, while architecture tables detail the generator and discriminator designs.

  • Additional results: The supplementary section identifies Figures 5–12 as additional generation results for five datasets.
  • Network architectures: Table 4 details the generator architecture, while Tables 5 and 6 detail the local and global discriminators.
  • Visual comparisons: Figures 5 and 6 compare selfie2anime and anime2selfie using source images, generator attention maps, discriminator attention maps, and outputs from multiple methods.Compared methods include U-GAT-IT, CycleGAN, UNIT, MUNIT, DRIT, and AGGAN; Figure 5 also includes CartoonGAN.
  • Visual comparisons: Figures 7–10 compare horse2zebra, zebra2horse, cat2dog, and dog2cat with source images, attention maps, U-GAT-IT results, and five baseline outputs.
  • Visual comparisons: Figures 11 and 12 compare photo2vangogh and photo2portrait using generator or generation attention maps, local and global discriminator maps, and baseline outputs.
Loading 1907.10830v4…