PaperlayerView exact revision on arXiv

Source-linked AI summary

AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks

Hao Tang, Hong Liu, Dan Xu, Philip H. S. Torr, Nicu Sebe

arXiv:1911.11897v5cs.CVcs.LGeess.IV

TL;DR

Existing unpaired image-to-image translation methods can produce artifacts because they do not adequately identify discriminative content and may alter backgrounds. AttentionGAN uses attention-guided generators and discriminators to focus translation on foreground regions while preserving backgrounds, and experiments across eight datasets report sharper, more realistic images than existing methods.

  • Problem

    Unpaired image-to-image translation remains challenging because existing models change unwanted parts and are affected by background changes.

  • Method

    AttentionGAN uses built-in attention-guided generators to produce attention and content masks, fuse them with input images, and discriminators that consider attended regions.

  • Results

    Experiments on eight popular datasets show photo-realistic images with clearer details and new state-of-the-art results compared with existing methods.

  • Takeaways & Limitations

    AttentionGAN focuses translation on discriminative foreground content while preserving source-background information across unpaired and multi-domain translation tasks.

Abstract

from arXiv · show

State-of-the-art methods in image-to-image translation are capable of learning a mapping from a source domain to a target domain with unpaired image data. Though the existing methods have achieved promising results, they still produce visual artifacts, being able to translate low-level information but not high-level semantics of input images. One possible reason is that generators do not have the ability to perceive the most discriminative parts between the source and target domains, thus making the generated images low quality. In this paper, we propose a new Attention-Guided Generative Adversarial Networks (AttentionGAN) for the unpaired image-to-image translation task. AttentionGAN can identify the most discriminative foreground objects and minimize the change of the background. The attention-guided generators in AttentionGAN are able to produce attention masks, and then fuse the generation output with the attention masks to obtain high-quality target images. Accordingly, we also design a novel attention-guided discriminator which only considers attended regions. Extensive experiments are conducted on several generative tasks with eight public datasets, demonstrating that the proposed method is effective to generate sharper and more realistic images compared with existing competitive models. The code is available at https://github.com/Ha0Tang/AttentionGAN.

I. INTRODUCTION

Unpaired image-to-image translation remains difficult because existing methods can alter irrelevant background content and struggle to capture discriminative semantic regions. AttentionGAN addresses this with attention-guided generation and discrimination, achieving sharper, more realistic results across public datasets.

  • I. INTRODUCTION: Unpaired translation methods can change unwanted image parts and be affected by background changes, while object-mask supervision is difficult to collect.ContrastGAN uses dataset-provided object masks, but obtaining such annotations is hard.
  • I. INTRODUCTION: AttentionGAN learns foreground and background attentions to transform discriminative foreground content while preserving background information from the input.Foreground attention selects generated foreground regions, whereas background attention maintains the input background.
  • I. INTRODUCTION: Its generators produce attention and content masks, fuse them with the input image, and use attention-guided discriminators that focus on attended foreground regions.The framework includes two attention-guided generation schemes and discriminators designed to consider attended regions.
  • I. INTRODUCTION: Experiments on eight public datasets report photo-realistic images with clearer details and new state-of-the-art results compared with existing methods.The paper also reports better results than GANimorph and CycleGAN in the horse-to-zebra example.
  • I. INTRODUCTION: The paper extends earlier work with a more advanced generation scheme, broader evaluation, ablations, hyper-parameter studies, mask visualizations, and additional baselines.The extension includes experiments on eight popular datasets and emphasizes architectural and implementation details for reproducibility.

II. RELATED WORK

AttentionGAN addresses unpaired image-to-image translation by learning attention and content masks within the generators, focusing translation on discriminative foreground regions while preserving background content. Unlike approaches relying on extra data or networks for attention, its mask-based schemes are integrated into the translation framework.

  • Motivation: Unpaired image-to-image translation is needed because paired input-output data are often difficult or expensive to obtain.This difficulty is especially pronounced for complex outputs such as artistic stylization.
  • Related approaches: Existing attention-guided methods obtain attention masks from extra annotations, segmentation models, or attention networks, increasing system parameters, training time, and storage.AttentionGAN instead generates attention masks within its own translation framework.
  • AttentionGAN: AttentionGAN disentangles foreground and background by generating attention masks and content masks, then translating salient objects while keeping other image regions unaltered.The framework uses two generators, G and F, for mappings between domains X and Y.
  • Attention-guided generation: Each generator combines a parameter-sharing encoder with separate attention-mask and content-mask generators, whose outputs are fused with the input image to form the translated image.The attention mask controls each pixel’s contribution from the generated content and the input image.
  • Scheme I limitations: Scheme I is limited on complex tasks because shared mask generation, a single attention mask, and a single content mask reduce image-generation quality and flexibility.It performs well when source and target domains have substantial overlap, such as facial-expression translation.
  • Attention-guided generation scheme II: Scheme II generates both foreground and background attention masks, enabling simultaneous foreground translation and background preservation.The foreground masks are flexible rather than mutually complementary, while the background mask complements them.
  • Attention-guided generation scheme II: Scheme II produces multiple intermediate content masks to enlarge the generation space for complex translation tasks such as horse-to-zebra conversion.The n−1 generated masks plus the input image form an n-image candidate pool and a 3n-channel generation space.

B. Attention-Guided Cycle

Attention-guided cycles extend cycle consistency to the mask-based generators so translations can be mapped back toward their original domains. Scheme II reconstructs inputs with separate attention and content subnetworks and multiple masks.

  • Cycle consistency: Cycle consistency requires translating an image to the other domain and back so the reconstruction remains close to the original input.The two cycles are x→G(x)→F(G(x))≈x and y→F(y)→G(F(y))≈y.
  • Attention-guided cycle: AttentionGAN applies modified cycle-consistency losses to both attention-guided generation schemes.The reconstructed images are required to remain very close to their corresponding original images.
  • Scheme II cycle: In scheme II, the reverse generator F uses a parameter-sharing encoder, an attention-mask generator, and a content-mask generator to reconstruct the original input.It generates multiple content masks and attention masks before fusing them with the translated image.
  • Scheme II cycle: The scheme-II reverse mapping reconstructs x from G(x), while the corresponding forward mapping reconstructs y from F(y).Both recovered images are constrained to be very close to their original inputs.

C. Attention-Guided Discriminator

The attention-guided discriminator receives attention masks with images and distinguishes fake from real image pairs, focusing on discriminative content. It is used only in scheme I, while scheme II did not improve from adding it.

  • Vanilla adversarial training distinguishes generated images from real images in each target domain.G minimizes LGAN(G, DY), while DY maximizes it; DX similarly distinguishes F(y) from x.
  • The attention-guided discriminator takes an attention mask and image, distinguishing fake pairs [Ay, G(x)] from real pairs [Ay, y].A corresponding discriminator handles [Ax, F(y)] and [Ax, x].
  • The attention-guided discriminator is intended to focus on discriminative content and ignore unrelated content.
  • The attention-guided discriminator is used only in scheme I because preliminary use in scheme II did not improve performance.The authors suggest scheme II may already learn the most discriminative content.

D. Optimization Objective

The optimization objectives combine adversarial, cycle-consistency, identity or pixel, attention, and total-variation terms. Scheme I uses stronger constraints, including attention and pixel losses, while scheme II uses GAN, cycle-consistency, and identity-preserving losses.

  • Scheme II optimizes GAN, cycle-consistency, and identity-preserving losses with λcycle and λid controlling their relative contributions.
  • Scheme I combines GAN, attention-guided GAN, cycle-consistency, total-variation, and pixel losses.λgan, λcycle, λpixel, and λtv control the relative contribution of these terms.
  • Attention masks are learned without ground-truth annotations from gradients of the generators, discriminators, and other losses.Total Variation regularization prevents masks from saturating to 1.
  • Scheme I uses pixel loss to reduce changes and constrain the generator, with L1 distance as its loss measurement.

E. Implementation Details

The implementation modifies the CycleGAN generator for two attention-guided schemes. Scheme I outputs one attention mask and one content mask, whereas scheme II produces multiple masks that are fused with the input.

  • Scheme I maps an RGB image to a one-channel attention mask and a three-channel content mask.
  • Scheme II outputs n attention masks and n−1 content masks, which are fused with the input image to produce final results.The experiments set n=10.
  • The attention generator uses modified CycleGAN components, and its ten masks designate one background mask and nine foreground masks.
  • The vanilla and attention-guided discriminators share the same architecture, but the latter also receives an attention mask with the image.

A. Experimental Setups

AttentionGAN is evaluated against prior image-translation models on eight public datasets spanning face and natural-image tasks. The experiments use fixed loss weights following previous work and compare against multiple state-of-the-art baselines.

  • The evaluation uses eight datasets: four face datasets and four natural-image datasets.The face datasets are CelebA, RaFD, AR Face, and Selfie2Anime; natural-image datasets include Horse2Zebra, Apple2Orange, Maps, and Style Transfer.
  • The experiments set λcycle=10, λgan=0.5, λpixel=1, and λtv=1e−6 for Eq. (12).
  • They also set λcycle=10 and λid=0.5 in Eq. (11), following prior unsupervised image-translation work.
  • The comparison includes unpaired translation baselines such as CycleGAN, DualGAN, DIAT, DiscoGAN, DistanceGAN, UNIT, MUNIT, and DRIT.

1) Ablation Study:

The ablations show that AttentionGAN’s attention-guided components are jointly important, while the refined generation scheme improves robustness on complex translation tasks.

  • Component analysis: Removing any attention-guided discriminator, generator, attention loss, or pixel loss substantially degrades AR Face and Horse2Zebra results.The reported AMT and PSNR ablations indicate that all four components are critical.
  • Architectural design: Removing any of the three architectural designs significantly decreases Horse2Zebra performance: separate mask networks, foreground and background masks, and intermediate content masks.The full model avoids the artifacts and background changes observed in the reduced variants.
  • Architectural design: Sharing one network for attention and content masks performs worse because grayscale attention masks and RGB content masks interfere with one another.The paper identifies separate sub-networks as the most important of the three designs and a prerequisite for the other two.
  • Generation schemes: Generation scheme I works for facial expression transfer but can reproduce inputs on horse-to-zebra, apple-to-orange, and map-to-aerial-photo tasks.Scheme II is reported to handle all these tasks, supporting its refinement over scheme I.

2) Experiments on Face Images:

Experiments on facial and related image-translation tasks show sharper, more accurate outputs and effective background preservation across varied conditions. Attention masks focus changes on task-relevant facial or foreground regions, while scheme II also performs strongly on selfie-to-anime translation.

  • Face-image results: AttentionGAN produces visually better facial attribute-transfer results than StarGAN without changing the background.The comparison is conducted on CelebA using a multi-domain extension.
  • Face-image results: AttentionGAN performs better than DIAT, CycleGAN, StarGAN, and IcGAN for facial expression transfer, and competitively with GANimation without action-unit annotations.GANimation requires OpenFace-derived action units as extra training data, limiting its practical application.
  • Face-image results: AttentionGAN consistently generates sharper images with correct facial expressions or attributes across four face-image datasets.The datasets vary in race, pose, style, skin color, illumination, occlusion, and background complexity.
  • Face-image results: AttentionGAN achieves the best quantitative results against competing models on facial expression translation and facial attribute transfer.Table V reports AMT and PSNR for expression translation, while Table VI reports AMT for attribute transfer.
  • Related translation tasks: AttentionGAN achieves the best selfie-to-anime KID result except for U-GAT-IT while using fewer discriminators than U-GAT-IT.U-GAT-IT uses auxiliary classifiers and four discriminators, whereas AttentionGAN uses two.
  • Mask visualization: Learned attention and content masks vary with facial expressions and attributes while leaving hair, cheeks, clothes, and backgrounds untouched.The visualizations support the model’s focus on facial changes rather than irrelevant regions.

3) Experiments on Natural Images:

Experiments across natural-image translation tasks show that AttentionGAN focuses changes on relevant objects while preserving background structure. Qualitative, perceptual, and distributional evaluations generally favor the proposed method, with limitations when objects are occluded or difficult to localize.

  • Horse2Zebra: AttentionGAN generates target objects while preserving the source background more effectively than competing methods in horse↔zebra translation.Compared methods often blur results or alter irrelevant background regions, whereas AttentionGAN changes the object of interest while leaving the background unchanged.
  • Limitations: A difficult occluded-horse example causes AttentionGAN to alter a wooden fence because the model attempts to translate the complete horse.The paper suggests additional object or instance masks as a possible future solution.
  • Other Natural-Image Translations: AttentionGAN better alters objects of interest without changing the background in zebra↔horse and apple↔orange translation.The method also outperforms several baselines qualitatively on these translation directions; UAIT requires an additional attention network that increases network parameters.
  • Style Transfer: AttentionGAN produces sharper and more diverse style-transfer results than CycleGAN.The comparison is reported for the style-transfer task using qualitative results.
  • Quantitative Comparison: AttentionGAN outperforms competing methods in perceptual preference, FID, and AMT evaluations across the reported translation tasks.The paper reports higher preference and AMT performance, and significantly better FID than baselines for horse-to-zebra translation.
  • Attention Masks: Attention masks are more accurate than SAT masks at attended-object boundaries, producing more photo-realistic translated boundaries.The comparison is reported for horse-to-zebra translation.
  • Attention Masks: AttentionGAN learns masks that highlight shared layout and structure in map↔aerial-photo translation, handling both shape and holistic changes.The source and target images differ in appearance but share structural organization.

V. CONCLUSION

AttentionGAN extends attention-guided generation to unpaired and multi-domain image translation. Its masks preserve input backgrounds and identify discriminative content, while experiments report more convincing details than numerous state-of-the-art methods.

  • Contribution: AttentionGAN addresses unpaired and multi-domain image-to-image translation with generators that produce attention and content masks.The masks and input images are combined to generate target images.
  • Mechanism: The built-in attention mechanism preserves input backgrounds and discovers discriminative content between source and target domains.Attention masks, content masks, and input images are fused during generation.
  • Conclusion: Extensive experiments show that AttentionGAN generates better results with more convincing details than numerous state-of-the-art methods.The conclusion summarizes results across several challenging tasks.
Loading 1911.11897v5…