PaperlayerView exact revision on arXiv

Source-linked AI summary

SPA-GAN: Spatial Attention GAN for Image-to-Image Translation

Hajar Emami, Majid Moradi Aliabadi, Ming Dong, Ratna Babu Chinnam

arXiv:1908.06616v3cs.CV

TL;DR

Unsupervised image-to-image translation must transfer relevant domain changes without paired examples while preserving domain-specific content. SPA-GAN transfers discriminator-derived spatial attention to the generator and adds feature-map loss, and the paper reports superior qualitative and quantitative benchmark performance with a lightweight design.

  • Problem

    Unsupervised image-to-image translation lacks paired examples and requires identifying the image regions subject to transfer while preserving other domain-specific features.

  • Method

    SPA-GAN computes spatial attention in the discriminator, feeds it to the generator, and adds generator feature-map loss during training.

  • Results

    SPA-GAN significantly outperforms other state-of-the-art image-to-image translation methods qualitatively and quantitatively across various benchmark datasets.

  • Takeaways & Limitations

    The model provides a lightweight attention-guided translation approach that requires neither additional attention networks nor supervision such as segmentation labels.

Abstract

from arXiv · show

Image-to-image translation is to learn a mapping between images from a source domain and images from a target domain. In this paper, we introduce the attention mechanism directly to the generative adversarial network (GAN) architecture and propose a novel spatial attention GAN model (SPA-GAN) for image-to-image translation tasks. SPA-GAN computes the attention in its discriminator and use it to help the generator focus more on the most discriminative regions between the source and target domains, leading to more realistic output images. We also find it helpful to introduce an additional feature map loss in SPA-GAN training to preserve domain specific features during translation. Compared with existing attention-guided GAN models, SPA-GAN is a lightweight model that does not need additional attention networks or supervision. Qualitative and quantitative comparison against state-of-the-art methods on benchmark datasets demonstrates the superior performance of SPA-GAN.

I. INTRODUCTION

SPA-GAN targets unsupervised image-to-image translation by directing the generator toward discriminative object regions identified by its discriminator. It combines this attention transfer with feature-map preservation and avoids additional attention networks or supervision.

  • Motivation: Unpaired image-to-image translation avoids expensive or impossible paired-data collection but requires identifying which image regions should be transferred.The paper illustrates this with translating oranges into apples while preserving unrelated image content.
  • SPA-GAN: SPA-GAN computes spatial attention in the discriminator and feeds the resulting maps to the generator to emphasize discriminative regions.This transfers knowledge from the discriminator back to the generator rather than using a separate attention network.
  • SPA-GAN: The model adds a modified cycle-consistency loss and generator feature-map loss to preserve domain-specific features during translation.The feature-map loss is computed from attended real and generated images in the decoder.
  • SPA-GAN: SPA-GAN is lightweight because it does not require additional attention networks or supervision such as segmentation labels during training.Earlier attention-guided approaches load extra attention networks alongside generators and discriminators, creating computational and memory limitations.
  • Results: Extensive qualitative and quantitative experiments report that SPA-GAN significantly outperforms state-of-the-art image-to-image translation methods on various benchmark datasets.The paper presents benchmark results in Section IV before concluding with the model’s reported advantages.

II. RELATED WORK

Related work develops paired and unpaired image-to-image translation methods, including latent-space and attention-based approaches. SPA-GAN extends this line by transferring discriminator-derived attention directly to the generator while retaining CycleGAN-style inverse mappings.

  • Image-to-Image Translation: Pix2pix learns conditional mappings from paired source and target images, whereas CycleGAN addresses translation without paired examples.CycleGAN learns inverse mappings between source and target domains using cycle consistency.
  • Image-to-Image Translation: UNIT assumes corresponding images from different domains share a latent representation, while DRIT separates domain-invariant content from domain-specific attributes.Other methods exploit shared distributions, common encoders, or segmentation masks for unpaired translation.
  • Attention Learning: Attention mechanisms encourage models to focus on relevant input regions and have been applied across vision and machine-learning tasks.Examples include classification, segmentation, captioning, visual question answering, and image-to-image translation.
  • Attention Learning: Prior attention-guided translation methods commonly decompose processing into an attention network that predicts regions and a transformation network that performs translation.The related-work discussion identifies additional attention networks as a characteristic design choice.
  • SPA-GAN: SPA-GAN differs by feeding discriminator-derived attention back to the generator while maintaining two inverse mappings through one generator and discriminator in each domain.This design is presented as feedback attention rather than a separate foreground-background attention module.

A. Spatial Attention Map from Discriminator

SPA-GAN uses its discriminator not only for real/fake classification but also to produce spatial maps highlighting discriminative image regions. These maps are normalized, resized, applied to the input, and passed to the generator.

  • Attention Map Construction: Given an input image, the discriminator produces a spatial attention map with the same size as the input.The map identifies regions on which the discriminator focuses when classifying the image as real or fake.
  • Attention Map Construction: The attention map sums absolute activation values across channels at each spatial location in a discriminator layer.The resulting values indicate the importance of hidden units for real/fake classification.
  • Attention Map Selection: SPA-GAN selects mid-level maps from the discriminator’s second-to-last layer because they typically correspond to discriminative object parts.Different classifier layers emphasize different features, with later layers generally focusing on whole objects.
  • Attention Transfer: The selected map is normalized and upsampled, then multiplied element-wise with the input before the attended sample enters the generator.This preprocessing gives the generator a spatially weighted input focused on discriminative parts.

B. Feature Map Loss

SPA-GAN adds a generator feature map loss to preserve domain-specific features during translation. The loss compares attended real and generated objects and is combined with adversarial and modified cycle-consistency losses.

  • Feature Map Loss: SPA-GAN adds a feature map loss to encourage generators to obtain domain-specific features during unsupervised image translation.The loss compares high-level abstractions of real and generated objects during decoding.
  • Feature Map Loss: The feature map loss compares feature maps of attended source samples with attended generated samples in the inverse mapping.The analogous loss is defined for the reverse mapping, and the total feature map loss combines both directions.
  • Feature Map Loss: Feature map loss is computed in the first decoder layer and added to the generator objective to preserve domain-specific features.The first decoder layer is used for the loss described in the method.
  • Feature Map Loss: The loss uses the L1 norm because it is widely used in image-to-image translation and can produce less blurry results.The paper states that its experiments show the feature map loss helps generate more realistic objects by maintaining domain-specific features.
  • Full Objective: The full SPA-GAN objective combines adversarial loss, modified cycle-consistency loss, and generator feature map loss.The relative importance of the cycle and feature map terms is controlled by λ_cyc and λ_fm.

IV. EXPERIMENTS

The experiments include ablation studies and comparisons with current state-of-the-art image-to-image translation methods. Evaluation is qualitative, quantitative, and based on user studies.

  • Experimental Design: The experimental section first analyzes the effect of each SPA-GAN component through ablation studies.
  • Experimental Design: The study then compares SPA-GAN with current state-of-the-art methods on benchmark datasets.The comparisons use qualitative, quantitative, and user-study evaluations.

A. Datasets and Experimental Setups

SPA-GAN is evaluated on object translation datasets containing challenging variations in object scale, with experiments using fixed optimization and training settings. The tasks translate one object type into another.

  • Datasets: The evaluation uses Horse ↔ Zebra, Apple ↔ Orange, and Lion ↔ Tiger datasets for object-level image-to-image translation.The Lion ↔ Tiger dataset contains 2,086 tiger images and 1,795 lion images.
  • Datasets: These datasets include objects at different scales and target translating a particular source object type into another target object type.Examples include translating oranges into apples.
  • Experimental Setup: All experiments use Adam with batch size 1 and an initial learning rate of 0.0002.The paper uses the architecture from and a least-squares loss.
  • Experimental Setup: The cycle-consistency weight is set to λ_cyc = 10, while the feature map loss weight is empirically set to λ_fm = 1.The paper reports trying λ_fm values of 1, 3, and 5, with 1 giving the best results.

B. Evaluation Metrics

The evaluation uses KID and classification accuracy for quantitative comparison, alongside ablations and training-curve analysis. The reported ablations examine attention transfer, feature map loss placement, and attention aggregation.

  • Evaluation Metrics: KID and classification accuracy are the two metrics used for quantitative comparison with state-of-the-art methods.KID measures distributional similarity using Inception representations, while classification accuracy reports top-1 performance on generated images.
  • Evaluation Metrics: Lower KID indicates higher visual similarity between generated and real images, whereas higher classification accuracy is better.KID has an unbiased estimator and does not assume a specific activation-distribution form.
  • Ablation Study: Removing attention transfer while retaining feature map loss performs slightly better than CycleGAN but substantially worse than attention-based SPA-GAN variants.The ablation indicates that feature map loss works better when computed on spatially attended discriminative regions.
  • Ablation Study: Feature map loss performs best in the first decoder layer, while encoder and later decoder placements yield higher KID and lower classification accuracy.The first decoder layer is associated with high-level, abstract, and discriminative semantics.
  • Ablation Study: Sum-based attention outperforms maximum-based attention in the reported ablation, with maximum-based attention producing higher KID and lower classification accuracy.The subsequent experiments therefore use sum-based attention.
  • Training Behavior: SPA-GAN shows lower generator and discriminator losses and milder oscillation than CycleGAN during apple →orange training.The training curves indicate better convergence for SPA-GAN in this task.

D. Qualitative Results

SPA-GAN focuses attention on discriminative object regions and produces more complete, realistic translations than competing approaches across the illustrated datasets.

  • SPA-GAN changes both object shape and texture on Apple↔Orange, while competing methods mainly change object color.The comparison attributes this robustness to localizing object parts and incorporating attention into the generative network.
  • SPA-GAN generates more realistic images than the compared approaches in the illustrated benchmark results.The qualitative comparisons cover Apple↔Orange, Zebra↔Horse, and Tiger↔Lion datasets.
  • On Zebra↔Horse and Tiger↔Lion, SPA-GAN preserves backgrounds and translates object parts more completely than methods that alter backgrounds or miss parts.Other methods either introduce target patterns into backgrounds, omit animal heads or bodies, or retain source-domain patterns.
  • SPA-GAN attention maps emphasize discriminative regions, including zebra patterns and orange boundaries, more precisely than AGGAN maps.AGGAN attends to disconnected zebra regions or whole oranges, whereas SPA-GAN highlights the relevant patterns, boundaries, and top portions.

E. Quantitative Comparison

SPA-GAN is evaluated with KID under target-only and combined-domain protocols, achieving the lowest target-only scores across translation tasks and stronger classification accuracy than competing methods.

  • Target-only and combined-domain KID are reported separately to evaluate SPA-GAN and state-of-the-art methods under both protocols.The authors distinguish target-only evaluation from averaging generated samples against both source and target domains.
  • SPA-GAN achieves the lowest target-only KID scores in all translation tasks.Target-only KID is computed against the real target domain, with lower values indicating better performance.
  • 5.32 is AGGAN’s KID value for Orange→Apple under the combined source-and-target protocol, while SPA-GAN has the smallest value under target-only evaluation.The authors argue that AGGAN’s combined-domain score reflects greater similarity to real oranges, whereas SPA-GAN’s target-only score reflects more realistic apples.
  • SPA-GAN-generated images outperform competing methods in top-1 classification accuracy across the evaluated translation datasets.The classifier predicts whether generated images belong to the target domain, with higher accuracy preferred.

F. User Study Evaluation

SPA-GAN was evaluated in a human perceptual study across three bidirectional translation tasks and in holistic translation settings. It received the strongest user-study performance, while attention maps focused on task-relevant regions such as seasonal scenery and facial features.

  • User Study: The user study awarded one point to the winning method for each image, with tied winners receiving 0.5 points.Ten participants selected the most realistic image from outputs generated by CycleGAN, AGGAN, and SPA-GAN.
  • User Study: SPA-GAN performed best in the user study across apple ↔orange, horse ↔zebra, and lion ↔tiger translation tasks.The study used 100 test images per task and compared SPA-GAN with CycleGAN and AGGAN.
  • Holistic Translation: On Winter ↔Summer translation, discriminator attention concentrated on ground and trees with different seasonal colors.These results concern holistic translation without a specific object type.
  • Holistic Translation: On Facescrub gender conversion, attention maps showed higher activations around facial regions including the eyes, nose, and lips.The evaluation also included holistic translation datasets such as GTA ↔Cityscapes.
  • Conclusion: The paper concludes that SPA-GAN achieves superior qualitative and quantitative performance over current state-of-the-art methods while remaining lightweight.The model computes spatial attention in the discriminator and transfers it to the generator.

Appendix

Appendix examples show SPA-GAN handling object shape, foreground, background, and holistic facial translation more reliably than several comparison methods. Other methods often altered backgrounds, retained source patterns, or produced incomplete or unnatural objects.

  • Apple ↔Orange: SPA-GAN was more robust to apple–orange shape changes than DualGAN, UNIT, MUNIT, DRIT, CycleGAN, Attention-GAN, and AGGAN.Several comparison methods either altered the background or changed only object color.
  • Lion ↔Tiger: In lion →tiger translation, SPA-GAN more successfully generated tiger patterns than the other methods while avoiding their background alterations.AGGAN generated tiger patterns in the background in rows 1 and 2.
  • Lion ↔Tiger: In tiger →lion translation, other methods retained tiger patterns or altered the input background, whereas SPA-GAN produced more realistic results.Background alterations were reported for DRIT, MUNIT, UNIT, and DualGAN.
  • Horse ↔Zebra: For horse →zebra translation, several methods missed object parts, generated horizontal skin patterns, or mixed source and target object content.These issues were reported for CycleGAN, Attention-GAN, and AGGAN.
  • Horse ↔Zebra: For zebra →horse translation, several methods altered backgrounds, missed object parts, or retained zebra patterns, while SPA-GAN detected the zebra foreground in one example.In that example, competing methods changed fence content instead of translating the zebra.
  • Holistic Translation: Facescrub gender conversion used attention maps with higher activation around facial regions such as the eyes, nose, and lips.The figure presents input images, attention maps, and translated images.
Loading 1908.06616v3…