Source-linked AI summary
DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)
Shuang Ma, Jianlong Fu, Chang Wen Chen, Tao Mei
TL;DR
Unsupervised image translation struggles to find instance-level correspondences from unpaired domains, since set-level constraints can produce artifacts and mode collapse. DA-GAN learns attention-based structured latent representations and combines instance- and set-level constraints; the paper reports visually appealing results and broad task applicability.
Problem
Unpaired image translation lacks instance-level correspondences, while set-level constraints can produce semantic or geometric artifacts and mode collapse.
Method
DA-GAN jointly learns a Deep Attention Encoder that projects samples into a structured latent space, using compound and multi-adversarial losses for instance- and set-level constraints.
Results
DA-GAN achieves visually appealing results across a large variety of translating tasks, with reported accuracy of 94.6 % for SVHN-to-MNIST translation versus 90.53 % for a previous state-of-the-art method.
Takeaways & Limitations
Attention-based instance translation broadens unsupervised domain translation beyond set-level matching to applications including pose morphing and domain adaptation.
Abstract
from arXiv · showhide
Unsupervised image translation, which aims in translating two independent sets of images, is challenging in discovering the correct correspondences without paired data. Existing works build upon Generative Adversarial Network (GAN) such that the distribution of the translated images are indistinguishable from the distribution of the target set. However, such set-level constraints cannot learn the instance-level correspondences (e.g. aligned semantic parts in object configuration task). This limitation often results in false positives (e.g. geometric or semantic artifacts), and further leads to mode collapse problem. To address the above issues, we propose a novel framework for instance-level image translation by Deep Attention GAN (DA-GAN). Such a design enables DA-GAN to decompose the task of translating samples from two sets into translating instances in a highly-structured latent space. Specifically, we jointly learn a deep attention encoder, and the instancelevel correspondences could be consequently discovered through attending on the learned instance pairs. Therefore, the constraints could be exploited on both set-level and instance-level. Comparisons against several state-ofthe- arts demonstrate the superiority of our approach, and the broad application capability, e.g, pose morphing, data augmentation, etc., pushes the margin of domain translation problem.
1. Introduction
Unsupervised image translation must discover meaningful instance-level correspondences from unpaired domains, because set-level distribution matching can miss semantic and geometric structure. DA-GAN addresses this by combining attention-based instance translation with set-level constraints and multi-adversarial training.
- Unpaired image translation is difficult because independent image sets lack pairing relationships needed to discover correct correspondences.
- Set-level GAN constraints can miss instance-level correspondences, producing semantic artifacts, inadequate geometry changes, and missing fine-grained details.
- DA-GAN is presented as broadly applicable and produces visually appealing results across a large variety of translation tasks.
- DA-GAN decomposes translation into instance-level operations in a highly structured latent space learned with a Deep Attention Encoder.
- A compound loss combines consistency, symmetry, and multi-adversarial terms to exploit both instance-level and set-level constraints.
- Multi-adversarial training encourages fair probability mass across target modes to mitigate mode collapse and improve coverage of target-domain modes.
2. Related Work
Prior image-translation work includes conditional GANs, shared representations, cycle consistency, and style transfer. DA-GAN differs by translating between image collections while using attention-based instance representations to discover correspondences across domains.
- Conditional GANs have been used for mappings such as photographs from sketches, attributes, and semantic layouts.
- Domain transformation and weight-sharing methods learn mappings or common representations across domains, including small-resolution faces and digits.
- DA-GAN projects source and target samples into a shared structured latent space where corresponding instance-level representations can be found.
- Cycle-consistency methods impose a constraint between domains for unsupervised image translation.
- Style transfer combines one image’s content with another image’s style, whereas domain translation learns mappings between two image collections.
3. Approach
DA-GAN learns image translation through attention-based instance representations in a shared latent space, while combining consistency, symmetry, and adversarial constraints to preserve instance structure and target-domain modes.
- DA-GAN maps source and target samples through a Deep Attention Encoder, generator, and two discriminators.
- 3.1. Deep Attention Encoder: The Deep Attention Encoder predicts attention-region locations from feature maps and uses masks to extract attended regions for latent representations.The predicted region centers define attention areas, and cropping is implemented by element-wise multiplication with the input.
- 3.1. Deep Attention Encoder: A sigmoid-based attention mask makes the cropping operation differentiable, while geometric regularization encourages the encoder output to match image labels.The sigmoid approaches a rectangular mask as its slope parameter increases.
- 3.2. Instance-Level Image Translation: Consistency and symmetry losses constrain source translations in the shared latent space and encourage deterministic mapping of target samples back toward themselves.The symmetry loss is described as an auto-encoder-type loss on target-domain samples.
- 3.3. Set-Level Image Translation: The first discriminator makes translated source samples resemble the target distribution, while a second discriminator penalizes missing target modes through reconstruction of target samples.The multi-adversarial procedure encourages probability mass to be distributed across different modes.
- The full objective combines adversarial, consistency, and symmetry terms, addressing both instance-level correspondence and set-level mode coverage.The consistency and symmetry terms are weighted by α and β in the stated objective.
F max D1,D2 L(F, D1, D2) (10)
The generator uses residual blocks to transform instance representations into images, while the discriminator downsamples generated images to produce a decision score.
- The generator concatenates instance-level representations across channels, processes them with residual blocks, and applies up-sampling layers to generate the translated image.
- The discriminator passes the generated image through down-sampling blocks and a one-node fully connected layer to produce a decision score.
4. Experiments
Experiments evaluate DA-GAN across domain adaptation, text-to-image synthesis, object configuration, pose morphing, data augmentation, and additional translation tasks. Results report strong quantitative performance, broader mode coverage, and improved handling of semantic and geometric changes.
- Text-to-Image Synthesis: DA-GAN generated more diverse samples with larger coverage than StackGAN in a 10-class CUB-200-2011 mode-collapse experiment.The experiment generated about 600 images from bird text descriptions.
- Domain Adaptation: 94.6% accuracy was achieved on SVHN-to-MNIST translation, exceeding the previous state-of-the-art result of 90.53%.
- Text-to-Image Synthesis: DA-GAN achieved 93.60% accuracy in text-to-image synthesis, while component variants reached 94.3% with DAE and 90.2% without DAE.
- Object Configuration: DA-GAN translated birds across large spatial and geometry differences, whereas VAT failed on limited correspondences and CycleGAN failed to change bird geometries.
- Pose Morphing and Data Augmentation: DA-GAN morphed bird poses across large geometry gaps and used about 10K morphed images for fine-grained classification data augmentation.A pre-trained VGG showed improvement on the augmented-data classification task.
- More Applications: Additional applications include human-face-to-animation translation and skeleton-to-cartoon synthesis, with DA-GAN preserving identity in the former task.
5. Conclusion
DA-GAN addresses unsupervised image translation by finding instance-level correspondences in a structured latent space through attention integrated into GANs. The authors report improved translation performance and note that some failures arise from incorrect attention learned under weak supervision.
- DA-GAN decomposes translation between two image sets into instance-level translation in a highly structured latent space.
- Integrating attention into GANs enables the model to find instance-level correspondences during unsupervised image translation.
- Some failure cases result from incorrect attention because weakly supervised instance learning can differ substantially from fully supervised attention.
Implementation Details
The experiments configure attention regions and instance granularity differently across tasks, covering parts or whole objects. Results include diverse, identity-preserving face translations, direct 256 × 256 text-to-image generation, and applications to paired skeleton-to-cartoon translation.
- Task-specific settings: Attention settings vary by task: CUB-200-2011 uses four bird parts, while colorization and domain adaptation attend to whole objects.
- Text-to-image synthesis: DA-GAN generates diverse 256 × 256 images directly from unpaired text-image data, whereas StackGAN requires two stages for that resolution.
- Face translation: DA-GAN generates diverse face translations while preserving identity attributes such as big round eyes and dark brown hair.
- Skeleton-to-cartoon translation: On skeleton-to-cartoon translation, DA-GAN learns from skeletons generated from approximately 1,200 cartoon images collected from about 20 figures.
- Applications: The figures present results for fine-grained translation, text-to-image synthesis, pose morphing, face translation, architectural labels-to-photos, and image colorization.