Source-linked AI summary
InstaGAN: Instance-aware Image-to-Image Translation
Sangwoo Mo, Minsu Cho, Jinwoo Shin
TL;DR
Prior image-to-image translation methods often struggle with multiple instances undergoing substantial shape changes. InstaGAN jointly translates images and instance attributes using permutation-invariant processing, context preservation, and sequential mini-batches. Across challenging datasets and tasks, it reports effective multi-instance transfiguration, including improved classification accuracy over CycleGAN and successful context preservation.
Problem
Prior image-to-image translation methods often struggle when images contain multiple target instances and the task requires substantial shape changes.
Method
InstaGAN augments images with sets of instance attributes and combines a permutation-invariant architecture, context preserving loss, and sequential mini-batch translation.
Results
InstaGAN produced successful multi-instance transfiguration across datasets, outperforming CycleGAN in all classification experiments, including 23.2% versus 8.5% accuracy for pants→shorts.
Takeaways & Limitations
Instance information can improve shape-transforming image-to-image translation while preserving background context and enabling control over which instances are translated.
Takeaways & Limitations
Sequential processing can require a trade-off: too many sequential steps may hurt the model’s permutation-invariance property.
Abstract
from arXiv · showhide
Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan
1 INTRODUCTION
Unsupervised image-to-image translation remains challenging when images contain multiple target instances and require substantial shape changes. InstaGAN addresses these cases by incorporating instance information, preserving context, and handling multi-instance translation efficiently.
- Motivation: Unsupervised image-to-image translation has progressed through GAN-based methods but remains challenging for multi-instance, shape-changing transfiguration.The introduction frames cross-domain generation as broad and highlights unsupervised image-to-image translation as a prominent GAN-based task.
- Approach: InstaGAN combines an instance-augmented architecture, context preserving loss, and sequential mini-batch inference/training.The architecture translates images with instance attributes, the loss preserves background context, and sequential processing supports many instances under limited memory.
- Approach: The architecture translates an arbitrary number of instance attributes while remaining permutation-invariant to their ordering.
- Approach: The context preserving loss focuses translation on target instances and encourages identity behavior outside them.
- Positioning: InstaGAN targets multi-instance transfiguration more directly than prior methods, which often handled single instances or reported limited performance on multi-instance cases.The paper emphasizes harmony between transformed instances and naturally rendered backgrounds, including shape transformation rather than merely attaching backgrounds to cropped instances.
2 INSTAGAN: INSTANCE-AWARE IMAGE-TO-IMAGE TRANSLATION
InstaGAN augments unsupervised image-to-image translation with instance attributes, designing permutation-aware networks, context preservation, and sequential mini-batch processing for multi-instance transfiguration.
- Instance-aware formulation: InstaGAN learns joint mappings between attribute-augmented spaces, translating an image together with its instance attributes such as segmentation masks.The approach uses instance information to disentangle different instances and support accurate, detailed translation.
- Instance-aware architecture: The generator is permutation equivariant and the discriminator permutation invariant, so instance ordering does not determine the translated image or discriminator output.Set features are aggregated by summation, then combined with identity mappings for equivariant processing.
- Instance-aware architecture: The architecture jointly encodes images and instance attributes so generated images and translated masks remain spatially consistent.Separate encodings could produce mismatched images and masks, whereas joint encoding lets the generator depict instances in areas consistent with their segmentation masks.
- Training losses: The context preserving loss applies a weighted pixel-wise ℓ1 penalty that preserves pixels belonging to background in both original and translated images.Its stated purpose is to translate target instances while keeping surrounding context, particularly when domains have different backgrounds after shape changes.
- Sequential mini-batch translation: Sequential mini-batch inference and training translates subsets of instances without increasing GPU memory with the total instance count.The method addresses the linear memory growth of processing all instances together and is intended to improve generalization to images containing many instances.
- Sequential mini-batch translation: Sequential training can improve generalization and translation quality, but too many sequential steps may hurt permutation invariance.The authors recommend increasing subset size as far as GPU memory allows.
3 EXPERIMENTAL RESULTS
Across several datasets, InstaGAN produces more plausible multi-instance translations than CycleGAN, preserves context, supports instance-level control, and benefits from sequential translation during training and inference.
- Image-to-image translation results: InstaGAN generates reasonable target-instance shapes while preserving original context and instance layout, whereas CycleGAN often fails.On sheep↔giraffe, InstaGAN preserves looking direction after translation while CycleGAN often loses the background.
- Image-to-image translation results: Conditioning on different input masks lets InstaGAN control which instances are translated, unlike CycleGAN.The method also reports good results with predicted segmentation masks during inference.
- Image-to-image translation results: 23.2% accuracy for pants→shorts is achieved by InstaGAN, compared with 8.5% for CycleGAN.The classification score is the ratio of images predicted as the target class by a pretrained classifier.
- Ablation study: The InstaGAN architecture keeps instances separate and disentangled, while CycleGAN+Seg often merges instances or fails to translate some.CycleGAN+Seg translates the union of all masks at once; InstaGAN translates the set of instance masks.
- Ablation study: The context preserving loss improves background preservation and translation quality by regularizing the mapping.The ablation attributes better background preservation and improved translation results to Lctx.
- Ablation study: Sequential translation improves generalization for many instances and translation quality for few instances through data augmentation.Using sequential schemes for both training and inference avoids blurry results and poor multi-instance generalization caused by mismatched schemes.
4 CONCLUSION
The paper proposes InstaGAN, which incorporates sets of instance attributes into image-to-image translation. Experiments show successful multi-instance transfiguration on challenging and new tasks.
- Conclusion: InstaGAN incorporates sets of instance attributes into image-to-image translation.The approach is evaluated on challenging multi-instance transfiguration tasks, including jeans-to-skirt translation in fashion images.
- Conclusion: Experiments show successful translation on challenging multi-instance transfiguration tasks, including new tasks such as jeans to skirt.The authors identify applying set-structured side information to other cross-domain generation tasks as future research.
A ARCHITECTURE DETAILS
The architecture uses CycleGAN components as building blocks, including ResNet generators and PatchGAN discriminators, with encoder and generator blocks arranged from these components.
- Architecture details: InstaGAN adopts ResNet 9-block generators and PatchGAN discriminators from CycleGAN as architectural building blocks.The ResNet generator contains downsampling, residual, and upsampling blocks.
- Architecture details: Downsampling and residual blocks are used for encoders, while upsampling blocks are used for generators.These components organize the encoder-generator architecture used in the proposed model.
B TRAINING DETAILS
Training uses fixed loss weights, Adam optimization, parallel GPUs, and a learning-rate schedule that holds the rate constant before linear decay.
- Training details: The loss weights are set to λcyc = 10, λidt = 10, and λctx = 10 for all experiments.These weights are used in the reported loss function.
- Training details: Networks are trained from scratch with Adam, batch size 4, and four GPUs in parallel.The generator learning rate is 0.0002 and the discriminator learning rate is 0.0001.
- Training details: The learning rate is held constant for 100 epochs and linearly decayed to zero over the next 100 epochs.This schedule is used for the multi-human parsing experiments.
C TREND OF TRANSLATION RESULTS
As training progresses, both translated images and masks smoothly adapt to target instances, with gradual removal of unwanted leg regions and construction of skirt shapes.
- Both image and mask translations smoothly adapt to target instances as epochs increase.The reported trend includes gradual changes in both outputs rather than abrupt transformations.
- Leg remnants gradually disappear while skirts progressively form triangular shapes.
D QUANTITATIVE RESULTS
Quantitative evaluation uses classification scores on CCP and COCO, including a masked score for COCO to address false positives in target-instance detection.
- Classification scores were evaluated on both CCP and COCO datasets.
- COCO evaluation uses a masked classification score because the classifier can produce false positives about target-instance placement.Inputs are masked by corresponding segmentations for this measurement.
- CycleGAN and InstaGAN achieve comparable naïve classification scores, while InstaGAN outperforms CycleGAN on the masked classification score.The masked score is intended to reduce the false-positive problem.
E MORE TRANSLATION RESULTS
Additional experiments examine generalization and comparisons across MHP and COCO translations. Against CycleGAN+Seg and a crop-and-attach baseline, the instance-aware method avoids several multi-instance errors and produces contextually better-fitting results.
- E MORE TRANSLATION RESULTS: Additional translation results cover MHP pants↔skirt and COCO animal transformations in both directions.The COCO examples include sheep↔giraffe, zebra↔elephant, bird↔zebra, and horse↔car.
- E MORE TRANSLATION RESULTS: CycleGAN+Seg often fails to translate instances, merges multiple instances, or generates multiple instances from one instance.These comparison results are reported for MHP and COCO tasks.
- E MORE TRANSLATION RESULTS: InstaGAN avoids the listed instance errors because it is instance-aware.The passage contrasts its behavior with CycleGAN+Seg in the reported comparisons.
- E MORE TRANSLATION RESULTS: Translated target masks are often much different from their nearest training neighbors, indicating the model does not simply memorize training masks.Nearest neighbors are measured using L2-norm.
- E MORE TRANSLATION RESULTS: The crop-and-attach baseline produces instances that fit the original contexts less well because pixel-space distance does not capture semantics.
I VIDEO TRANSLATION RESULTS
Video translation results use predicted segmentations for each frame and show temporally coherent outputs across successive frames, despite no explicit temporal regularization.
- The video experiment uses a predicted segmentation for each frame.The segmentation is generated by a pix2pix model.
- InstaGAN produces temporally coherent video translations without explicit temporal regularization.The results are presented on successive frames for visualization.
J RECONSTRUCTION RESULTS
Figure 30 reports translation and reconstruction results for the method, showing good performance on both and indicating that translated results preserve the original context well.
- The method shows good reconstruction results in Figure 30.
- The method also shows good translation results in Figure 30.
- The combination of good translation and reconstruction results indicates that translated results preserve the original context well.