Source-linked AI summary
Spatial Fusion GAN for Image Synthesis
Fangneng Zhan, Hongyuan Zhu, Shijian Lu
TL;DR
Existing GAN-based synthesis often achieves realism in either geometry or appearance, limiting the realism and usefulness of generated images. SF-GAN addresses both through interconnected geometry and appearance synthesizers trained end-to-end, and its synthesized images improve scene-text recognition performance while also extending to portrait-wearing tasks.
Problem
Most existing GANs address synthesis realism in either geometry or appearance, and generated images may lack sufficient realism or annotation for training deep models.
Method
SF-GAN interconnects geometry and appearance synthesizers to transform and place foreground objects, harmonize their appearance, preserve details with guided filters, and train end-to-end.
Results
SF-GAN achieves realism in both spaces, improves scene-text recognition accuracy by up to 3% on average, and is applicable to scene-text and portrait-wearing synthesis.
Takeaways & Limitations
SF-GAN synthesizes images useful for training recognition models and can extend beyond scene text to matching hats and glasses with real portraits.
Takeaways & Limitations
End-to-end training requires sequential updating; without the geometry loss, the appearance synthesizer produces blurry foreground objects.
Abstract
from arXiv · showhide
Recent advances in generative adversarial networks (GANs) have shown great potentials in realistic image synthesis whereas most existing works address synthesis realism in either appearance space or geometry space but few in both. This paper presents an innovative Spatial Fusion GAN (SF-GAN) that combines a geometry synthesizer and an appearance synthesizer to achieve synthesis realism in both geometry and appearance spaces. The geometry synthesizer learns contextual geometries of background images and transforms and places foreground objects into the background images unanimously. The appearance synthesizer adjusts the color, brightness and styles of the foreground objects and embeds them into background images harmoniously, where a guided filter is introduced for detail preserving. The two synthesizers are inter-connected as mutual references which can be trained end-to-end without supervision. The SF-GAN has been evaluated in two tasks: (1) realistic scene text image synthesis for training better recognition models; (2) glass and hat wearing for realistic matching glasses and hats with real portraits. Qualitative and quantitative comparisons with the state-of-the-art demonstrate the superiority of the proposed SF-GAN.
1. Introduction
GAN-based image synthesis often targets either appearance or geometry realism, while SF-GAN combines both through interconnected geometry and appearance synthesizers. It is designed to generate realistic composed images and support annotated-image creation for recognition models.
- SF-GAN is evaluated for scene-text synthesis and portrait wearing of hats and glasses.Figure 1 depicts synthesized scene-text images and images combining hats or glasses with portrait backgrounds.
- GANs can generate realistic images, but existing methods commonly target either geometry or appearance realism rather than both.
- SF-GAN combines geometry and appearance synthesizers to achieve realism in both spaces concurrently.The geometry synthesizer transforms and places foreground objects using background geometry, while the appearance synthesizer adjusts color, brightness, and style.
- The two synthesizers use mutual references and can be trained end-to-end with little supervision.
- Guided filters are introduced to preserve details during appearance transfer.
2. Related Work
Prior image-composition and GAN methods address geometric or appearance realism separately and often rely on manual transformations or blending. SF-GAN learns both aspects with limited supervision while aiming to preserve details during appearance transfer.
- Image Synthesis: Image composition creates new images by placing foreground objects into existing backgrounds, with realism depending on size, orientation, and blending.
- Image Synthesis: Hand-crafted geometric transformations and manually selected blending features are prone to unnatural alignments and artifacts.
- GAN: Many GANs target appearance realism, while methods such as ST-GAN target geometric realism, leaving few approaches that address both.
- GAN: Synthesized images may be unsuitable for training deep models when they lack annotation or synthesis realism.
- Guided Filter: Guided filters use one image to guide filtering of another and are recognized for detail-preserving filtering.
- Guided Filter: Appearance transfer in image-to-image GANs can suppress edges and texture, motivating detail-preserving methods.
3. The Proposed Method
SF-GAN combines geometry and appearance synthesizers in an end-to-end system to improve realism in both spaces. Its geometry branch transforms and places foreground objects, while the appearance branch performs detail-preserving image translation using guided filters and mutual references.
- Geometry Synthesizer: The geometry synthesizer uses an STN, composition module, and discriminator to transform foreground objects before placing them into background images.The STN predicts transformations such as thin plate spline or homography from the concatenated foreground object and background image.
- Geometry Synthesizer: The geometry branch uses adapted real images as references because they retain realistic geometry while matching the composed image's appearance.This reduces interference from appearance differences during geometry training.
- Appearance Synthesizer: The appearance synthesizer uses a cycle structure with G1 and G2 to translate between composed-image and real-image domains.D1 and D2 provide adversarial discrimination, while cycle and identity losses constrain the mappings.
- Guided Filter: Guided filtering preserves background and foreground details that can be lost during appearance transfer.The translated G1 output is filtered using the composed image as guidance, with the filter producing a detail-preserving synthesized image.
- Joint Training: The two synthesizers are interconnected as local GANs with coordinated adversarial objectives for geometry and appearance realism.The geometry and appearance objectives are trained as mutually referenced components within the SF-GAN.
4. Experiments
The experiments evaluate SF-GAN on scene-text synthesis and portrait wearing, using recognition accuracy, human realism judgments, qualitative comparisons, and ablations. SF-GAN improves both geometry and appearance realism across these tasks.
- Scene Text Synthesis: SF-GAN achieves the highest recognition accuracy on most of six scene-text datasets and improves average accuracy by up to 3%.One million synthesized images are used to train recognition models.
- Ablation Analysis: Combining the geometry and appearance synthesizers improves average recognition accuracy by 6% beyond the baseline SF-GAN(BS).The ablation compares SF-GAN(GS), SF-GAN(AS), and SF-GAN(BS).
- Scene Text Synthesis: SF-GAN produces more realistic scene-text images than prior synthesis techniques in AMT user studies.Table 3 reports the percentage of images classified as real by Turkers.
- Qualitative Comparison: ST-GAN aligns geometry but produces unrealistic appearance, whereas CycleGAN adapts appearance but ignores real geometry.SF-GAN combines improved alignment, curved-text synthesis, and appearance adaptation from neighboring background texts.
- Portrait Wearing: In portrait wearing, SF-GAN produces harmonious brightness, contrast, and blending while achieving better geometric alignment than ST-GAN.The experiment matches cropped glasses or hats with CelebA face backgrounds.
5. Conclusions
The paper concludes that SF-GAN jointly achieves geometry and appearance realism for foreground-background image synthesis. Its scene-text and portrait-wearing studies demonstrate usefulness for recognition training and applicability to related tasks.
- Contribution: SF-GAN is an end-to-end trainable network that synthesizes realistic images from foreground objects and background images.It targets realism in geometry and appearance spaces concurrently.
- Applications: Scene-text experiments show that SF-GAN can synthesize useful images for training better recognition models.The conclusion identifies portrait wearing as a second demonstrated application.
- Applications: Portrait-wearing experiments indicate that SF-GAN can extend to other tasks.The paper states that full-image synthesis for training better detection models remains future work.