Source-linked AI summary
Generative Image Modeling using Style and Structure Adversarial Networks
Xiaolong Wang, Abhinav Gupta
TL;DR
Existing generative image models overlook the separate structural and stylistic components of image formation. S^2-GAN addresses this by generating surface normals with Structure-GAN and images with conditional Style-GAN, reporting more realistic and interpretable generation plus unsupervised RGBD representations.
Problem
Existing generative approaches generate images end-to-end from uniform noise while overlooking the separate structure and style components of image formation.
Method
S^2-GAN factorizes generation into Structure-GAN surface-normal synthesis and Style-GAN image synthesis conditioned on surface normals, with adversarial and surface-normal constraints.
Results
The authors report more realistic and interpretable images, unsupervised RGBD representation learning, and improved realism judgments and image-quality metrics over baselines.
Takeaways & Limitations
Factoring structure from style supports controllable image rendering and applications that alter 3D structure before rendering a new image.
Takeaways & Limitations
The method assumes that sufficiently realistic generated images can be used to reconstruct surface normal maps, and initially keeps the normal-estimation FCN fixed because early generated images are poor.
Abstract
from arXiv · showhide
Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (S^2-GAN). Our S^2-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our S^2-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.
1 Introduction
S^2-GAN factorizes image generation into structure and style, addressing image formation with separate Structure-GAN and Style-GAN processes. The authors report interpretability, realistic generation, unsupervised RGBD representation learning, and rendering applications.
- 1 Introduction: Images combine scene structure, which encodes geometry, and style, which encodes texture and illumination.
- 1 Introduction: S^2-GAN uses Structure-GAN to generate a 3D structure representation and Style-GAN to generate an image conditioned on that structure and noise.
- 1 Introduction: Factoring style and structure is reported to improve realism, stability, interpretability, and unsupervised RGBD representation learning.
- 1 Introduction: Style-GAN is also presented as a learned rendering engine that can render synthetic scenes and modify underlying 3D structure before producing new images.
- 1 Introduction: The model is initialized by independently training Structure-GAN and Style-GAN on NYUv2 RGBD data, including Kinect surface normals.
2 Related Work
Prior unsupervised representation learning uses discriminative auxiliary tasks or generative models, including non-parametric methods, reconstruction models, and GAN variants. S^2-GAN builds on GANs while adding structural constraints and factorizing image generation.
- 2 Related Work: Unsupervised visual representation learning commonly uses discriminative auxiliary tasks or generative frameworks.
- 2 Related Work: Generative approaches include database matching, texture synthesis, super-resolution, auto-encoders, and restricted Boltzmann machines.
- 2 Related Work: GAN research progressed from adversarial image generation and Laplacian pyramids to DCGAN training practices and conditional generation.
- 2 Related Work: S^2-GAN combines adversarial loss with 3D surface-normal prediction loss to impose additional constraints during generative learning.
3 Background for Generative Adversarial Networks
GANs train a generator and discriminator adversarially: the generator maps uniform noise to images, while the discriminator distinguishes generated from real images. Training alternates optimization of the two networks.
- 3 Background for Generative Adversarial Networks: A GAN contains a generator G that maps uniform noise z to images and a discriminator D that classifies images as real or generated.
- 3 Background for Generative Adversarial Networks: The discriminator and generator compete, with D learning to reject generated images and G learning to fool D.
- 3 Background for Generative Adversarial Networks: GAN training alternates between optimizing D with G fixed and optimizing G with D fixed.
- 3 Background for Generative Adversarial Networks: The discriminator loss classifies real images as label 1 and generated images as label 0 using binary entropy loss.
4 Style and Structure GAN
S2-GAN factorizes image generation into Structure-GAN surface-normal generation and conditional Style-GAN image generation, then adds surface-normal supervision and joint learning to improve structural alignment.
- Structure-GAN and Style-GAN: Structure-GAN generates surface normal maps from uniform latent noise, while Style-GAN generates RGB images conditioned on normals and a second noise input.Structure-GAN outputs 72×72×3 normal maps; Style-GAN outputs 128 × 128 × 3 RGB images.
- Style-GAN: The conditional Style-GAN discriminator receives images together with surface normals, encouraging generated images to appear real and match their conditioning normals implicitly.Real RGB images and their corresponding Kinect surface normals are used as positive examples.
- Multi-task Learning with Pixel-wise Constraints: A pixel-wise surface-normal constraint is added because unconstrained Style-GAN outputs are noisy and their edges are poorly aligned with the input normal maps.The constraint is implemented through an FCN that predicts surface normals from generated images.
- Multi-task Learning with Pixel-wise Constraints: The FCN uses a 40-class quantization of surface normals and pixel-level softmax loss, with K = 128 matching the input image resolution.The FCN is trained from scratch on RGBD data with ground-truth surface normals, and its parameters are initially fixed during multi-task learning.
- Joint Learning for S2-GAN: After independent training, the networks are merged for joint learning, passing Style-GAN loss into Structure-GAN while removing the FCN constraint because generated normals replace ground-truth normals.The joint objective combines Structure-GAN adversarial loss with Style-GAN loss, using λ = 0.1 and a smaller Structure-GAN learning rate.
5 Experiments
Experiments evaluate S2-GAN qualitatively and quantitatively for image generation, then test its learned representations on scene classification and object detection.
- Qualitative Results for Image Generation: Style-GAN generates images aligned with ground-truth surface normals and realistic renderings of synthetic 3D scenes.With ground-truth normals, generated images can vary illumination, color, and texture while preserving structure.
- Qualitative Results for Image Generation: Interpolating either structure or style latent inputs produces smooth, interpretable changes in generated images.Changing structure can grow a 3D cube while fixed style remains consistent; changing style can gradually shut a room window.
- Qualitative Results for Image Generation: 71% of AMT judgments preferred S2-GAN images over DCGAN images as more realistic.The user study collected 1000 randomly generated image pairs.
- Qualitative Results for Image Generation: Nearest-neighbor retrieval finds generated images semantically related to, but stylistically and structurally different from, real neighbors.AlexNet Places Pool5 features were used to retrieve the top 7 real images for each generated query.
- Quantitative Results for Image Generation: S2-GAN is around 2% better than baselines on the maximum norm of Places-AlexNet classification outputs for generated images.The comparison uses DCGAN, DCGANv2, and DCGANv2+LAPGAN, with 10K generated images per model.
- Quantitative Results for Image Generation: At a 0.3 detection threshold, S2-GAN images produce 2.2 detections per image versus 1.72 for DCGAN.A Fast-RCNN detector fine-tuned on NYUv2 was used to measure average detections across thresholds.
- Representation Learning for Recognition Tasks: For recognition, S2-GAN representations are 8.2% better than DCGAN in SUN RGB-D scene classification and yield a 1.5% improvement over training from scratch for NYUv2 object detection.Scene-classification models were not fine-tuned on the dataset; detection used RGBD inputs with surface normals representing depth.
6 Conclusion
The paper concludes that S2-GAN factorizes image generation into style and structure, producing interpretable and more realistic images while learning unsupervised RGBD representations.
- Conclusion: S2-GAN factorizes image generation and is reported to be more interpretable and more realistic than the baselines.The model also learns RGBD representations in an unsupervised manner.
7 Supplementary Material: Generated Normals and Images from S2-GAN
Supplementary material presents output surface normals and output images generated by S2-GAN.
- Generated Normals and Images from S2-GAN: The supplementary material displays S2-GAN output normals alongside output images.The material labels repeated pairs as “Output Normals” and “Output Images.”