PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Image Modeling using Style and Structure Adversarial Networks

Xiaolong Wang, Abhinav Gupta

arXiv:1603.05631v2cs.CV

TL;DR

End-to-end image generators overlook that images combine scene structure and style. S²-GAN factorizes generation into a structure-to-surface-normal stage and a conditional style-to-image stage; the authors report more realistic images than baselines and unsupervised RGBD representation learning.

  • Problem

    Existing generative approaches overlook that images combine underlying scene structure with style, including texture and illumination.

    Source: §1 · p. 1

  • Method

    S²-GAN factorizes image generation into a Structure-GAN that produces 3D structure and a conditional Style-GAN that uses it to generate images.

    Source: §1 · p. 2

  • Results

    S²-GAN is reported to generate more realistic images than baselines and to be more interpretable.

    Source: §1 · p. 2, §6 · p. 15

  • Takeaways & Limitations

    Unsupervised RGBD representation learning with S²-GAN may be useful for robotics and graphics applications.

    Source: §1 · p. 2, §6 · p. 15

    The normal-reconstruction constraint assumes that sufficiently realistic generated images can reconstruct their surface-normal maps.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (S^2-GAN). Our S^2-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our S^2-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.

1 Introduction

S^2-GAN factors image generation into structure and style: Structure-GAN generates a surface-normal representation, and Style-GAN uses it to generate an image. The paper presents this factorization as a route to interpretable generation, unsupervised RGBD representations, and image rendering from 3D structure.

1 Introduction

  • Structure-GAN generates a surface normal map from noise, and Style-GAN conditions on that map plus noise to generate an image.The paper uses surface normals to represent structure because texture perspective distortion is more directly related to normals than to depth.Source: §1 · pp. 1–3, Figure 1 · p. 2
  • The authors argue that factoring structure and style simplifies generation, supports more realistic high-resolution images, and makes errors easier to interpret by separating structure from texture failures.They also present Style-GAN as a learned rendering engine for 3D inputs, including modified structure in an input image.Source: §1 · p. 2
  • The paper proposes S^2-GAN as a way to learn RGBD representations without supervision, with potential relevance to robotics and graphics applications.The introduction also situates generative representation learning alongside discriminative approaches using auxiliary tasks such as context prediction and temporal embedding.Source: §1 · pp. 1–2
  • S^2-GAN is trained by first learning its generators independently, then jointly; Style-GAN uses a realism discriminator and a surface-normal predictor to constrain image–structure alignment.The initial networks use more than 200K NYUv2 RGBD frames, with Kinect ground-truth normals for Structure-GAN and ground-truth normals conditioning Style-GAN.Source: §1 · pp. 2–3

2 Related Work

The related work contrasts discriminative auxiliary-task learning with generative approaches and positions S2-GAN as a GAN-based method that factors image generation into structure and style using surface-normal supervision.

2 Related Work

  • Discriminative unsupervised learning uses label-free auxiliary tasks, including predicting patch locations, video ego-motion, and physical signals.Source: §2 · p. 3
  • Generative approaches include database-matching non-parametric methods and parametric models such as auto-encoders and RBMs.Other generative models include CNNs used to generate chairs.Source: §2 · p. 3
  • Latent-space sampling makes new-image generation difficult for many reconstruction models; VAEs address this but are described as limited to simple datasets such as MNIST.VAEs were extended to condition on captions and graphics code, alongside other generative models.Source: §2 · p. 3
  • Building on GANs and DCGAN training practices, this work simplifies image generation by factoring out 3D structure and style.Earlier GAN extensions include Laplacian-pyramid generation and conditional generation.Source: §2 · pp. 3–4
  • S2-GAN combines adversarial training with 3D surface-normal prediction loss, drawing on multi-loss generative modeling and prior image factorization.The authors use NYUv2 RGBD data to factorize and learn S2-GAN.Source: §2 · p. 4

3 Background for Generative Adversarial Networks

GANs pair a generator that turns uniform-noise latent vectors into images with a discriminator that classifies images as real or generated. They train adversarially, alternating discriminator and generator updates.

3 Background for Generative Adversarial Networks

  • GANs use a generator to create images from uniform-noise latent vectors and a discriminator to distinguish generated images from real ones.The generator tries to produce images that fool the discriminator, while the discriminator learns to avoid being fooled.Source: §3 · p. 4
  • Training alternates between optimizing the discriminator with the generator fixed and optimizing the generator with the discriminator fixed.The procedure uses gradient descent with batch size M.Source: §3 · p. 4
  • The discriminator is trained to label real images 1 and generated images 0, while the generator minimizes loss to make generated images appear real.Each batch contains half real images and half generated images; the discriminator uses binary entropy loss.Source: §3 · p. 4

4 Style and Structure GAN

S²-GAN factorizes image generation into a Structure-GAN that generates surface normals and a Style-GAN that generates RGB images conditioned on those normals. Pixel-wise surface-normal supervision aligns generated images with their conditioning, and joint training connects the two GANs.

4 Style and Structure GAN

  • S²-GAN separates generation into a Structure-GAN that produces surface normals and a Style-GAN that uses normals and latent noise to generate images.Both models are trained using RGBD data, with ground-truth surface normals obtained from depth.Source: §4 · p. 5

4.1 Structure-GAN

  • The Structure-GAN maps a 100-dimensional uniform-noise vector to a 72 × 72 × 3 surface-normal map, which its discriminator distinguishes from maps derived from depth.Source: §4.1 · p. 5

4.2 Style-GAN

  • The conditional Style-GAN takes surface normals as inputs to both its generator and discriminator, implicitly encouraging images to match normals while appearing realistic.It generates 128 × 128 × 3 RGB images from a 128 × 128 × 3 normal map and a 100-dimensional latent vector.Source: §4.2 · p. 6

4.3 Multi-task Learning with Pixel-wise Constraints

  • To align generated images with their input normals, the Style-GAN uses an FCN surface-normal estimation loss as an explicit pixel-wise constraint.The FCN quantizes normals into 40 classes and is initially held fixed during multi-task learning because early generated images are poor training examples.Source: §4.3 · pp. 7–8

4.4 Joint Learning for S2-GAN

  • After independent training, joint learning passes Style-GAN gradients to the Structure-GAN so its normals support realistic RGB generation as well as realism themselves.Generated normals are bilinearly upsampled before Style-GAN input, and the FCN constraint is removed; experiments use λ = 0.1 and a smaller Structure-GAN learning rate to prevent divergence.Source: §4.4 · p. 9

5 Experiments

The experiments evaluate image generation quality and whether Style-GAN discriminator features transfer to recognition tasks, using NYUv2 and SUN RGB-D data. Results cover qualitative and quantitative image-generation tests, scene classification, and object detection.

5.1 Qualitative Results for Image Generation

  • S2-GAN generated more structured indoor scenes than DCGAN and DCGAN+LAPGAN, while LAPGAN added little qualitative improvement over DCGAN.Given noise inputs, the model generates 72 × 72 surface-normal maps and 128 × 128 RGB images.Source: §5.1 · pp. 11–12, Figure 8 · p. 11
  • Interpolating either latent input produced smooth transitions, enabling changes to scene structure while holding style fixed, or changes to style while holding structure fixed.Examples include gradually growing a cube in a room and gradually shutting a window.Source: §5.1 · p. 12, Figure 9 · p. 12
  • In a user study of 1,000 image pairs, AMT workers judged S2-GAN images more realistic than DCGAN images 71% of the time.Workers chose which image in each randomly generated pair looked more realistic.Source: §5.1 · p. 12
  • Nearest-neighbor retrievals were semantically related to generated images but differed in style and structure, supporting the novelty test’s assessment of generated images.The test used Places-pretrained AlexNet Pool5 features to retrieve real images from the training and test sets.Source: §5.1 · pp. 12–13, Figure 10 · p. 13

5.2 Quantitative Results for Image Generation

  • S2-GAN’s classification score was about 2% better than the baselines; at a 0.3 detection threshold, it produced 2.2 detections per image versus DCGAN’s 1.72.The classification metric is the mean maximum softmax probability from Places-AlexNet; detection uses Fast-RCNN fine-tuned on NYUv2.Source: §5.2 · pp. 13–14, Figure 11 · p. 14

5.3 Representation Learning for Recognition Tasks

  • On SUN RGB-D scene classification without fine-tuning, the model was 8.2% better than DCGAN and 3.7% away from Places-AlexNet.Features from the second-to-last layer were used to train an SVM; the model takes RGB images and surface normals as input.Source: §5.3 · p. 14, Figure 11 · p. 14
  • For NYUv2 RGBD object detection, the approach improved results by 1.5% over training from scratch.The detection pipeline uses surface normals to represent depth and fine-tunes the network end-to-end.Source: §5.3 · p. 15, Table 2 · p. 15

6 Conclusion

The paper presents a Style and Structure GAN that factorizes image generation. The authors report improved interpretability and realism over baselines, and unsupervised learning of RGBD representations.

6 Conclusion

  • The Style and Structure GAN factorizes the image-generation process.The conclusion identifies this factorization as the model's central contribution.Source: §6 · p. 15
  • The model is reported to be more interpretable and to generate more realistic images than the baselines.Source: §6 · p. 15
  • The method can learn RGBD representations in an unsupervised manner.Source: §6 · p. 15

Related papers

Loading 1603.05631v2…