PaperlayerView exact revision on arXiv

Source-linked AI summary

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, Sanja Fidler

arXiv:2010.09125v2cs.CVcs.LG

TL;DR

Inverse graphics methods often require 3D labels or multi-view supervision, while GAN latent codes can encode viewpoint changes without physically interpretable controls. This paper links a GAN, differentiable renderer, and inverse graphics network to train and disentangle those controls; it reports stronger reconstruction results and user preference, with lighting and some other cases remaining challenging.

  • Problem

    Inverse graphics training commonly relies on 3D labels or multi-view imagery with known cameras, while GAN latent spaces typically lack physical interpretation and independently manipulable 3D properties.

    Source: §1 · pp. 1–2

  • Method

    The method uses StyleGAN-generated multi-view images to train an inverse graphics network with a differentiable renderer, then uses that network to disentangle StyleGAN’s latent code.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    The authors report significantly better inverse-graphics results overall, user preference for their 3D predictions on Pascal3D, and domain-dependent 2D IoU performance: each model performs best on its own test set.

    Source: §1 · p. 2, §4.1 · p. 7

  • Takeaways & Limitations

    The disentangled model provides controllable 3D generation and image manipulation as a neural renderer.

    Source: §1 · p. 2, §5 · p. 9

    The lighting model does not handle advanced effects successfully, background disentanglement is partial, and faithful shapes for out-of-distribution objects remain challenging.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D "neural renderer", complementing traditional graphics renderers.

1 INTRODUCTION

Inverse graphics models often depend on 3D labels or multi-view supervision, while GANs encode viewpoint changes without physically interpretable, disentangled controls. The paper links a GAN and differentiable renderer to train an inverse graphics network and use it to disentangle the GAN’s latent space.

1 INTRODUCTION

  • Most inverse graphics models use synthetic 3D-labeled data, yet their performance often drops on real photographs because of the synthetic-to-real domain gap.Differentiable-rendering approaches also commonly rely on multi-view images with known cameras or related supervision.Source: §1 · pp. 1–2
  • GAN latent codes can produce different viewpoints of a scene, but typically lack physical interpretation and do not independently control properties such as shape and color.Source: §1 · p. 2
  • The paper uses StyleGAN to generate multi-view imagery for training an inverse graphics network with a differentiable renderer, then uses that network to disentangle StyleGAN’s latent space.The linked architecture is trained iteratively with cycle-consistency losses.Source: §1 · p. 2
  • The authors report that their approach significantly outperforms inverse graphics networks trained on existing datasets and supports controllable 3D image generation and manipulation.Source: §1 · p. 2

2 RELATED WORK

Related work spans supervised and differentiable-rendering approaches to single-image 3D reconstruction, unsupervised reconstruction, GAN-based neural rendering, and latent disentanglement. The paper distinguishes its approach by using GAN-generated multi-view data and graphics-derived knowledge to support 3D interpretation and control.

2 RELATED WORK

  • Single-image reconstruction methods commonly train on 3D labels, restricting them to synthetic datasets where performance on real imagery shows a gap.Source: §2 · p. 2
  • Differentiable renderers enable image-to-3D learning without 3D ground truth, but practical methods often still use known-camera multi-view losses or annotated keypoints.The cited real-image reconstructions are described as still failing to produce accurate results.Source: §2 · p. 2
  • Unsupervised reconstruction methods recover shape and viewpoint using constraints such as semantics, symmetry, GAN loss, or viewpoint distributions, but are typically limited to 2.5D and produce lower-quality results than supervised approaches.Source: §2 · pp. 2–3
  • The paper’s GAN-generated multi-view datasets are presented as a route to accurate 3D results and interpretable 3D neural renderers.Figure 2 illustrates the multi-view setup: columns fix a viewpoint while rows sample other latent dimensions.Source: §2 · p. 3, Figure 2 · p. 3
  • Prior GAN neural-rendering work can condition images on attributes, but the paper identifies little work on generation through interpretable 3D physics and contrasts its controls with Zhu et al. (2018).Unlike Zhu et al. (2018), the paper says it needs no 3D dataset for its geometry prior, gives texture 3D physical meaning, and controls background.Source: §2 · p. 3
  • Existing GAN disentanglement methods often lack physical interpretability; this work instead uses graphics knowledge to disentangle the latent space.Source: §2 · p. 3

3 OUR APPROACH

The approach uses StyleGAN to create multi-view training examples, trains a differentiable-rendering inverse graphics network, and maps predicted 3D properties back into StyleGAN’s latent space. Joint camera optimization and iterative fine-tuning support noisy viewpoint labels and more interpretable controls.

3.1 STYLEGAN AS SYNTHETIC DATA GENERATOR

  • The first four transformed StyleGAN layers control viewpoint, while other layers influence shape, texture, and background.Source: §3.1 · p. 4
  • Fixing StyleGAN’s viewpoint code while sampling remaining latent dimensions produces different objects from a shared camera view; the authors use selected views to build a multi-view dataset.They annotate 12 azimuth bins with coarse elevation and distance, reporting that annotation takes one minute for the dataset.Source: §3.1 · p. 4

3.2 TRAINING AN INVERSE GRAPHICS NEURAL NETWORK

  • The inverse graphics network predicts mesh shape and texture from an image, and the method omits lighting estimation because its performance was weak on realistic imagery.Source: §3.2 · p. 4
  • DIB-R renders predicted shape and texture at a camera view, and training combines masked image and perceptual losses, mask IoU, and shape regularizers.The regularizers include Laplacian, flatten, and deformation losses; deformation is encouraged to remain uniform and small.Source: §3.2 · p. 4
  • The network and viewpoint cameras are optimized jointly, allowing it to handle noisy initial camera annotations; multi-view consistency uses paired views of the same object.The method assumes images generated from the same viewpoint code share a camera, and randomly samples view pairs for efficiency.Source: §3.2 · pp. 4–5

3.3 DISENTANGLING STYLEGAN WITH THE INVERSE GRAPHICS MODEL

  • A mapping network converts viewpoint, shape, texture, and background into StyleGAN codes, assigning viewpoint to the first four layers and the other properties to the last twelve.The method keeps the inverse graphics network fixed while fine-tuning StyleGAN and the mapping network.Source: §3.3 · p. 5
  • Shape, texture, and background codes are softly combined, with normalized property masks intended to let each latent dimension be explained by one property.Source: §3.3 · p. 5
  • Training first warms up the mapping network, then fine-tunes it with StyleGAN using cycle consistency; background consistency is constrained in code space to avoid blurry images.The authors report that view, shape, and texture can be disentangled in the original model, while background remains entangled before fine-tuning.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments assess 3D reconstruction and StyleGAN-R rendering and manipulation. Reconstructions are shown for cars, horses, and birds, with additional tests on real-image editing.

4.1 3D RECONSTRUCTION RESULTS

  • Without multi-view consistency loss, predicted shape and texture are worse, especially in invisible regions.The ablation compares networks trained with and without the loss.Source: Figure 6 · p. 7, §4.1 · p. 7
  • On the Pascal3D test set, StyleGAN-trained predictions show noticeably higher quality in other views than the Pascal3D-trained baseline.The baseline looks visually good from the input view, but its rendered predictions in other views are lower quality.Source: §4.1 · p. 7, Figure 5 · p. 7
  • For re-projected 2D IOU, the StyleGAN model outperforms the Pascal3D model on the StyleGAN test set, while the Pascal3D model performs better on the Pascal test set.The authors attribute this pattern to a domain gap and note that this input-view metric does not reflect actual 3D shape or texture quality. Their StyleGAN dataset contains 10 times more images and takes about 1 minute to annotate, versus 200–350 hours for Pascal3D labeling.Source: §4 · p. 6, §4.1 · p. 7, Table 1 · p. 7
  • In an AMT study of 220 Pascal3D test images, users significantly preferred our 3D shape and texture predictions over the baseline.Users saw the input and predictions rendered in six views.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 DUAL RENDERERS

  • StyleGAN-R renders predicted mesh and texture into a desired background, while DIB-R renders only the object; StyleGAN-R preserves shape and texture with slight background content shifts.Both renderers receive the same predicted 3D properties.Source: §4.2 · p. 8, Figure 7 · p. 8

4.3 3D IMAGE MANIPULATION WITH STYLEGAN-R

  • Changing viewpoint while freezing shape, texture, and background yields meaningful results, whereas direct StyleGAN latent-code optimization produces implausible images.The comparison uses an L2 image reconstruction loss for latent-code optimization.Source: Figure 8 · p. 8, Figure 9 · p. 8, §4.3 · p. 9
  • StyleGAN-R swaps car shape, texture, and background while keeping viewpoint fixed, and also works with real input images.Texture swaps slightly modify the background, indicating that disentanglement remains incomplete.Source: Figure 10 · p. 8, Figure 11 · p. 8, §4.3 · p. 9

4.4 LIMITATIONS

  • The model recovers faithful geometry and texture but fails to predict correct lighting, and it only partly disentangles background and struggles with out-of-distribution shapes.The authors identify reflections, transparency, and shadows as lighting effects their spherical harmonic model cannot handle successfully.Source: §4.4 · p. 9

5 CONCLUSION

The paper links image synthesis and differentiable graphics renderers to improve 3D reconstruction while reducing annotation effort. Its layer analysis finds viewpoint information in early StyleGAN blocks and shape, texture, and background information in later blocks.

5 CONCLUSION

  • The linked renderers produce higher-quality 3D reconstructions with 10,000× less annotation effort than standard datasets, and support 3D neural rendering and image manipulation.The image synthesis network generates training data for inverse graphics, which in turn teaches the synthesis network physical 3D controls.Source: §5 · p. 9

A OVERVIEW

  • The appendix covers StyleGAN layer visualizations, dataset creation, camera initialization, additional inverse-graphics and user-study results, disentanglement examples, ablations, and limitations.Source: A OVERVIEW · p. 13

B STYLEGAN LAYERS VISUALIZATION

  • StyleGAN’s first two blocks control viewpoint, while car shape emerges in blocks 3–5 and finer shape and background appear in blocks 6–8.The feature maps are visualized by summing across channels and applying the learned RGB transformation from the last block.Source: B STYLEGAN LAYERS VISUALIZATION · p. 13

C OUR “STYLEGAN” DATASET

The StyleGAN dataset supplies multi-view examples across cars, horses, and birds, with varied shapes and textures. Its views are sufficient for training inverse-graphics networks, but missing overhead views and inconsistent articulation constrain reconstruction.

C OUR “STYLEGAN” DATASET

  • The selected views suffice to train accurate inverse-graphics networks, but the lack of higher-up viewpoints makes accurate prediction of object tops challenging.The missing views reflect the original StyleGAN training data, which lacked views showing, for example, car roofs or the backs of horses.Source: C OUR “STYLEGAN” DATASET · p. 13
  • Across views, car shape and texture are highly consistent, but horse and bird articulation is not perfectly preserved, complicating multi-view consistency training.Further investigation of articulated objects is left to future work.Source: C OUR “STYLEGAN” DATASET · p. 14
  • The dataset includes varied object shapes, textures, and viewpoints across car, horse, and bird classes.Examples include several car types, while rare truck shapes occur with lower probability.Source: C OUR “STYLEGAN” DATASET · p. 14, Figure C · p. 15

D CAMERA INITIALIZATION

The authors initialize cameras for StyleGAN-generated views with coarse pose bins rather than keypoint-based SfM, then optimize camera parameters during network training. The resulting model performs comparably to the keypoint-initialized model with much less annotation effort.

D CAMERA INITIALIZATION

  • The view-model uses 39, 22, and 8 pose bins for cars, horses, and birds, respectively, allowing all views to be annotated in one minute.The rough poses are initialized from selected azimuth bins and jointly optimized with network parameters during training.Source: D CAMERA INITIALIZATION · p. 16
  • Keypoint-based SfM initialization takes about 200 minutes for 39 selected car views, compared with 200–350 hours to annotate every Pascal3D object.The keypoint annotations are made on one car example; the corresponding keypoints on other synthesized examples indicate viewpoint alignment, not accurate keypoints for those cars.Source: D CAMERA INITIALIZATION · p. 16
  • Coarse view initialization achieves comparable final performance to keypoint initialization, and the camera systems converge to very similar positions after training.The average camera differences are 1.43° for rotation axes and 0.42° for rotation angles; the maximum differences are 2.95° and 1.11°, respectively.Source: D CAMERA INITIALIZATION · p. 16

E 3D INFERENCE

The StyleGAN-trained inverse-graphics model predicts better shapes and textures than the Pascal3D-trained model across the reported test imagery, especially in other rendered views. A user study evaluates shape, texture, and overall quality using randomized comparisons and reports annotator agreement.

E 3D INFERENCE

  • The StyleGAN-model produces better shape and texture predictions than the PASCAL-model on both Pascal3D test images and web images, especially in other rendered views.Source: E 3D INFERENCE · pp. 16, 18, 19
  • The model has some difficulty reconstructing the top of a horse’s back, where the training views are lacking.Source: E 3D INFERENCE · p. 20

F USER STUDY

  • The user study presents predictions in six views, randomizes model order, and separately asks users to judge shape, texture, and overall quality.It evaluates 220 Pascal3D test images with three users per image, totaling 660 votes.Source: F USER STUDY · p. 21, Figure I · p. 21
  • At least two of three annotators agree in 88.2% of shape cases, 89.2% of texture cases, and 87.2% of overall-quality cases.Source: F USER STUDY · p. 21

G STYLEGAN-R DISENTANGLEMENT

The inverse graphics network’s predictions can be mapped back to StyleGAN-R for neural rendering and image editing. Fine-tuning yields more consistent results, while lighting remains a limitation.

G STYLEGAN-R DISENTANGLEMENT

  • The inverse graphics network’s predicted shape, texture, and background can be mapped back to the latent code for StyleGAN-R synthesis.This enables rendering a new image from an input image’s inferred properties.Source: G STYLEGAN-R DISENTANGLEMENT · p. 21
  • After fine-tuning, StyleGAN-R produces more consistent results than the original StyleGAN with only its mapping network learned.The comparison includes DIB-R-rendered predictions, StyleGAN-R results, and original StyleGAN results with fixed weights.Source: G STYLEGAN-R DISENTANGLEMENT · pp. 21–22
  • The lighting prediction is weak: lit and unlit renderings are similar, with reflection and high-specular effects merged into texture.The paper attributes this limitation to its lighting model’s inability to handle advanced lighting effects.Source: G STYLEGAN-R DISENTANGLEMENT · p. 22
  • StyleGAN-R edits car size, azimuth, and elevation while preserving shape, texture, and a consistent background.The examples vary scale, azimuth, and elevation across rows.Source: G STYLEGAN-R DISENTANGLEMENT · p. 22

H ABLATION STUDIES

Ablations distinguish the contributions of multi-view consistency and perceptual losses. StyleGAN-R manipulation is also shown for birds, extending beyond cars.

H ABLATION STUDIES

  • Without multi-view consistency, texture worsens in invisible regions; without perceptual loss, textures become smoother and lose detail.The ablation identifies multi-view consistency as important for shape accuracy and perceptual loss for realistic texture.Source: H ABLATION STUDIES · p. 22
  • Real-image car edits vary size, azimuth, and elevation while preserving shape, texture, and a consistent background.Examples show changes to scale, azimuth, and elevation.Source: Figure L · p. 23

I STYLEGAN MANIPULATION

  • StyleGAN-R manipulation is presented as generalizable to another class, with bird results illustrating the claim.The bird camera controller varies azimuth, scale, and elevation while keeping the content code fixed.Source: I STYLEGAN MANIPULATION · p. 23

J FAILURE CASES

The inverse graphics network struggles with out-of-distribution shapes, producing unrepresentative reconstructions for examples such as Batmobile and Flintstone cars. The supplied bird examples also show viewpoint control and component transfer.

J FAILURE CASES

  • Out-of-distribution shapes and textures can lead to irrelevant reconstructions, including unrepresentative results for Batmobile and Flintstone cars.The paper notes that GANs may not capture distribution tails well and suggests more diverse training objects as a possible remedy.Source: J FAILURE CASES · p. 23
  • Bird viewpoint synthesis varies azimuth, scale, and elevation while keeping the content code fixed.The camera controller demonstrates manipulation in new viewpoints.Source: Figure M · p. 24
  • Bird component edits transfer one bird’s shape, another’s texture, and a third bird’s background to the others.The examples identify unchanged birds with boxes.Source: J FAILURE CASES · p. 24

K LIMITATIONS

The paper identifies limitations in separating lighting from texture and in reconstructing out-of-distribution objects. Ablations also show how removing either training loss changes predicted textures.

K LIMITATIONS

  • The spherical-harmonics lighting model fails to separate light from texture, with reflections and high-specular lighting absorbed into texture.The paper identifies this as a limitation for future work.Source: K LIMITATIONS · p. 24
  • The method tends to produce irrelevant shapes for out-of-distribution objects or textures, with examples spanning cars, birds, and horses.The paper anticipates that more diverse objects in GAN training data could address part of the issue.Source: K LIMITATIONS · p. 25
  • Removing multi-view consistency worsens texture in invisible regions, while removing perceptual loss smooths textures and removes detail.The ablation attributes these effects to the two losses’ distinct roles in training.Source: K LIMITATIONS · p. 25

Related papers

Loading 2010.09125v2…