PaperlayerView exact revision on arXiv

Source-linked AI summary

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, Ping Luo

arXiv:2011.00844v4cs.CV

TL;DR

The paper asks whether pretrained 2D GANs encode useful 3D structure and how to use it to recover shape from a single image. GAN2Shape iteratively exploits viewpoint and lighting variations in the GAN manifold to refine 3D shapes. The authors report reconstruction across several object categories and 3D-aware image manipulation.

  • Problem

    It is unclear whether 2D GANs implicitly capture underlying 3D structure or how that knowledge could recover shape from a single image.

    Source: §1 · p. 2

  • Method

    GAN2Shape renders ellipsoid-based pseudo samples under varied viewpoints and lighting, projects them through GAN inversion, and iteratively uses them to refine the shape.

    Source: §1 · p. 2

  • Results

    The method recovers shapes for human faces, cats, cars, and buildings, and supports 3D-aware rotation and relighting.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    The results indicate that pretrained 2D GANs contain useful knowledge of underlying 3D geometry across multiple object categories.

    Source: §5 · p. 9

    A depth-map mesh cannot represent object backs, and a simple convex prior may not capture viewpoint and lighting variation for sophisticated shapes such as horses.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.

1 INTRODUCTION

The paper asks whether 2D GANs encode enough 3D structure to reconstruct shape from a single image, and how to exploit that knowledge. GAN2Shape uses viewpoint and lighting variations in the GAN image manifold to iteratively refine shapes without 2D keypoints or 3D annotations.

1 INTRODUCTION

  • The central challenge is finding disentangled, unsupervised directions in a GAN image manifold that control viewpoint and lighting.Manually inspecting and annotating manifold samples is laborious and time-consuming.Source: §1 · p. 2
  • The method iteratively renders pseudo samples under varied viewpoints and lighting, projects them through GAN inversion, and uses them to refine the shape.The refined shape becomes the initialization for the next iteration.Source: §1 · p. 2, Figure 2 · p. 2
  • GAN2Shape uses existing GANs trained only on 2D images to accurately reconstruct 3D shapes from single unconstrained images across categories including human faces, cars, and buildings, without 2D keypoints or 3D annotations.Source: §1 · p. 2
  • The authors report photo-realistic 3D-aware rotation and relighting without external 3D models, and say the method does not rely on shape symmetry.They also report outperforming a strong 3D-shape-learning baseline on the BFM benchmark.Source: §1 · pp. 2–3

2 RELATED WORK

Prior approaches to 3D-aware generation often rely on explicit 3D representations, external models, or shape assumptions. This work instead mines a pretrained 2D GAN's image manifold to obtain viewpoint and lighting variations for shape learning and manipulation.

2 RELATED WORK

  • For 3D-aware manipulation, prior work discovers latent directions with manual annotations or external 3D face models, whereas this method targets pose and lighting without either.The authors say this allows generalization beyond human faces, including cats and cars.Source: §2 · p. 3
  • Some unsupervised monocular shape-learning methods use 3DMM or SMPL priors, 2D keypoints, category-specific templates, or symmetry assumptions.The paper notes that symmetry assumptions can miss asymmetric object features.Source: §2 · p. 3
  • Unlike methods that explicitly model 3D representations during training, this approach mines a pretrained 2D GAN's image manifold to obtain multiple views and lighting conditions.Prior 3D GANs face memory or training difficulty, while this method is described as a memory-efficient route to 3D shape and texture generation.Source: §2 · p. 3
  • The method is presented as an alternative to symmetry-based approaches and is reported to perform better than Unsup3d, capturing asymmetric object aspects.The authors identify asymmetric objects such as buildings as difficult for methods that assume symmetry.Source: §2 · pp. 3–4

3 METHODOLOGY

GAN2Shape mines viewpoint and lighting variations from a pretrained 2D GAN to infer 3D shape without explicit 3D supervision. It begins from an ellipsoid prior, projects varied pseudo samples back onto the GAN image manifold, and iteratively refines the recovered shape.

3 METHODOLOGY

  • The method uses StyleGAN2’s synthesis network as an image manifold and factorizes each image into depth, albedo, viewpoint, and lighting for rendering.The rendering process recomposes those factors through lighting and viewpoint reprojection.Source: §3 · p. 4, §3.1 · pp. 4–5

3.1 FROM 2D GANS TO 3D SHAPES

  • An ellipsoid initializes depth; randomly sampled viewpoints and lighting render pseudo samples that provide cues for exploring changes in the GAN image manifold.The ellipsoid is roughly aligned to the object using a scene-parsing model, and albedo is optimized to reconstruct the input.Source: §3.1 · p. 5
  • An encoder inverts pseudo samples into latent offsets, and the generator projects them onto its natural-image manifold while retaining similar viewpoint and lighting changes.The method uses discriminator-feature L1 distance and an L2 penalty on the offsets.Source: §3.1 · p. 5
  • With projected samples showing the same instance under varied views and lighting, networks infer sample-specific viewpoints and lighting alongside shared depth and albedo.The original image is also reconstructed to keep inferred factors from deviating from it; sampled pseudo-sample views and lighting are not reused because they may not match the projected samples.Source: §3.1 · pp. 5–6
  • The reconstruction loss decreases when depth, albedo, viewpoints, and lighting are correctly inferred, while the shape shifts toward the underlying shape and pseudo-sample distortions and shadows are mitigated.The cycle of shape initialization, projection, and shape learning is repeated; experiments use four cycles unless otherwise stated.Source: §3.1 · p. 6

3.2 DISCUSSION

  • A mapping-network-based regularizer constrains latent offsets to valid offsets, addressing the possibility that weak L2 regularization leaves projected samples with unnatural distortions.The proposed offset is F(E(Ii)) − F(0).Source: §3.2 · p. 6
  • Joint training across instances is described as improving generalization and convergence speed; it replaces absolute offsets with relative offsets E(Ii) − E(I).The relative offset is motivated by its potential to generalize better across target instances with diverse viewpoints and lighting.Source: §3.2 · p. 6

4 EXPERIMENTS

Experiments evaluate GAN2Shape on unconstrained RGB datasets for shape recovery and 3D-aware image manipulation. The method outperforms Unsup3D on BFM metrics and shows qualitative advantages across object categories, while its shape prior and mesh representation impose limits.

4 EXPERIMENTS

  • Experiments use StyleGAN2 pretrained on CelebA, BFM, cat, LSUN Car, and LSUN Church unconstrained RGB images.The cat dataset combines images from Zhang et al. (2008) and Parkhi et al. (2012).Source: §4 · p. 6

4.1 UNSUPERVISED 3D SHAPE RECONSTRUCTION

  • Across faces, cats, cars, and buildings, reconstructions capture fine details; Unsup3D misses some facial asymmetry and struggles with asymmetric objects or large viewpoint changes.Figure 4 compares reconstructed meshes, surface normals, and textured meshes; Figure 5 presents building relighting and shape comparisons.Source: §4.1 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
  • On BFM, the method outperforms other methods on both SIDE and MAD errors; without the symmetry assumption, it significantly outperforms Unsup3D.Source: Table 1 · p. 8, §4.1 · p. 8
  • Asymmetric, shifted, and weaker shape priors only slightly affect performance, but a flat shape prior performs worse because it cannot indicate viewpoint and lighting changes.Table 2 lists the tested prior variants, including asymmetric, shifted, weaker, and absent priors.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 3D-AWARE IMAGE MANIPULATION

  • Recovered shapes support rotation and relighting; 3D renders follow object structure, while their GAN-manifold projections look more photo-realistic, including for real images.The real-image example uses GAN inversion.Source: §4.2 · p. 8
  • For identity-preserving face rotation, rendered recovered 3D shapes significantly outperform HoloGAN, GANSpace, and SeFa; GAN-manifold manipulation also better preserves identity.The evaluation rotates 100 faces across yaw angles from -20 to 20 degrees and measures identity distances using ArcFace embeddings.Source: §4.2 · p. 9, Table 3 · p. 9
  • Horse shapes may be inaccurate because a simple convex prior poorly reflects their viewpoint and lighting variation, and the depth-map mesh cannot represent object backsides.Despite these limits, the method captures rough head and belly shapes and achieves reasonable relighting.Source: §4.2 · p. 9

5 CONCLUSION

The paper uses off-the-shelf 2D GANs to recover 3D object shapes, finding that they contain useful 3D knowledge across several categories. It also applies the recovered shapes to rotation and relighting.

5 CONCLUSION

  • Off-the-shelf 2D GANs contain sufficient knowledge to recover 3D shapes of faces, cats, cars, and buildings.The method explores viewpoint and lighting variations in the GAN image manifold and iteratively refines shapes from a weak convex prior.Source: §5 · p. 9
  • The method enables 3D-aware image manipulations, including object rotation and relighting.Source: §5 · p. 9

A APPENDIX

The appendix provides additional qualitative results and implementation details.

A APPENDIX

A.1 QUALITATIVE EXAMPLES

The appendix shows additional reconstruction and image-manipulation examples, including iterative refinement, results without symmetry, and challenging cases. It also examines how shape priors and GAN training-set size affect results.

A.1 QUALITATIVE EXAMPLES

  • Later iterative-training stages produce more precise reconstructed shapes.Source: §A.1 · p. 13, Figure 11 · p. 14
  • The method achieves competitive results without the symmetry assumption and reasonable results on many challenging cases.Source: §A.1 · p. 13, Figure 12 · p. 14, Figure 13 · p. 14
  • Shifted or asymmetric ellipsoid priors can be progressively refined, but large shifts and flat priors yield worse results because they fail to reveal viewpoint and lighting changes.The qualitative comparison includes original, asymmetric, shifted, weaker, and flat priors.Source: §A.1 · p. 14, Figure 14 · p. 15
  • 160k to 3k training samples causes only a marginal performance drop, while reducing the dataset to 1k makes results notably worse as GAN image quality deteriorates.The authors suggest that recent GAN data augmentation could potentially address the small-data issue.Source: §A.1 · p. 15

A.2.1 PHOTO-GEOMETRIC AUTOENCODING

The photo-geometric autoencoding model represents images through depth, albedo, viewpoint, and lighting, then recomposes them through illumination and reprojection.

A.2.1 PHOTO-GEOMETRIC AUTOENCODING

  • The lighting function Λ produces an illuminated texture from albedo, depth, and light direction.It computes surface normals from depth and combines ambient and diffuse terms; the lighting network predicts their weights and the light direction.Source: §A.2.1 · p. 15
  • The reprojection function Π maps illuminated texture to a new viewpoint using depth and the viewpoint transform.It warps pixels through a differentiable renderer; the viewpoint includes rotations and translations.Source: §A.2.1 · pp. 15–16
  • Depth values back-project image pixels into camera-space 3D points using the inverse camera matrix.The perspective camera field of view is assumed to be approximately 10 degrees.Source: §A.2.1 · p. 15

A.2.2 REGULARIZING THE LATENT OFFSET

The method regularizes latent offsets by constraining them to valid shifts produced by the GAN mapping network. This avoids a real-image inversion difficulty associated with instead predicting offsets in the original latent space.

A.2.2 REGULARIZING THE LATENT OFFSET

  • Predicting offsets in the original latent space makes inversion of real natural images harder than inversion in the intermediate W space.This difficulty would impede applying that alternative to real images rather than GAN samples.Source: §A.2.2 · p. 16
  • Constraining the latent offset to a valid mapping-network-created offset provides stronger regularization than a weak L2 penalty.The weak penalty can allow the resulting latent vector to fall outside the GAN’s intermediate latent distribution, producing unnatural distortions in projected samples.Source: §A.2.2 · p. 16
  • The regularization strength is adjustable by applying only F1 of the mapping network, with its depth d controlling strength.The paper uses d = 8 for BFM and d = 2 for other datasets, where full-network regularization can be too strong and sometimes sub-optimal.Source: §A.2.2 · p. 16

A.2.3 TRAINING DETIALS

The appendix specifies the networks, training settings, assumptions, and evaluation procedure used across the experiments. Settings vary by dataset, including image resolution, symmetry use, viewpoint sampling, and background masking.

A.2.3 TRAINING DETIALS

  • The BFM experiments use 64²-resolution images and the Wu et al. (2020) architectures for D, A, L, and V; other datasets use 128² images with added layers.The GAN encoder uses a ResNet-alike architecture; the appendix provides its architecture and those of the other networks.Source: §A.2.3 · p. 16, Table 7 · p. 17, Table 6 · p. 17
  • Joint pre-training on 200 samples is used for CelebA, Cat, and BFM to improve generalization when fine-tuning on new samples.Gradients from each sample in a randomly selected batch are averaged after back-propagation.Source: §A.2.3 · p. 17
  • Symmetry is optional and used for CelebA, Cat, and BFM unless stated otherwise; results without it are also reported and described as satisfactory.The assumption can be dropped after the first training stage.Source: §A.2.3 · pp. 17–18
  • Viewpoint sampling uses dataset-specific estimated statistics for BFM, CelebA, and Cat, while LSUN Car and Church use FFHQ statistics.Lighting directions and diffuse and ambient terms are sampled from dataset-specific uniform ranges.Source: §A.2.3 · p. 18
  • For LSUN Car and Church, PSPNet50 supplies an object mask once for the original view; other datasets do not mask backgrounds.For novel views, the mask is warped with the current shape and used to mask background in the image equations.Source: §A.2.3 · p. 18
  • Face-rotation evaluation measures ArcFace identity distances from the front view across 20 poses spanning yaw −20° to 20° and averages mean and maximum distances over 100 samples.Face angles are estimated with the viewpoint prediction network of Wu et al. (2020).Source: §A.2.3 · p. 18

Related papers

Loading 2011.00844v4…