PaperlayerView exact revision on arXiv

Source-linked AI summary

HoloGAN: Unsupervised learning of 3D representations from natural images

Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang

arXiv:1904.01326v2cs.CV

TL;DR

Implicit representations favored by generative models are not designed for explicit 3D transformations, leading to visual artifacts and blurriness in view manipulation. HoloGAN learns explicit 3D features from unlabelled natural images, transforms them to a target pose, and projects and renders them as images. HoloGAN disentangles pose, shape, and appearance while generating images with similar or higher visual quality than 2D-based GANs.

  • Problem

    Implicit representations favored by generative models are not designed for explicit 3D transformations, leading to visual artifacts and blurriness in view manipulation.

    Source: §1 · p. 2

  • Method

    HoloGAN learns explicit 3D features from unlabelled natural images, transforms them to a target pose, and projects and renders them as images.

    Source: §1 · p. 2

  • Results

    HoloGAN disentangles pose, shape, and appearance while generating images with similar or higher visual quality than 2D-based GANs.

    Source: §5.2 · p. 6, §5.4 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The learned representation provides direct control over pose, shape, and appearance while retaining image fidelity across datasets of varying complexity.

    Source: §1 · p. 2, §6 · p. 8

    Performance depends on the variety and distribution of training poses; limited elevation variation in CelebA and Cats impairs elevation recovery.

    Source: §6 · p. 8

Abstract

from arXiv · show

We propose a novel generative adversarial network (GAN) for the task of unsupervised learning of 3D representations from natural images. Most generative models rely on 2D kernels to generate images and make few assumptions about the 3D world. These models therefore tend to create blurry images or artefacts in tasks that require a strong 3D understanding, such as novel-view synthesis. HoloGAN instead learns a 3D representation of the world, and to render this representation in a realistic manner. Unlike other GANs, HoloGAN provides explicit control over the pose of generated objects through rigid-body transformations of the learnt 3D features. Our experiments show that using explicit 3D features enables HoloGAN to disentangle 3D pose and identity, which is further decomposed into shape and appearance, while still being able to generate images with similar or higher visual quality than other generative models. HoloGAN can be trained end-to-end from unlabelled 2D images only. Particularly, we do not require pose labels, 3D shapes, or multiple views of the same objects. This shows that HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner.

1. Introduction

Existing generative models offer limited attribute control and struggle with 3D-dependent view synthesis, while some alternatives require labels or costly 3D data. HoloGAN learns manipulable 3D features directly from unlabelled images.

1. Introduction

  • Understanding 3D objects and 2D images matters across computer vision and graphics, including novel-view synthesis and relighting.Source: §1 · p. 1
  • 2D-kernel models struggle with novel-view synthesis, while alternatives may require labelled multi-view data or costly 3D ground truth.Source: §1 · pp. 1–2
  • HoloGAN learns an explicit 3D representation from natural images, then transforms it, projects it to 2D, and renders the image.Source: §1 · p. 2
  • The design targets disentangled pose, shape, and appearance, enabling direct manipulation of view and identity-related factors.Source: §1 · p. 2

2. Related work

HoloGAN sits at the intersection of GANs, structure-aware image synthesis, and disentangled representation learning. The related work contrasts its explicit control and 3D-aware design with existing approaches that rely on labels, supervision, or restricted settings.

2.1. Generative adversarial networks

  • GAN research emphasizes stability and fidelity, while StyleGAN separates image variation but does not provide explicit control over attributes.Source: §2.1 · p. 2

2.2. 3D-aware neural image synthesis

  • Prior 3D-aware methods may require 3D shapes, multiple views, pose information, or paired supervision to synthesize novel views.Source: §2.2 · p. 2
  • Its generator combines 3D convolutions, rigid transformation, projection, and 2D convolutions, with z controlling AdaIN from a learned constant tensor.Source: Figure 3 · p. 3
  • HoloGAN learns rendering from scratch for complex natural images, unlike VON’s explicit-3D, simple-background setting and Pix2Scene’s hand-crafted renderer.Source: §2.2 · p. 3

2.3. Disentangled representation learning

  • Unlike label- or set-supervised disentanglement, HoloGAN redesigns the generator to separate pose, shape, and appearance and enable editing on complex natural images.Source: §2.3 · p. 3

3. Method

HoloGAN generates images by transforming learned 3D features into a target pose, projecting them to 2D, and rendering them. Its design separates pose from identity, and further separates identity into shape and appearance.

3. Method

  • HoloGAN learns a 3D representation, applies rigid-body pose transformations, then projects and renders it, without neural-renderer pretraining or paired 3D-shape/2D-image data.The paper contrasts this view-dependent mapping with GANs that map noise directly to 2D features.Source: §3 · p. 3

3.2. Learning with view-dependent mappings

  • Random pose perturbations are intended to produce view-compatible 3D features; the setup assumes one global pose per image and leaves supported translation unused.The authors say the single-pose assumption also works for images containing multiple objects.Source: §3 · p. 3, §3.2 · p. 4

3.1. Learning 3D representations

  • A learned constant tensor supplies the 3D template, while noise z controls AdaIN parameters that adjust feature statistics across layers.The authors describe this as transforming a template to match feature means and standard deviations associated with image style.Source: §3.1 · p. 4
  • HoloGAN separates pose, shape, and appearance through 3D transformations, 3D features, and 2D features, respectively.The paper reports competitive image quality at 128×128 using 3D features up to 16×16×16×64, compared with VON’s 128×128×128×1 voxel grid.Source: §3.1 · p. 4

3.2. Learning with view-dependent mappings

  • The projection unit maps 4D 3D features to 2D features, using 3D convolutions to form a perspective frustum and an MLP to learn occlusion.The model uses this learned perspective handling because camera intrinsics are unknown.Source: §3.2 · p. 4

3.3. Loss functions

  • For 128×128 images, an identity regularizer encourages the generated image’s reconstructed latent vector to match z, helping preserve identity across pose changes.Encoder F shares most discriminator convolution layers and adds a fully connected layer to predict the reconstructed latent vector.Source: §3.3 · p. 4
  • Multi-scale style discriminators classify feature means and standard deviations as real or fake; the authors report that they help prevent mode collapse and enable longer training.Source: §3.3 · p. 4

4. Experiment settings

Experiments use six image datasets at 64×64 or 128×128 resolution, with dataset-specific training and normalization settings. Only Chairs has multiple views of the same object.

Data

  • Training covers Basel Face, CelebA, Cats, Chairs, Cars, and LSUN bedroom, at 64×64 for Cats and Chairs and 128×128 for the others.Source: §4 · p. 5
  • Only Chairs has multiple views per object; its images come from 60 randomly sampled views of each of 6,778 ShapeNet chair models.Batches contain different chair types to prevent using repeated views of the same chair as set supervision.Source: §4 · p. 5

Implementation details

  • The generator uses adaptive instance normalization, while the discriminator combines instance normalization and spectral normalization.Source: §4 · p. 5
  • Training uses Adam, random z sampled from U(−1,1), and random poses; |z| is 128 except for 128×128 Cars, where it is 200.Source: §4 · p. 5

5. Results

The results assess qualitative pose and identity separation, image fidelity, comparison with explicit 3D geometry, shape–appearance disentanglement, and network and training ablations.

5.1. Qualitative evaluation

  • Across datasets, HoloGAN varies azimuth and elevation while preserving identity; on CelebA, it controls azimuth better than InfoGAN and recovers elevation despite limited variation.At fixed pose, noise interpolation also smoothly interpolates identities, without pose labels or 3D shapes.Source: §5.1 · p. 5, Figure 5 · p. 5, Figure 6 · p. 6

5.2. Quantitative results

  • HoloGAN achieves competitive KID on CelebA and better KID on Chairs and Cars, using the same hyperparameters across datasets except azimuth ranges.KID measures visual fidelity, not feature disentanglement or control.Source: Table 1 · p. 6, §5.2 · p. 6

5.3. Deep 3D representation vs. 3D geometry

  • Against VON, HoloGAN maintains car identity through full 360° azimuth, generates complex backgrounds and shadows, and uses a 16×16×16×64 deep voxel representation.It also generates an entire bedroom scene using a 16×16×16×64 representation.Source: §5.3 · pp. 6–7, Figure 7 · p. 7

5.4. Disentangling shape and appearance

  • HoloGAN’s 3D features control shape, while 2D features control appearance, including texture and lighting.Different 3D-feature resolutions also separate overall shape from finer details such as gender or makeup.Source: §5.4 · p. 7, Figure 8 · p. 8

5.5. Ablation studies

  • Without random 3D transformations during training, manually rotated features fail to produce meaningful views; the trained HoloGAN generates the same objects in different poses.The authors propose that random transformations encourage features that undergo geometric changes while still fooling the discriminator.Source: §5.5 · pp. 7–8
  • Feeding noise to the generator’s first layer confuses pose with identity, whereas HoloGAN’s traditional style input allows pose changes while identity remains unchanged.HoloGAN starts from a learned constant tensor and uses noise as a style controller at different levels.Source: §5.5 · p. 8

6. Discussion and conclusion

HoloGAN learns meaningful 3D representations across datasets and disentangles pose, shape, and appearance while generating images with similar or higher visual quality than 2D-based GANs. Pose recovery depends on training-pose coverage, and ablations show that removing random 3D transformations or replacing the learned constant input impairs rotation or disentanglement.

6. Discussion and conclusion

  • Pose recovery depends on training-pose variety and distribution; CelebA and Cats show weaker elevation recovery than azimuth because their images offer limited elevation variation.Most face images in these datasets are taken at eye level.Source: §6 · p. 8
  • Without random 3D transformations during training, objects fail to rotate; replacing the learned constant tensor with an input mapped from z fails to disentangle pose and identity.The ablation illustrates these failures as azimuth changes from left to right.Source: Figure 9 · p. 8
  • Uniform random pose sampling may map angles incorrectly when its pose distribution does not match the training data.The authors suggest learning the pose distribution from training data and further disentangling appearance factors such as texture and illumination.Source: §6 · p. 8
  • HoloGAN learns meaningful 3D representations across datasets, disentangles pose, shape, and appearance, and generates images with similar or higher visual quality than 2D-based GANs.The authors argue that explicit deep 3D representations are a step forward for GAN interpretability and controllability.Source: §6 · p. 8

Related papers

Loading 1904.01326v2…