PaperlayerView exact revision on arXiv

Source-linked AI summary

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, Gordon Wetzstein

arXiv:2012.00926v2cs.CVcs.GR

TL;DR

Existing 3D-aware image synthesis methods may lack multi-view consistency or fine detail. π-GAN combines a SIREN-based implicit radiance field with FiLM conditioning and volumetric rendering, achieving state-of-the-art results across real and synthetic datasets. The method remains computationally expensive and can exhibit artifacts or geometric ambiguities under limited pose distributions.

  • Problem

    Existing approaches may lack an underlying 3D representation or view-consistent rendering, and their representation networks may be insufficiently expressive for fine image quality.

  • Method

    π-GAN conditions a SIREN-based implicit radiance field with FiLM and renders it from arbitrary camera poses using differentiable volume rendering.

  • Results

    π-GAN achieves state-of-the-art 3D-aware image synthesis results on real-world and synthetic datasets and demonstrates generalization to new viewpoints.

  • Takeaways & Limitations

    The neural radiance field provides view consistency, explicit camera control, and an interpretable 3D proxy shape without explicit supervision.

  • Takeaways & Limitations

    π-GAN is computationally expensive compared with traditional 2D GANs and still falls short of state-of-the-art 2D-GAN image quality.

Abstract

from arXiv · show

We have witnessed rapid progress on 3D-aware image synthesis, leveraging recent advances in generative visual models and neural rendering. Existing approaches however fall short in two ways: first, they may lack an underlying 3D representation or rely on view-inconsistent rendering, hence synthesizing images that are not multi-view consistent; second, they often depend upon representation network architectures that are not expressive enough, and their results thus lack in image quality. We propose a novel generative model, named Periodic Implicit Generative Adversarial Networks ($π$-GAN or pi-GAN), for high-quality 3D-aware image synthesis. $π$-GAN leverages neural representations with periodic activation functions and volumetric rendering to represent scenes as view-consistent 3D representations with fine detail. The proposed approach obtains state-of-the-art results for 3D-aware image synthesis with multiple real and synthetic datasets.

1. Introduction

π-GAN addresses limitations in 3D-aware image synthesis by combining a SIREN-based implicit radiance field with volumetric rendering, FiLM conditioning, and progressive training. It targets fine image detail, view consistency, explicit camera control, and state-of-the-art performance on real and synthetic datasets.

  • Motivation: Existing 3D-aware methods trade off multi-view consistency, fine detail, resolution, or camera-pose generalization.Voxel methods are computationally limited, while learned black-box rendering does not guarantee consistency or broad camera-pose generalization.
  • Method: π-GAN conditions a SIREN-based implicit radiance field on noise and renders it from arbitrary camera poses using differentiable classical volume rendering.The field maps 3D locations and viewing directions to view-dependent radiance and view-independent density.
  • Results: π-GAN improves image quality and view consistency, with periodic activations enabling finer details and sharper images than ReLU implicit representations.The underlying neural radiance field also supports rendering across a wide range of camera poses and provides an interpretable 3D structure.
  • Method: FiLM conditioning and a progressive growing discriminator are key components for achieving high-quality results with the SIREN-based implicit GAN.The mapping network conditions SIREN layers through feature-wise linear modulation, while progressive growing accelerates training and offsets 3D-GAN complexity.
  • Results: π-GAN achieves state-of-the-art 3D-aware image synthesis results on CelebA, Cats, and CARLA using unsupervised 2D data.The approach demonstrates view consistency and explicit camera control as advantages of neural radiance fields with classical rendering.

2. Related Work

Prior work spans neural implicit representations, generative 3D-aware synthesis, and conditional 3D reconstruction. π-GAN combines SIREN and neural rendering for unconditional generative radiance fields while addressing resolution and expressiveness limitations in earlier representations.

  • Neural representations and rendering: Neural implicit representations provide continuous, memory-efficient 3D structure for objects and scenes, with SIREN and NeRF most closely related to π-GAN.π-GAN combines SIREN as its representation network with neural rendering inspired by NeRF for generative applications.
  • Generative 3D-aware image synthesis: Voxel and explicit-volume approaches model 3D structure but are constrained in resolution, while meshes and primitives lack sufficient expressiveness for high-fidelity images.These representation choices motivate more expressive implicit scene representations.
  • Generative 3D-aware image synthesis: GRAF is the closest prior work, but π-GAN differs by using SIREN, FiLM conditioning through a mapping network, and a distinct training strategy.GRAF uses positionally encoded ReLU MLPs and concatenates separate shape and appearance noise codes.
  • Conditional reconstruction: Conditional 3D reconstruction from partial observations is distinct from π-GAN’s goal of learning an unconditional generative model of radiance fields.Prior reconstruction methods use voxels, meshes, point clouds, depth maps, or implicit functions.

3. Methods

π-GAN generates implicit radiance fields from noise and renders them into images using neural volume rendering. Its SIREN-based representation is conditioned with FiLM, while progressive discriminator growth and ray sampling support training at increasing image resolutions.

  • Methods: π-GAN maps input noise to a conditioned implicit radiance field, then renders it into a 2D image from camera pose ξ.At test time, the field can be rendered from arbitrary camera poses.
  • SIREN-Based Implicit Radiance Field: The radiance field takes 3D coordinates and viewing directions and predicts spatial density σ(x) and view-dependent color c(x,d).A StyleGAN-inspired mapping network conditions the SIREN through FiLM.
  • SIREN-Based Implicit Radiance Field: Each SIREN layer applies an affine transform followed by a sine activation whose frequencies and phase shifts are produced from noise by a ReLU mapping network.The mapping network supplies γ_i and β_i to condition the layers.
  • Neural Rendering: Neural volume rendering casts rays from the camera origin, samples density and color along each ray, and integrates them into pixel color C.The implementation uses stratified and hierarchical sampling from NeRF.
  • Neural Rendering: Neural rendering provides explicit control over camera pose, focal length, and aspect ratio beyond the training poses.The method uses a pinhole camera model and renders rays through the implicit volume.
  • Training Details: Training uses a progressively growing convolutional discriminator, beginning at 32×32 resolution while the generator remains fixed and samples rays more densely.The discriminator gains layers as resolution increases; the generator resolution changes through denser ray sampling.

4. Experiments and Analysis

The experiments evaluate π-GAN’s image quality, view consistency, camera-pose generalization, 3D interpretability, and architectural choices. Across qualitative comparisons and analyses, π-GAN combines fine detail with view-consistent rendering and broad camera control.

  • Experimental setup: The experiments evaluate CelebA, Cats, and CARLA at 128 × 128 resolution against HoloGAN and GRAF using image-quality metrics including FID, KID, and Inception Score.The authors report significant image-quality improvements over baselines, particularly on real-world datasets with fine details.
  • Image quality: π-GAN better preserves fine details and identity across viewpoints than HoloGAN and GRAF while recovering a wide range of camera angles.The comparisons highlight individual teeth, whiskers, minimal identity shift, and inherently view-consistent rendering.
  • Camera-pose generalization: The learned radiance field supports rendering from steep or unseen camera poses, although artifacts increase near the edges of CelebA’s imbalanced pose distribution.CARLA’s more uniformly distributed poses did not show the same issue.
  • 3D representation: π-GAN’s view-independent density enables extracting an interpretable proxy 3D shape through marching cubes.The proxy mesh can be produced from the density field, though it may be lower quality than shapes from specialized single-view reconstruction methods.
  • Ablations and training: Progressive growing improves generated-image quality and diversity by enabling larger stabilizing batch sizes during early low-resolution training.The motivation is that computational complexity grows quadratically with image size.

5. Discussion

The discussion highlights π-GAN’s applications and interpretable 3D representations while identifying geometric ambiguities, mesh-quality limits, computational cost, image-quality gaps, and ethical risks.

  • Applications to novel view synthesis: A trained generator can perform single-view reconstruction without architectural modifications by optimizing per-layer frequencies and phase shifts.The procedure freezes the implicit representation and searches for parameters whose rendered radiance field matches the target image.
  • 3D representations: π-GAN produces interpretable, view-consistent 3D representations, but extracted meshes may require refinement.The paper describes proxy shapes extracted with marching cubes as fair approximations, while noting they may be lower quality than task-specific reconstructions.
  • Failure modes: In some cases, rendered views appear viable even when the inferred radiance field violates the expected 3D shape.The paper attributes such ambiguities to the constrained pose range observed by the discriminator during training.
  • Limitations and future work: π-GAN remains below state-of-the-art 2D convolutional GANs in image quality and is computationally expensive because training scales with image size and ray depth.More efficient rendering could reduce this computational barrier and support larger, sharper images.
  • Ethical considerations: Extending inverse rendering to fake photos or videos of real people presents a societal threat, while CelebA contributes implicit bias and limited facial diversity.The authors explicitly reject using the work to spread misinformation or tarnish reputations.

A. Novel View Synthesis Details

The novel-view-synthesis procedure reuses a trained π-GAN generator for single-view reconstruction by optimizing its latent representation against a target image.

  • Novel View Synthesis Details: A trained π-GAN generator can perform single-view reconstruction without modifications using an inverse projection procedure.The method is based on the procedure outlined by Karras et al.
  • Novel View Synthesis Details: The procedure freezes the implicit representation and optimizes each layer’s frequencies and phase shifts so the rendered radiance field matches the target image.The optimization minimizes mean-squared error after initialization from averages computed over random noise inputs.

B. Model Details

The model uses a mapping network, a FiLMed-SIREN implicit radiance-field backbone, and a progressively growing discriminator.

  • Model Details: The mapping network is an MLP with three hidden layers of 256 units using leaky-ReLU activations with slope 0.2.This passage specifies the mapping-network architecture.
  • Model Details: The generator’s implicit radiance field uses eight FiLMed-SIREN hidden layers with 256 units each.The backbone is described as a SIREN-based implicit radiance field.
  • Model Details: The discriminator progressively grows from low resolutions, fading in new stages over ten-thousand iterations while using CoordConv layers and residual connections.The architecture also considered a patch discriminator but found it produced uneven image quality.

C. Additional Training Details

Training uses progressive image resolutions, memory-saving strategies, dataset-specific rendering settings, and standard evaluation and reconstruction comparisons.

  • Additional Training Details: Training begins at 32×32 resolution with batch size 120, then reduces batch size fourfold at each upsample to fit memory.At higher resolutions, mini-batch aggregation and automatic mixed precision further reduce memory usage.
  • Additional Training Details: COLMAP reconstructions on CelebA use default parameters without known camera poses, with GRAF results taken from its supplement.This establishes the comparison setup for the reconstruction figure.
  • Additional Training Details: Precision-recall plots compare π-GAN, GRAF, and HoloGAN on CelebA, Cats, and CARLA.The figure spans three datasets and three 3D-aware synthesis methods.
  • Additional Training Details: Table 3 reports the discriminator architecture and Table 4 reports FID, KID mean × 100, and IS across CelebA, Cats, and CARLA.The supplied captions identify the evaluation metrics and datasets but not their numerical values.
  • Additional Training Details: π-GAN training takes 10 hours at 32×32, 10 hours at 64×64, and 36 hours at 128×128.The reported schedule reflects progressively higher-resolution training.
  • Additional Training Details: The method uses 24 samples per ray for CelebA and Cats and 64 for CARLA, with camera fields of view of 12º, 12º, and 30º respectively.Rendering and camera parameters were tuned according to each dataset.

D. π-GAN results @ 64 × 64

Additional quantitative results are evaluated at 64 × 64 to enable comparison with models reported at lower resolutions.

  • 64 × 64 evaluation enables comparisons between π-GAN and models evaluated at lower resolutions.

E. Additional Visual Results

Additional visual results illustrate π-GAN’s fine detail, broad camera-pose support, and multi-view consistency across generated faces, cats, and cars. The method also supports latent interpolation and fidelity–diversity control at test time.

  • π-GAN renders fine detail on larger images and supports varied camera poses for generated faces, cats, and cars.Figures show faces and cats across wide camera trajectories, plus cars with varying elevation and rotation.
  • COLMAP reconstruction demonstrates multi-view consistency, while π-GAN proxy shapes produce more pleasing novel views than GRAF.
  • Linear interpolation between latent frequencies and phase shifts smoothly interpolates between generated samples.
  • Truncation enables a test-time fidelity–diversity trade-off, but evaluation metrics are reported without truncation because it reduces image diversity.

H. Precision and Recall

Precision–recall plots independently assess fidelity and diversity, comparing π-GAN with GRAF and HoloGAN across CelebA, Cats, and CARLA. Additional figures show curated generated samples and multi-view displays for these datasets.

  • Precision–recall plots compare π-GAN, GRAF, and HoloGAN on CelebA, Cats, and CARLA.The plots are used to independently evaluate fidelity and diversity.
  • Curated CelebA examples are shown as generated samples and from multiple viewing angles.
  • Curated Cats examples are displayed from multiple viewing angles.
  • Curated CARLA examples are displayed from multiple viewing angles.
Loading 2012.00926v2…