Source-linked AI summary
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
Katja Schwarz, Yiyi Liao, Michael Niemeyer, Andreas Geiger
TL;DR
2D generative models lack precise control over viewpoint and pose, while voxel-based 3D approaches face resolution or disentanglement limits. GRAF uses conditional radiance fields with a multi-scale patch discriminator to synthesize high-resolution, 3D-consistent images from unposed 2D data, achieving high-fidelity rendering and improved multi-view consistency.
Problem
2D generative models struggle to disentangle 3D shape and viewpoint, while existing voxel-based methods can produce coarse outputs or entangled representations inconsistent across views at high resolutions.
Method
GRAF uses conditional radiance fields and a multi-scale patch-based discriminator for high-resolution 3D-aware image synthesis from unposed images.
Results
GRAF generates high-resolution images with better multi-view consistency than voxel-based approaches and renders high-fidelity images from arbitrary views.
Takeaways & Limitations
Radiance fields are a powerful representation for generative image synthesis, supporting 3D-consistent models trained from unposed 2D images alone.
Takeaways & Limitations
The results are limited to simple scenes with single objects and do not yet scale to complex real-world scenes.
Abstract
from arXiv · showhide
While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.
1 Introduction
2D GANs synthesize high-resolution images but do not reliably disentangle 3D shape and viewpoint. GRAF addresses this with conditional radiance fields and a multi-scale patch discriminator for high-resolution, 3D-consistent synthesis from unposed images.
- 2D generative models struggle to disentangle 3D shape and viewpoint despite strong high-resolution image-synthesis performance.
- 3D-aware synthesis methods provide explicit control over scene content and camera pose through differentiable rendering, but 3D supervision or posed images are difficult to obtain.
- Learned 3D-to-2D mappings can entangle representations, causing views to become inconsistent at high resolutions.
- GRAF uses conditional radiance fields to support high-resolution 3D-aware synthesis from unposed images, with controllable viewpoint, shape, and appearance.
- A multi-scale patch-based discriminator samples images at multiple scales to learn high-resolution generative radiance fields efficiently.
2 Related Work
Prior 3D-aware generative methods often require 3D supervision, voxel representations, or learned projections that limit resolution or view consistency. GRAF instead learns shape and texture from 2D images using a continuous representation that renders at arbitrary resolution.
- Some 3D-aware generative methods require 3D supervision or 3D information as input, whereas GRAF learns shape and texture from 2D images alone.
- Voxel-based representations are memory intensive, which limits image synthesis at high resolutions.
- Learned 3D-to-2D projections can entangle object identity and viewpoint, particularly at high resolutions.
- Continuous implicit representations avoid spatial discretization and are not restricted in topology, although related hybrid grids may require 3D input and omit texture.
3 Method
GRAF represents generated scenes with conditional radiance fields, separating camera, shape, and appearance factors while rendering images through differentiable volume rendering. It trains from unposed images using randomly sampled patches and a convolutional discriminator, then renders full-resolution images at inference.
- 3.1 Neural Radiance Fields: Radiance fields provide a continuous scene representation that maps 3D locations and viewing directions to color and volume density.Positional encoding improves fitting of high-frequency signals, with fewer encoding components typically used for viewing directions.
- 3.1 Neural Radiance Fields: Volume rendering combines sampled colors and densities along each camera ray to produce one RGB pixel value.Novel views are synthesized by applying the rendering operator independently to each pixel or ray.
- 3.2 Generative Radiance Fields: GRAF extends radiance fields into an adversarial generative model trained on unposed images rather than many posed images of one scene.The generator samples camera parameters, pose, image patches, and shape and appearance latents to produce synthesized patches.
- 3.2.1 Generator: Training predicts randomly scaled and translated K × K patches, while inference predicts one color value for every target-image pixel.Continuous patch coordinates define rays; training uses R = K^2 rays, whereas inference uses R = WH rays.
- 3.2.1 Generator: The conditional radiance field uses shape and appearance codes, making density depend on position and shape while color additionally depends on viewing direction and appearance.This separation encourages multi-view consistency and models view-dependent effects such as specularities.
- 3.2.2 Discriminator and 3.2.3 Training and Inference: A convolutional discriminator compares generated patches with bilinearly sampled real patches, and training uses a non-saturating GAN objective with R1 regularization.The discriminator operates on continuous displacements and scales while retaining high-frequency details from the real image.
4 Experiments
Experiments evaluate GRAF on synthetic and real-world datasets against voxel-based baselines using image fidelity, high-resolution synthesis, 3D reconstruction, and disentanglement analyses. GRAF renders high-fidelity arbitrary views, scales to higher resolutions, supports shape–appearance control, and benefits from multi-scale patch sampling.
- Experimental setup: Experiments use two synthetic and three real-world datasets, including Chairs, Cars, Faces, and images up to 512^2 pixels.Evaluation measures FID, KID, and 3D reconstruction accuracy using COLMAP and chamfer distance.
- Comparison with voxel-based approaches: At 64^2 resolution, GRAF produces fewer visible artifacts than PLATONICGAN and HoloGAN while disentangling object identity from camera viewpoint.HoloGAN achieves similar FID on Faces and Cats, where azimuth variation is limited.
- High-resolution synthesis: GRAF improves high-resolution synthesis over naïve bilinear upsampling and achieves the smallest FID when trained at full resolution.PLATONICGAN becomes memory intensive at higher resolutions, while HoloGAN shows training instabilities and mode collapse at 512^2.
- 3D consistency: GRAF yields more effective multi-view reconstruction than HoloGAN without 3D convolutions, whose learned 2D upsampling produces fewer cross-view correspondences.Reconstruction accuracy is assessed with COLMAP on 100 Cars reconstructions and compared using chamfer distance.
- Disentanglement: GRAF disentangles shape and appearance: appearance controls object color for Cars and Chairs, and skin or hair color for Faces.The controls are available through separate shape and appearance codes during inference.
- Ablations: Random-scale patch sampling is crucial for robust performance, whereas local-only patches prevent the generator from learning the correct shape.The multi-scale discriminator performs similarly to or better than a full-image discriminator in the reported ablation.
- Ablations: FID remains similar for focal lengths from 0.7f_data to 1.0f_data and stays good up to 1.8f_data or orthographic projection.Very small focal lengths cause border distortions and higher FID values.
5 Conclusion
GRAF enables high-resolution 3D-aware image synthesis with better multi-view consistency than voxel-based approaches, but remains limited to simple single-object scenes.
- GRAF generates high-resolution images for 3D-aware image synthesis.
- GRAF achieves better multi-view consistency than voxel-based approaches.
- The current results are limited to simple scenes containing single objects.
- Depth maps or symmetry could help extend GRAF to more challenging real-world scenarios.
Broader Impact
3D-aware image synthesis has not yet scaled to complex real-world scenes, limiting immediate societal applications. The paper identifies potential future uses in virtual reality, data augmentation, robotics, simulation, and reducing the real-to-synthetic domain gap.
- 3D-aware image synthesis does not yet scale to generating complex real-world scenes.
- This limitation prevents immediate applications for society.
- Potential applications include virtual reality, data augmentation, and robotics.
- Generative 3D models could support cost-efficient data collection and simulation for systems such as autonomous vehicles.
- These models could help close the domain gap between real-world and synthetic data.