Source-linked AI summary
GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation
Yu Deng, Jiaolong Yang, Jianfeng Xiang, Xin Tong
TL;DR
Existing NeRF-based 3D-aware generators struggle to produce highly realistic fine details because volumetric sampling is costly and unstable at practical sample budgets. GRAM instead learns radiance on shared implicit surfaces and samples rays at their intersections, yielding realistic fine details and strong visual 3D consistency. The method outperforms prior methods across multiple datasets, while remaining behind traditional 2D GANs in quality and speed and being limited by shared geometry assumptions.
Problem
Volumetric NeRF generators have high memory and computation costs that restrict ray samples, limiting fine-detail modeling and causing unstable Monte Carlo noise during GAN training.
Method
GRAM jointly learns shared implicit surfaces and radiance fields, using differentiable ray-surface intersections as point samples for rendering.
Results
GRAM significantly outperforms prior methods across FFHQ, Cats, and CARLA, generating realistic images with geometrically consistent fine details.
Takeaways & Limitations
Learning radiance on 2D manifolds supports high-quality 3D-aware images with fine details and strong visual 3D consistency, including thin structures.
Takeaways & Limitations
Shared surfaces can cause artifacts and restrict GRAM to categories with similar geometry, while its quality and speed remain below traditional 2D GANs.
Abstract
from arXiv · showhide
3D-aware image generative modeling aims to generate 3D-consistent images with explicitly controllable camera poses. Recent works have shown promising results by training neural radiance field (NeRF) generators on unstructured 2D images, but still can not generate highly-realistic images with fine details. A critical reason is that the high memory and computation cost of volumetric representation learning greatly restricts the number of point samples for radiance integration during training. Deficient sampling not only limits the expressive power of the generator to handle fine details but also impedes effective GAN training due to the noise caused by unstable Monte Carlo sampling. We propose a novel approach that regulates point sampling and radiance field learning on 2D manifolds, embodied as a set of learned implicit surfaces in the 3D volume. For each viewing ray, we calculate ray-surface intersections and accumulate their radiance generated by the network. By training and rendering such radiance manifolds, our generator can produce high quality images with realistic fine details and strong visual 3D consistency.
1. Introduction
GRAM addresses the quality and training instability of volumetric 3D-aware image generation by learning radiance on shared implicit surfaces and sampling rays at deterministic surface intersections. The method reports realistic fine details, strong 3D consistency, and performance gains over prior methods.
- Motivation: Volumetric NeRF generators remain costly because ray-based volume integration requires many samples during training and inference.Reduced sampling limits image quality and increases computational and memory demands.
- Motivation: Insufficient Monte Carlo samples create unstable color noise across adjacent pixels, harming GAN training with full-image discrimination.Patch rendering is an alternative, but patch discriminators may reduce image generation quality.
- Method: GRAM learns point sampling and radiance fields on shared, jointly trained implicit surfaces, using ray-surface intersections as samples.A manifold predictor defines the surfaces, while a radiance generator predicts color and occupancy from the intersections.
- Advantages: Deterministic, smoothly varying intersections reduce sampling noise and help the generator learn thin structures and texture details that project onto images.The authors state that high-quality results remain possible with very few surface samples, and surfaces with radiance can be pre-extracted for real-time rendering.
- Results: GRAM significantly outperforms prior methods across FFHQ, Cats, and CARLA, producing realistic images with geometrically consistent fine details.The authors describe this as narrowing the quality gap between 3D-aware and traditional 2D image generation.
2. Related Work
Related work follows either learned 3D-aware features with neural rendering or direct 3D representations with physically based rendering. GRAM belongs to the latter direction while targeting the limitations of prior volumetric and mesh-based approaches.
- Neural scene representation: NeRF uses continuous MLP-based implicit representations to capture complex scene structures and synthesize 3D-consistent images with fine details.The broader neural representation literature emphasizes continuous 3D-level control over image synthesis.
- 3D-aware feature rendering: One line of 3D-aware methods represents scenes with 3D-aware features and uses neural renderers, typically CNNs, to synthesize images.Examples include HoloGAN, BlockGAN, and methods that project generated 3D primitives or volumetric features before image synthesis.
- 3D-aware feature rendering: Learned black-box rendering can produce higher-resolution images but sacrifices exact multiview consistency.This trade-off motivates approaches using more explicit scene representations and rendering processes.
- Direct 3D representation: A second line directly models 3D scenes and uses physically based rendering to achieve stricter 3D consistency.Mesh-based methods use rasterization, while NeRF-based methods use neural radiance fields.
- Direct 3D representation: Mesh-based representations cannot handle complicated structures with non-Lambertian reflectance such as hair and fur effectively.This limitation is distinct from the volumetric sampling challenge addressed by GRAM.
3. Approach
GRAM learns shared implicit isosurfaces to restrict radiance sampling, renders their ray intersections, and trains the generator end-to-end with adversarial learning. This design supports detailed, viewpoint-controllable, 3D-consistent image generation while reducing sampling randomness.
- Overall generator: GRAM’s generator takes latent code z and camera pose θ to synthesize an image under the specified viewpoint.The generator is composed of a manifold predictor M and radiance generator Φ.
- 3.1. Manifold Predictor: The manifold predictor M uses a light-weight MLP to define shared implicit isosurfaces that reduce the domain for point sampling and radiance learning.The isosurface levels are predefined constants and are learned jointly with the generative model.
- 3.1. Manifold Predictor: For each viewing ray, GRAM samples intersections with the learned surfaces and passes only those points to the radiance generator for rendering.Intersections are computed differentiably by searching sampled ray intervals and linearly interpolating between endpoints; 64 points are used to obtain accurate intersections.
- 3.2. Radiance Generator: The radiance generator Φ is an MLP that predicts occupancy α and color c for manifold points, using output-layer skip connections to combine different detail levels.Because radiance is defined on surfaces rather than throughout the volume, the method predicts occupancy instead of NeRF’s volume density σ.
- 3.3. Manifold Rendering: GRAM renders each ray by accumulating radiance from ordered manifold intersections, whose deterministic and smoothly varying locations avoid Monte Carlo sampling noise.This rendering scheme differs from NeRF’s hierarchical random sampling throughout the volume.
- 3.4. Training Strategy: The full framework is trained adversarially by sampling latent codes and camera poses, synthesizing images, and training a discriminator against real images.An optional pose-regularization loss uses an additional discriminator branch when adversarial training can produce incorrect concave geometry for convex objects.
4. Experiments
GRAM generates detailed, view-controllable images with strong visual 3D consistency across faces, cats, and cars. Experiments show that learned manifolds support fine details, competitive image quality, and effective rendering with sparse sampling.
- Generation Results: GRAM is trained on FFHQ, Cats, and CARLA, using 256^2 resolution with 24 surfaces for faces and cats and 128^2 resolution with 48 surfaces for cars.The evaluation uses three datasets and explicitly controllable camera viewpoints.
- Generation Results: The learned manifolds adapt to viewpoint coverage: face and cat surfaces become near-planar, while car surfaces remain curved across the wider viewing range.Faces and cats have small angle variations, whereas CARLA spans 360° azimuth and 90° elevation.
- Generation Results: GRAM learns high-frequency details and thin structures on its manifolds and produces detailed proxy 3D shapes for generated instances.Examples include cat whiskers; proxy geometry is extracted with marching cubes and supports consistency across fine details.
- Comparison with Previous Methods: GRAM achieves the best visual quality among the compared 3D-aware methods, while GRAF and pi-GAN miss high-frequency details and GIRAFFE shows 3D inconsistency.The comparison covers three datasets and highlights realistic details together with strong 3D consistency.
- Comparison with Previous Methods: GRAM significantly improves FID and KID over GRAF and pi-GAN and achieves lower FID and KID than GIRAFFE.Metrics are computed between 20K generated images and 20K real images; Table 1 also includes StyleGAN2 for reference.
- Ablation Study: Learnable manifolds outperform NeRF-H, planar sampling, and fixed spherical initialization, with high-quality results using as few as 6 surfaces.NeRF-H largely fails below 12 points and remains worse than GRAM with 48 points; it can also produce concave forehead geometry.
5. Conclusions
GRAM regulates radiance generation on 2D manifolds and reports improved generation quality and 3D consistency, while retaining important scope and fidelity limitations.
- GRAM regulates point sampling and radiance learning on 2D manifolds to support 3D-aware image generation.
- Extensive experiments report superiority over previous methods in both generation quality and 3D consistency.
- Generated content can contain visual artifacts, unnatural texture patterns, and unpredictable failures detectable by humans and fake-image detectors.
- Shared class-wide surfaces can cause artifacts, constrain applicable categories to similar geometries, and limit handling of complex multi-subject scenes.
- GRAM’s generation quality and speed still fall behind traditional 2D GANs, motivating better representations for fidelity and efficiency.
Supplementary Material
Figure I details the network structures of the manifold predictor and radiance generator.
- Figure I presents the detailed structures of the manifold predictor M and radiance generator Φ.
A.1. Data Preparation
The supplementary data preparation describes alignment and pose estimation for FFHQ and Cats, direct resizing and pose sampling for CARLA, and network structures for M and Φ.
- Data Preparation: FFHQ faces are aligned using five facial landmarks, then resized and cropped by matching detected and corresponding 3D keypoints.
- Data Preparation: Cat images are aligned and resized using dataset landmarks, with camera poses estimated from 2D landmarks and manually selected 3D cat-mesh landmarks.
- Data Preparation: The estimated Cats pose distribution is close to FFHQ’s, so training uses the same Gaussian pose distribution.
- Data Preparation: CARLA car images are resized directly to 128^2 resolution, while camera poses are uniformly sampled from the upper hemisphere.
- Network Structures: The manifold predictor M is an MLP with three hidden layers and an output layer, using empirically selected channel dimensions by dataset.
- Network Structures: The radiance generator Φ combines a three-hidden-layer mapping MLP with eight 256-dimensional FiLM SIREN blocks and one view-conditioned 259-dimensional block.
A.3. More Training Details
Training samples latent codes and camera poses, jointly optimizes the manifold predictor, radiance generator, and discriminator, and uses memory and batching strategies for higher resolutions.
- A.3. More Training Details: Training randomly samples latent code z from a normal distribution and camera pose θ from known or estimated dataset distributions.
- A.3. More Training Details: The manifold predictor M, radiance generator Φ, and discriminator D are jointly learned using the paper’s adversarial losses.
- A.3. More Training Details: Geometric initialization gives M sphere-like initial isosurfaces, with dataset-specific centers producing spherical or hemispherical manifolds.
- A.3. More Training Details: GRAM training runs for 120K, 80K, and 70K iterations on FFHQ, Cats, and CARLA, respectively, taking 3 to 7 days.
B.1. Qualitative Results
GRAM produces realistic images with strong multiview consistency, outperforming prior methods visually and avoiding several sampling-related and geometric failures observed with NeRF-H. It also supports camera manipulation, latent interpolation, and style mixing.
- GRAM generates realistic images with strong multiview consistency across FFHQ, Cats, and CARLA.The reported visual results cover all three datasets.
- GRAM achieves the best visual image quality and 3D consistency among the compared 3D-aware generation methods.
- Compared with NeRF-H, GRAM produces finer details and avoids failures with fewer than 12 sampling points, including unreasonable concave forehead geometry.NeRF-H still produces hollow-face artifacts even with 48 points, whereas GRAM uses 6 surfaces in the comparison.
- For some generated subjects, hair fringes can exhibit exaggerated parallax because category-shared surfaces provide inaccurate depth for every structure.The authors identify instance-specific surfaces as a possible future mitigation.
- GRAM generates reasonable camera zoom-in and zoom-out effects while preserving view-controllable image synthesis.
- Latent interpolation yields continuous semantic changes, and style mixing separates geometry control in layers 1–5 from appearance control in layers 6–9.The style-mixing demonstration was not used during training.