Source-linked AI summary
VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, Andreas Geiger
TL;DR
3D-aware generative models need fast rendering without entangling camera viewpoint and generated content. VoxGRAF replaces coordinate-based MLP scene representations with sparse voxel grids and 3D convolutions, generating a complete scene in one forward pass. It achieves high-fidelity, 3D-consistent rendering with efficient viewpoint changes, while retaining a trade-off between 3D consistency and image fidelity.
Problem
Coordinate-based MLPs make ray-wise rendering slow, while neural upsampling can entangle viewpoint with geometry or appearance.
Method
VoxGRAF uses sparse voxel grids with progressive growing, pruning, regularization, and a sparse 3D CNN in place of a coordinate-based MLP.
Results
A single forward pass generates the full 3D scene, after which images render within milliseconds from different viewpoints; existing approaches require another forward pass two orders of magnitude slower.
Takeaways & Limitations
Sparse voxel representations provide efficient rendering while retaining high visual fidelity and 3D consistency across viewpoints.
Takeaways & Limitations
A trade-off remains between 3D consistency and image fidelity, with the shallow CNN modeling some dataset bias such as eyes looking into the camera.
Abstract
from arXiv · showhide
State-of-the-art 3D-aware generative models rely on coordinate-based MLPs to parameterize 3D radiance fields. While demonstrating impressive results, querying an MLP for every sample along each ray leads to slow rendering. Therefore, existing approaches often render low-resolution feature maps and process them with an upsampling network to obtain the final image. Albeit efficient, neural rendering often entangles viewpoint and content such that changing the camera pose results in unwanted changes of geometry or appearance. Motivated by recent results in voxel-based novel view synthesis, we investigate the utility of sparse voxel grid representations for fast and 3D-consistent generative modeling in this paper. Our results demonstrate that monolithic MLPs can indeed be replaced by 3D convolutions when combining sparse voxel grids with progressive growing, free space pruning and appropriate regularization. To obtain a compact representation of the scene and allow for scaling to higher voxel resolutions, our model disentangles the foreground object (modeled in 3D) from the background (modeled in 2D). In contrast to existing approaches, our method requires only a single forward pass to generate a full 3D scene. It hence allows for efficient rendering from arbitrary viewpoints while yielding 3D consistent results with high visual fidelity.
1 Introduction
3D-aware GANs address the lack of 3D constraints in 2D image synthesis, but voxel and coordinate-based approaches face resolution, rendering-speed, or consistency challenges. VoxGRAF revisits sparse voxel grids to generate scenes efficiently while preserving viewpoint control and 3D consistency.
- 3D-aware GANs combine 3D generators, differentiable rendering, and adversarial training to synthesize images with explicit camera-pose control.
- Voxel-based methods either limit image resolution through cubic memory growth or use neural rendering that can entangle viewpoint and generated content.
- Coordinate-based NeRF representations model color and density with an MLP and project them into images through differentiable volume rendering.
- Sparse voxel grids can retain visual fidelity while accelerating density and color queries for volumetric rendering.
- VoxGRAF generates a 3D scene in one forward pass, then renders high-resolution views efficiently with improved 3D consistency.After scene generation, rendering takes milliseconds, whereas existing approaches require another forward pass that is two orders of magnitude slower.
2 Related Work
Related work progresses from 2D GANs and low-resolution voxel grids toward physically inspired 3D-aware models, but resolution, rendering cost, and consistency remain competing concerns. VoxGRAF uses sparse voxel grids to speed rendering without reducing the volume-rendering sample size.
- 2D GANs achieve photorealistic high-resolution synthesis, but precise 3D-consistent camera control remains difficult because they operate without a physical 3D interpretation.
- Dense voxel grids are constrained by cubic memory growth, while low-resolution grids paired with neural rendering can lack 3D consistency.
- Several methods reduce volume-rendering queries, whereas VoxGRAF instead uses sparse voxel grids so fast feature interpolation preserves the sample size.
- Sparse voxel grids support faster convergence, higher rendering speed, and trilinear interpolation beyond the grid resolution.
3 Method
VoxGRAF replaces the coordinate-based MLP radiance-field generator with a sparse 3D convolutional voxel-grid generator, retaining differentiable volume rendering while enabling a single forward pass for full-scene generation. The method combines pose-conditioned 3D foreground generation, a 2D background generator, voxel pruning, progressive growing, and regularization for efficient, 3D-consistent rendering.
- Sparse voxel scene representation: VoxGRAF replaces the coordinate-based MLP with a sparse 3D CNN that generates radiance-field values on a voxel grid.The generator predicts voxel colors and densities rather than querying an MLP at individual 3D points.
- Foreground and background generation: The model generates the 3D foreground and combines its rendered image with a background produced by a 2D GAN through alpha composition.Modeling only the object in 3D reduces computation while preserving a separate background-generation pathway.
- Sparse voxel scene representation: Conditioned on a latent code and camera pose, the foreground generator produces voxel colors and densities that are rendered using differentiable volume rendering.Voxel values along camera rays are obtained through trilinear interpolation, while the pose is conditioned per image rather than per ray.
- Pruning and progressive growing: At resolutions beyond 32^3, sparse convolutions reduce memory consumption, although coordinate-management overhead can increase computation.The architecture replaces 2D StyleGAN2 operations with 3D equivalents and investigates sparse convolutions at higher resolutions.
- Pruning and progressive growing: Density-based pruning removes occluded or empty voxels, and progressive growing upsamples retained voxels through sparse transposed convolutions.Pruning is implemented during training with custom CUDA kernels to support scaling sparse voxel grids to higher resolutions.
- Regularization and consistency: Pose conditioning and depth-variance, grid-total-variation, and foreground/background coverage regularization support view-dependent effects and sharp, nonempty foreground surfaces.Pose conditioning is matched to the rendering pose in 50% of training cases and fixed at inference to retain 3D consistency.
4 Results
VoxGRAF combines sparse voxel grids with pruning and regularization to produce high-fidelity, 3D-consistent scenes efficiently. It improves consistency and non-neural-renderer image quality while enabling fast novel-view rendering after one scene-generation pass.
- Evaluation: The evaluation uses Carla, FFHQ, and AFHQv2 Cats, with FID measured between 20k generated images and each full dataset.Runtime comparisons use a single Tesla V100 GPU with batch size 1.
- Ablation Study: Depth variance regularization increases voxel-grid sparsity from 74% to 95%, reducing memory consumption and scene-generation and rendering times.The authors train with dense convolutions and zero pruned voxels because this is faster despite higher memory use than sparse convolutions.
- Baseline Comparison: VoxGRAF renders the final image directly from the 3D representation, unlike baselines that use low-resolution features followed by 2D upsampling.This places VoxGRAF in the comparison group without a neural renderer.
- Baseline Comparison: VoxGRAF shows high multi-view consistency, whereas StyleNeRF can add faces or hair across views and GRAM produces stripe artifacts at large viewpoint ranges.The comparison is shown on FFHQ at resolution 2562.
- Baseline Comparison: Among methods without neural rendering, VoxGRAF significantly improves over π-GAN and GOF and surpasses GRAM in FID.StyleNeRF and EG3D achieve better FID overall, which the authors attribute to the flexibility of neural rendering, but StyleNeRF has reduced 3D consistency.
- Baseline Comparison: At 2562 resolution, VoxGRAF generates novel views at 167 FPS after scene generation, compared with 20 FPS for StyleNeRF.The method separates one scene-generation pass from efficient rendering of multiple viewpoints.
5 Limitations and Discussion
The method improves 3D consistency and image fidelity through sparse voxel grids, but retains a trade-off between these goals and has speed limitations on complex scenes.
- The method reduces the gap to heavily neural-rendered models, but a trade-off between 3D consistency and image fidelity remains.A shallow CNN refines rendered images while remaining weaker than standard neural-rendering approaches.
- Rendering speed depends on scene sparsity and will likely decrease on more complex datasets than those commonly used in literature.
- The authors identify whether neural rendering is inherently needed for best performance as an important direction for future research.
Checklist
The checklist reports that the paper addresses contributions, limitations, societal impacts, and ethics guidance, while several reproducibility materials and repeated-run error bars are unavailable.
- The authors report that they described their contributions and scope, limitations, and potential negative societal impacts.
- The paper states that it follows ethics review guidelines and includes no theoretical results requiring assumptions or complete proofs.
- Code, data, and reproduction instructions were not included, although the authors state that code will be provided upon acceptance.
- The authors did not report error bars because multiple runs were computationally infeasible for their models.
- The checklist records no new assets, no applicable consent or personally identifiable-information discussion, and no crowdsourcing or human-subject procedures.
A Implementation
The implementation replaces 2D generator operations with 3D voxel-grid operations for the foreground, combines pruning and regularization, and uses a 2D background and optional refinement network.
- The foreground generator replaces StyleGAN2’s 2D operations with 3D equivalents and uses progressive voxel-grid generation.
- The foreground generator predicts color and density on a sparse voxel grid using degree-0 spherical harmonics.
- Dense convolutions with zeroed pruned voxels can run faster than sparse convolutions because coordinate management adds overhead.
- A StyleGAN2 background generator and an optional shallow 2D CNN separate background modeling from foreground 3D rendering and refine images.
- Depth-variance and total-variation regularization encourage sharp, compact, smooth geometry for efficient volume rendering.
- A hinge loss constrains foreground and background mask coverage so both generators contribute to the image.
- Rendering uses CUDA kernels, equidistant sampling, voxel skipping, and early termination based on density and transmittance thresholds.
B Baselines
The paper evaluates against established 3D-aware image-synthesis baselines using qualitative comparisons and FID measurements, with specified truncation and sample-count conventions.
- Qualitative comparisons include StyleNeRF and GRAM, using truncation ψ = 0.7 for all methods.
- The quantitative comparison reports FID for the full dataset across the listed baselines and the proposed method.
- Figure 6 shows background disentanglement on FFHQ and AFHQ at resolution 256^2 with truncation ψ = 0.7.
- Evaluation uses 50k generated images by default, with annotated exceptions of 1k for GRAF on Carla and 8k for π-GAN on Carla.
C Results
The model improves 3D-aware generation through pose conditioning, foreground–background disentanglement, and regularization, while retaining identifiable failure cases and evaluation caveats. Results indicate faster scene generation and rendering, with qualitative evidence of view consistency and stable training benefits.
- Disentanglement: Foreground–background disentanglement encourages pose-dependent content to remain in the foreground while the background stays fixed across viewpoints.Regularization encourages both generators to contribute to the final image.
- Multi-View Consistency: The method shows qualitative multi-view consistency relative to competing methods, whose examples include changing eye shape, moving hair, face distortion, and layered artifacts.The comparison is qualitative and is supplemented by view-consistency metrics.
- Evaluation Caveats: View-consistency metrics have large standard deviations and are sensitive to latent codes and sampled poses, limiting direct numerical comparison with prior work.The evaluation pipeline and pose sampling may differ across implementations.
- Pose Conditioning: Fixing pose conditioning during inference enables view-consistent images, although conditioning can also alter an instance’s general appearance.The same latent code is used while varying rendering and conditioning poses in the pose-conditioning analysis.
- Regularization: Regularizers do not significantly change FID, but they help stabilize training, and LDV also speeds training.This finding is reported for FFHQ models with RI = 128 and RG = 64.
- Failure Cases: Failure cases include incorrect foreground–background assignment, whiskers connected to a cat’s body, and hair directed inward toward the head.The authors associate these errors with ambiguous disentanglement, regularization effects, and training-data ambiguity, respectively.
- Additional Results: Across FFHQ, AFHQ, and Carla, the method provides additional uncurated samples, while FID changes little between evaluating 20k and 50k generated images.The FID comparison addresses bias toward the number of generated images.
D Societal Impact
The work targets photorealistic scene rendering with potential applications in virtual reality, augmented reality, gaming, and simulation. The authors also flag risks from image manipulation and training-data bias, including reduced diversity in generated faces.
- Potential Applications: Photorealistic scene rendering may support applications in virtual reality, augmented reality, gaming, and simulation.The authors frame these as potential downstream applications.
- Risks: The technology could enable edited imagery of real people, creating a manipulation risk.This risk is identified alongside broader concerns about photorealistic 3D-scene generation.
- Risks: Biases in training data may produce insufficient diversity in generated faces and should be addressed before downstream use.The authors identify this as a limitation of data-driven approaches.