Source-linked AI summary
EpiGRAF: Rethinking training of 3D GANs
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, Peter Wonka
TL;DR
High-resolution NeRF-based generation is costly, and 2D upsamplers can compromise multi-view consistency and geometry resolution. EpiGRAF instead trains patch-wise with a scale/location-aware discriminator and annealed Beta scale sampling, achieving strong image quality with lower training cost.
Problem
NeRF rendering is expensive per pixel, and full-image GAN discriminators create scaling problems for generative NeRFs.
Source: §1 · p. 2Method
EpiGRAF trains a pure NeRF-based generator patch-wise, using a scale/location-aware discriminator and annealed Beta sampling of patch scales.
Source: §1 · p. 2, §6 · p. 10Results
EpiGRAF achieves state-of-the-art FID on Cats, M-Plants, and M-Food except against StyleNeRF, and has 2.5× lower training cost than upsampler-based generators.
Source: §1 · p. 3, §4.2 · p. 8Takeaways & Limitations
The authors propose combining efficient volumetric representations, regularized 2D upsamplers, and patch-wise training as a future direction for 3D GANs.
Source: §6 · p. 10In 2D-generation experiments, the authors could not match full-resolution StyleGAN2 image quality, even with longer training.
Source: §5 · p. 9
Abstract
from arXiv · showhide
A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns the geometry in a low fidelity. In this work, we show that it is possible to obtain a high-resolution 3D generator with SotA image quality by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location- and scale-aware discriminator to work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. The resulted model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator, and we test it on four datasets (two introduced in this work) at $256^2$ and $512^2$ resolutions. It obtains state-of-the-art image quality, high-fidelity geometry and trains ${\approx} 2.5 \times$ faster than the upsampler-based counterparts. Project website: https://universome.github.io/epigraf.
1 Introduction
EpiGRAF addresses the cost of high-resolution NeRF-based generation with patch-wise training, a scale/location-aware discriminator, and annealed Beta scale sampling. The paper reports high or comparable image quality, lower training cost, and high-fidelity geometry without the limitations attributed to 2D upsamplers.
1 Introduction
- NeRF rendering is computationally expensive per pixel, while GAN discriminators typically require full images, creating a scaling problem for generative NeRFs.NeRF renders pixels by aggregating color and density along rays.Source: §1 · p. 2
- Upsampler-based generators can break multi-view consistency and represent geometry only at low resolution; patch-wise training instead uses random crops at variable scales.Variable-scale crops can provide both low-resolution global information and high-resolution patches.Source: §1 · p. 2
- A hypernetwork modulates discriminator filters using patch scale and location, adapting the discriminator to patches with different semantic receptive fields.The paper argues that one unconditioned discriminator must otherwise mix filter responses for local high-resolution and global low-resolution patches.Source: §1 · p. 2
- Annealed Beta scale sampling is reported to stabilize and accelerate convergence; EpiGRAF is evaluated on FFHQ, Cats, and new Megascans Plants and Food datasets.The authors say the schedule starts learning high-frequency details early and focuses more on structure after growth finishes.Source: §1 · pp. 2–3
- On FFHQ 512², EpiGRAF’s geometry has more high-frequency detail than EG3D’s, whose shapes appear over-smoothed.The comparison extracts surfaces from 512³ density fields; the paper attributes the difference to EpiGRAF learning geometry at full resolution versus EG3D’s lower rendering resolution.Source: Figure 2 · p. 3
- EpiGRAF reports higher or comparable FID and 2.5× lower training cost than upsampler-based generators.The model uses a pure NeRF-based backbone and does not suffer from multi-view synthesis artifacts.Source: §1 · p. 3
2 Related work
Related work spans 3D generation with differing supervision, NeRF-based generators, and variable-scale patch training. The paper situates its approach among methods that use 2D upsamplers to address volumetric-rendering costs.
2 Related work
- NeRFs combine neural networks with volumetric rendering for tasks including novel-view synthesis, image-to-scene generation, and surface reconstruction.This work employs NeRFs for 3D-aware generation from RGB images.Source: §2 · p. 3
- 3D generative models have used explicit 3D data or latent spaces, RGB data with masks, keypoints, or multiple views, and single-view RGB for mesh or 3D-structure generation.Neural rendering advances also enabled NeRF-based generators trained from RGB data from scratch.Source: §2 · p. 3
- EpiGRAF’s generator uses a tri-plane backbone with a StyleGAN2 decoder and no 2D upsampler, while its StyleGAN2-based discriminator is modulated by patch location and scale.The figure caption reports convergence 2–3 times faster than upsampler-based architectures.Source: Figure 3 · p. 4
- NeRF-based generators include noise-conditioned GRAF, π-GAN, GRAM, GNeRF, GIRAFFE, and CAMPARI, with different generator or scene-representation designs.Recent methods also pair volumetric rendering with 2D decoders to upsample low-resolution features; the cited passage says their regularizations do not currently provide pure-NeRF multi-view consistency.Source: §2 · pp. 3–4
- Patch-wise generation has been used for texture learning and, with variable-scale crops, for generative NeRFs, adversarial neural representations, and variable-resolution 2D GANs.Variable-scale patches can carry global information about an image.Source: §2 · p. 4
3 Model
EpiGRAF trains a pure tri-plane NeRF on multiscale image patches, using a patch-conditioned discriminator and annealed beta scale sampling. The model combines this design with a training setup inherited from StyleGAN2-ADA.
3 Model
- The model replaces StyleGAN2’s generator with a tri-plane NeRF and trains on random-scale r × r patches instead of full images.The patch resolution is r = 64; the tri-plane generator maps noise to feature planes and renders color and density with a NeRF decoder.Source: §3 · p. 4, §3.1 · p. 4
3.2 2D scale/location-aware discriminator
- A hypernetwork-modulated discriminator handles patches at continuously varying scales, avoiding separate discriminators for different scales.Its inputs include patch scale and cropping offsets, which condition convolutional-kernel modulations.Source: §3.2 · p. 5
- ≈15% lower FID than the standard discriminator is reported for the scale/location-aware discriminator.The discriminator’s convolutional filters are modulated according to patch scale and location.Source: §3.2 · p. 5
3.3 Patch-wise optimization with Beta-distributed scales
- Patch-wise optimization renders random r × r crops at sampled scales and offsets, reducing the need to process dense full-resolution images.The paper presents this as a way to obtain high-resolution image quality and training speed without the cited upsampler limitations.Source: §3.3 · pp. 5–6
- Annealed beta scale sampling shifts training focus between local texture and global structure, addressing the uniform schedule’s slower convergence and lower stability.The paper uses β(T) = 0.8 to retain more focus on structure at the end of annealing.Source: Figure 4 · p. 5, §3.3 · p. 6
3.4 Training details
- Training follows StyleGAN2-ADA with Adam at learning rate 0.002, batch size 64, and about 15M or 20M images seen by D at 256² or 512² resolution, respectively.The setup uses β(T) = 0.8 and T = 10000; FFHQ and Cats use pose supervision for D.Source: §3.4 · p. 6
4 Experiments
Across four datasets, EpiGRAF combines competitive image quality with faster training and high-fidelity geometry; ablations support its scale/location-aware discriminator and beta-based patch sampling.
4.1 Experimental setup
- The evaluation covers FFHQ, Cats, and two introduced Megascans benchmarks, M-Food and M-Plants, which vary more in geometry.FID measures image quality, and training cost is measured in V100 GPU-days; the baselines include upsampler and non-upsampler generators.Source: §4.1 · pp. 6–7
4.2 Results
- EpiGRAF outperforms other baselines on Cats, M-Plants, and M-Food FID except for similar-performing StyleNeRF, and ranks 4/9 on FFHQ.On FFHQ, it loses noticeably only to EG3D, which uses a different FFHQ version and pose conditioning. Table 1 reports FID scores; Figure 5 compares samples.Source: Figure 5 · p. 7, §4.2 · p. 8, Table 1 · p. 8
- 2–3× faster training: EpiGRAF finishes in 2 days at 256^2 and 3 days at 512^2, versus about a week for existing methods.The reported efficiency is achieved without a 2D upsampler.Source: §4.2 · p. 8
- EpiGRAF learns high-fidelity geometry with a pure NeRF backbone that supports multi-view consistency and full-dataset-resolution geometry.Figure 6 compares learned shapes; its surfaces were extracted from density fields sampled on a 256^3 grid.Source: Figure 6 · p. 8, §4.2 · p. 9
- EpiGRAF incorporates NeRF++ background separation by adapting and reusing its inverse-sphere parametrization code.The authors describe this as a demonstration of how the pure NeRF generator can use techniques from NeRF scene reconstruction.Source: §4.2 · p. 9
4.3 Ablations
- A single scale/position-aware discriminator achieves the best performance, outperforming a standard discriminator by about 15% on average.A traditional discriminator is about 15% worse; multiple discriminators train more slowly and substantially degrade image quality, while too-small patches produce about 80% worse FID.Source: §4.3 · p. 9, Table 2 · p. 9
- On Cats 256^2, beta sampling with T=10k converges more robustly to lower values than uniform sampling with T=5k or T=10k.It also fluctuates less than uniform sampling with T=1k.Source: §4.3 · p. 9, Figure 7 · p. 9
- About 40% of discriminator filters switch on and off depending on patch scale, indicating that the modulation network learns scale-dependent modulation.Other filters remain always on or always off, suggesting potential room for pruning.Source: §4.3 · p. 9
5 Limitations
The authors report constraints on patch-size choice, global context for the discriminator, and close-up image sharpness, as well as weaker patch-wise performance for 2D generation.
5 Limitations
- For 2D generation, patch-wise StyleGAN2 did not match full-resolution StyleGAN2 image quality, even with longer training.The table caption notes that a 64^2 patch contains 1/64 the information of a 512^2 image; AnyResGAN is noted as an exception reaching state-of-the-art with 256^2 patches versus 1024^2 images.Source: §5 · p. 9, Table 3 · p. 10
- 32^2 patches worsen performance without a dramatic speed gain, while 128^2 patches increase cost by 30% without clear performance benefits.The authors report that decreasing patch size further would worsen performance; their patch-size options are therefore constrained.Source: §5 · pp. 9–10
- Small-scale patches leave the discriminator without global image context; attempts to add that context did not improve performance, though the authors say it was under-explored.Source: §5 · p. 10
- FFHQ 512^2 samples show some close-up blurriness that FID may miss because it resizes images to 299 × 299.The authors attribute the issue to patch-wise training’s emphasis on structure and believe it could be resolved.Source: §5 · p. 10
6 Conclusion
EpiGRAF builds a state-of-the-art 3D GAN without a 2D upsampler, using a pure NeRF generator trained on multi-scale patches. The conclusion also identifies limitations and an avenue for future work.
6 Conclusion
- EpiGRAF combines a pure NeRF generator with multi-scale patch-wise training, using a scale/location-aware discriminator and beta-based patch-scale schedule.The authors report that the schedule supports faster, more robust convergence.Source: §6 · p. 10
- The authors propose future work combining efficient volumetric representations, regularized 2D upsamplers, and patch-wise training.Source: §6 · p. 10
- The authors identify lost global context and possible low-resolution artifacts as limitations; they also flag potential societal impact from face- and human-generating 3D GANs.They report that attempts to add global context did not improve results.Source: §6 · p. 10
Checklist
The checklist reports that the work releases two datasets and provides code, data, and additional visualizations. It also points to supplementary training details and notes that error bars were not reported because they were too computationally expensive.
Checklist
- The authors report providing code, data, and additional visualizations online, with key training details in §3.4 and further details in Appendix B and source code.Source: Checklist · p. 15
- Error bars were not reported because running multiple experiments was too computationally expensive, while single-run results are described as typically reliable in the GAN field.Source: Checklist · p. 15
- The authors report releasing two new datasets, Megascans Plants and Megascans Food, and discussing asset licensing in Appendix E.Source: Checklist · p. 15
- The checklist says dataset collection information and the absence of personally identifiable information or offensive content are discussed in Appendix E.Source: Checklist · p. 15
A Additional limitations due to aliasing
The appendix describes aliasing as a limitation of patch-wise training and gives implementation, resource, and annealing-schedule details. It notes that alias-free patch sampling is difficult for the tri-plane generator because intermediate high-resolution representations are computationally costly.
A Additional limitations due to aliasing
- Naive patch extraction can alias, so the generator may learn patch marginals rather than the full joint image distribution; alias-free sampling is difficult with tri-planes because intermediate high-resolution representations are costly.In the toy example, aliased downsampling hides corner-color correlations in large patches, potentially leaving the generator to omit those high-frequency correlations.Source: A Additional limitations due to aliasing · p. 16, Figure 9 · p. 16
B.1 Hyper-parameters and optimization details
- Training uses batch size 64 until the discriminator sees 25,000,000 real images; the hypernetwork encodes patch scale and offsets into a 512-dimensional patch representation adapted for each convolutional layer.For FFHQ and Cats, the discriminator also uses camera conditioning; the generator uses full precision while the discriminator uses mixed precision.Source: §B.1 · pp. 16–17
B.2 Utilized computational resources
- Training EpiGRAF on four A100 GPUs gives the same training speed as eight V100 GPUs; the project used approximately 4 A100 GPU-years and 4 V100 GPU-years.The authors report that A100s provided a 2× speed-up over V100s in practice by allowing a larger batch size.Source: §B.2 · p. 17
B.3 Annealing schedule details
- The appendix visualizes the GRAF and GNeRF patch-scale annealing schedules and reports that both are close to the lerp-based schedule described in §3.3.Source: §B.3 · p. 17
C Failed experiments
This section catalogs experiments that did not work as expected, including discriminator conditioning, patch sampling, view-direction conditioning, and tri-plane optimization. It also compares annealing schedules.
C Failed experiments
- Conditioning the discriminator on global context worsened performance; in both tested approaches, the generator learned low-resolution images well but produced poor texture.The authors hypothesize that the discriminator relied on the low-resolution image and ignored harder-to-discriminate high-resolution patches.Source: C Failed experiments · pp. 17–18
- Figure 10 compares annealing schedules for GRAF [57], GNeRF [37], and the lerp-based schedule, which the exposition treats as representative of the first two.The text says that the simplified description is very close to reality.Source: Figure 10 · p. 18
- Gradient-based patch importance sampling helped some NeRF-Synthetic scenes but worsened performance on others.Patches from regions with high average gradient norm were sampled more frequently.Source: C Failed experiments · p. 18
- On FFHQ, conditioning radiance on ray direction led to poor multi-view consistency, with radiance changing as the camera moved.The authors suspect this issue would not occur on Megascans, which has richer view coverage.Source: C Failed experiments · p. 18
- Tri-planes produced by StyleGAN2’s synthesis network were much harder to optimize for reconstruction than directly optimized tri-planes.The network-produced planes plateaued quickly in PSNR and loss.Source: C Failed experiments · p. 18
D Additional patch size ablation
The patch-size ablation finds the best performance at 64^2 resolution, while larger patches trade training speed for quality and raise project-wide computational costs.
D Additional patch size ablation
- 64^2 patch resolution achieved the best performance in the fixed-hyperparameter comparison.Table 4 reports FID@2k for a grid search over patch size and R1 regularization weight γ on Cats 256^2; each model saw 5M images.Source: D Additional patch size ablation · p. 18, Table 4 · p. 19
- Increasing patch size should improve performance at the cost of training speed, but the authors judged the quality gain from 128^2 patches not worth the project-wide computational overhead.The fixed-hyperparameter comparison did not adjust γ for training resolution, motivating the patch-size and γ grid search.Source: D Additional patch size ablation · pp. 18–19
E.1 Megascans dataset
The paper introduces Megascans Plants and Megascans Food to address benchmark limitations in geometry diversity and camera coverage. The appendix describes dataset construction, release constraints, and dataset-statistics comparisons.
E.1 Megascans dataset
- Existing benchmarks often have similar global object geometry and poor camera coverage, limiting evaluation of generators’ ability to represent underlying geometry.The authors also note that camera-correlated 3D biases in some datasets make it harder to separate methodological gains from preprocessing effects.Source: §E.1 · p. 19
- Megascans Plants and Megascans Food use high-quality scans from about 1,500 Quixel models and are introduced as more geometrically complex benchmarks.Plants, Mushrooms, and Food categories were gathered; Mushrooms and Plants were combined into Food because each had too few models independently.Source: §E.1 · p. 19
- The rendering pipeline samples 128 views per object from uniformly distributed sphere positions; filtering yields 1,108 plant models and 199 food models.Plants with mean pixel intensity below 0.03 were removed; source models and textures cannot be released because of copyright restrictions, while rendered images are released under CC BY-NC-SA 4.0.Source: §E.1 · p. 19
- The dataset figures show Megascans Plants’ complex geometry and texture, and the appendix compares yaw/pitch-angle distributions across datasets.The Food dataset includes Amanita muscaria, which the authors caution is poisonous if ingested without specific preparation.Source: §E.1 · p. 19, Figure 11 · p. 20, Figure 12 · p. 20
E.2 Datasets statistics
- Table 5 compares dataset properties; Megascans Plants and Food have more complex geometry and better camera coverage than FFHQ or Cats.CARLA statistics are included for comparison only, not as a benchmark, because the dataset is small and has simple geometry and texture.Source: Table 5 · p. 21, §E.2 · p. 21
F Additional samples
The appendix provides random samples from EpiGRAF and GRAM, then discusses potential societal impacts and ethical considerations. It flags deepfake misuse while noting that the model does not yet produce perceptually indistinguishable media.
F Additional samples
- The appendix provides non-cherry-picked EpiGRAF samples and additional GRAM samples illustrating its mode collapse on Megascans Plants and Food.The figure caption attributes GRAM’s collapse to its use of the same iso-surfaces for every sample, which struggles with variable structure.Source: F Additional samples · p. 21, Figure 14 · p. 22, Figure 15 · p. 22
G Potential negative societal impacts
- Deepfake creation is a potential concern, although the authors say EpiGRAF’s current quality is not perceptually indistinguishable from real media.The concern is framed as relevant to further development of media-synthesis technology.Source: G Potential negative societal impacts · p. 21
H Ethical concerns
- The authors state that the released datasets contain no personally identifiable information and are not biased against groups because they are not human-derived.They report reviewing ethics guidelines and say the work complies.Source: H Ethical concerns · p. 23