3D generation on ImageNet

1 INTRODUCTION

• Existing frameworks assume aligned, same-category scenes, making domain-specific cropping, rotation, and scaling unsuitable for multi-category datasets without a canonical position.The paper gives a panorama and a spoon as examples that cannot be aligned to one canonical position.Source: §1 · p. 2
• 3DGP uses imperfect monocular depth predictions as generic 3D cues to learn reasonable scenes from loosely curated, non-aligned datasets such as ImageNet.The approach is intended to extend 3D synthesis to in-the-wild datasets.Source: §1 · p. 2
• Its six-degree-of-freedom Ball-in-Sphere camera model learns distribution parameters and uses a gradient penalty to prevent collapse to a delta distribution.The model targets variation in focal length and look-at position, including close-up snouts and full-body dog images.Source: §1 · p. 2
• Adversarial depth supervision concatenates depth with RGB; a shallow adaptor bridges rendered and estimated depth distributions to guide geometry learning.Real-image depth comes from an off-the-shelf monocular predictor, while fake-image depth is rendered from the synthesized radiance field.Source: §1 · p. 2
• Discriminator distillation transfers pre-trained ResNet50 features and improves FID for both 2D and 3D generators by at least 40%.The technique adds 1% computational overhead compared with standard training.Source: §1 · p. 3
• 3DGP achieves better image appearance and geometry quality than modern 3D-aware generators on three non-aligned single-category datasets, and demonstrates multi-category synthesis on ImageNet.The single-category datasets are SDIP Dogs, SDIP Elephants, and LSUN Horses.Source: §1 · p. 3

2 RELATED WORK

• NeRF-based 3D-aware generators use RGB supervision and GANs, with efficiency strategies including patch-wise training, MPI rendering, and separate 2D upsampling.NeRF was introduced by Mildenhall et al. (2020); the passage lists Schwarz et al. (2020), Meng et al. (2021), and Skorokhodov et al. (2022) for patch-wise training.Source: §2 · p. 3
• Unlike methods that estimate or fine-tune poses from multi-view observations, this work learns a camera-pose distribution from sparse single-view images of diverse categories.CAMPARI (Niemeyer & Geiger, 2021a) is identified as the closest work because it also learns a camera distribution.Source: §2 · p. 3
• Prior GAN methods transfer generic image-model knowledge, but their encoder-input assumptions do not suit patch-wise training or depth-channel discriminator inputs.3DGP instead distills features from an image-classification model into its discriminator.Source: §2 · p. 3
• Unlike GSN, which uses ground-truth depth, 3DGP uses predicted depth and adapts it to account for estimator imprecision.DepthGAN applies predicted depth to guide a 2D GAN, while concurrent work by Yu et al. (2022) uses monocular depth for geometry reconstruction.Source: §2 · p. 3

3 METHOD

• The generator builds tri-planes from latent codes, renders images and depth at sampled camera poses, and gives the discriminator RGB plus depth, with ResNet feature distillation.It builds on EpiGRAF’s patch-wise training; the camera generator is trainable.Source: §3 · pp. 3–4

3.1 LEARNABLE “BALL-IN-SPHERE” CAMERA DISTRIBUTION

• The six-degree-of-freedom Ball-in-Sphere camera adds field of view and an inner-sphere look-at point to the conventional outer-sphere camera parameters.The camera distribution is learned during training, with class-conditional datasets conditioning on class label and latent code.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 ADVERSARIAL DEPTH SUPERVISION

• Adversarial depth supervision concatenates rendered or estimated depth with RGB, while a learnable adaptor bridges the depth domains and models estimator artifacts.The adaptor is kept to three convolutional layers because an overly powerful adaptor could fake depth and prevent geometry learning.Source: §3.2 · p. 5

3.3 KNOWLEDGE DISTILLATION FOR DISCRIMINATOR

• Discriminator distillation matches its real-sample feature representation to a pretrained ResNet representation, providing knowledge transfer for the modified discriminator architecture.The approach is presented as an alternative to using a pretrained model as the discriminator backbone, which does not suit the added depth input and patch conditioning.Source: §3.3 · p. 6

3.4 TRAINING

• Generator training combines adversarial loss with Camera Gradient Penalty, weighting camera-origin parameters more strongly than field-of-view and look-at parameters.The reported weights are 0.3 for camera position, 0.03 for field of view, and 0.003 for look-at; using P(d̄) = 0.5 gave better geometry than equal-probability depth sampling.Source: §3.4 · pp. 6–7

4 EXPERIMENTAL RESULTS

• Experiments cover SDIP Dogs40k, SDIP Elephants, LSUN Horses40k, and ImageNet, with metrics always measured on the full ImageNet.The ImageNet training set filters out two-thirds of its images; the single-category dataset filtering reduces SDIP Dogs and LSUN Horses to 40K samples each.Source: §4 · p. 7
• FID measures image quality, while NFS is the average entropy of normalized depth-map histograms and helps quantify flatness, not overall geometry quality.Figure 6 shows repetitive and diverse geometry alongside their depth histograms and NFS.Source: §4 · p. 7, Figure 6 · p. 7

4.1 3D GENERATION FOR SINGLE CATEGORY DATASETS

• At least 2× better FID2k than EG3D and greater NFS on all three datasets; 3DGP renders realistic side views across them.It is slightly worse than StyleGAN2 on FID2k; widening the baseline camera range reduced flatness but worsened FID2k by up to 500%.Source: §4.1 · p. 7, Table 1 · p. 8
• P(d̄) = 0.5 gives the best geometry and side views; P(d̄) = 0 and 1 cannot render side views.Without adversarial depth supervision, NFS is worst; when the adaptor is never used, the discriminator can exploit the domain gap and FID falls.Source: §4.1 · pp. 7–8
• Knowledge distillation adds stability to adversarial training and significantly improves FID, though classification-network strategies can improve FID without improving visual quality.Source: §4.1 · p. 8
• Camera regularization with a gradient penalty yields a wider posterior and valid geometry, unlike no regularization or residual-based prediction, which collapse to nearly deterministic distributions.A later appendix discusses entropy maximization as a simpler, more flexible regularization strategy.Source: §4.1 · pp. 8–9, Figure 7 · p. 8

4.2 3D SYNTHESIS ON IMAGENET

• 3DGP produces geometry with rich details on ImageNet, while EG3D and EpiGRAF produce flat or repetitive geometry.The 2D generators report better FID and IS than 3D generators; StyleNeRF and VolumeGAN substantially under-perform in visual quality.Source: §4.2 · p. 9, Table 2 · p. 9
• 3DPhoto lifting of StyleGAN-XL works well for negligible camera variations but starts diverging when the camera moves more than 10°; FID/IS scores use a narrow camera distribution, σyaw ∼N(0, 0.3) and σpitch ∼N(π/2, 0.15).The test generated 10k images with StyleGAN-XL.Source: §4.2 · p. 9

• 3DGP is presented as the first 3D synthesis framework for in-the-wild, multi-category datasets such as ImageNet.It uses imprecise monocular depth and latent feature representations as generic priors, plus a regularized Ball-in-Sphere camera model.Source: §5 · p. 9
• The reported limitations are sticking backgrounds, lower visual quality than 2D generators, and no reliable quantitative measure of generated geometry.The project consumed approximately 12 Nvidia A100 GPU years.Source: §5 · p. 9

• The planned release includes source code, generator checkpoints, and pre-processed datasets with extracted depth maps.The paper states that architecture and optimization details are provided in Sections 3 and 4, Appendix B, and figures throughout the text.Source: §6 · p. 10

• The authors flag misleading fake content and licensing, authorship, or distribution abuse as ethical concerns for synthesis.They note that the generator would still need improved synthesis quality to fool an attentive human observer and encourage discussion of prevention mechanisms.Source: §7 · p. 10

A LIMITATIONS

• 3D generators still have lower visual quality than 2D generators despite providing a more reasonable scene representation.Source: A LIMITATIONS · p. 16
• Background sticking merges foreground and background into one shape; frontal-photo bias and tri-plane bias toward flat geometry are proposed causes.Source: A LIMITATIONS · p. 16
• Non-Flatness Score captures tri-plane geometry flatness, not geometry quality across varied failure cases.Source: A LIMITATIONS · p. 16
• Camera generator C fails to learn class-specific focal-length control and instead models with larger fields of view, attributed to tri-plane bias.Figure 8 plots learned ImageNet focal-length means and 0.05 and 0.95 quantiles.Source: A LIMITATIONS · p. 16, Figure 8 · p. 16

B IMPLEMENTATION DETAILS

• The generator maps latent codes and class labels through an MLP to StyleGAN2-style tri-planes, which a decoder uses to predict color and density.Source: B IMPLEMENTATION DETAILS · p. 16
• Camera Gradient Penalty uses |g| + 1/|g| to discourage constant or extreme camera-generator outputs that collapse into a delta distribution.The passage says SoftPlus is crucial because the penalty is unstable for non-smooth activations at small learning rates.Source: B IMPLEMENTATION DETAILS · p. 17
• Outlier removal fits Gaussian models to image features, using separate class-wise models with diagonal covariance regularization for ImageNet.Source: B IMPLEMENTATION DETAILS · pp. 17–18

C KNOWLEDGE DISTILLATION

• Projected D varies heavily by generator architecture, costs more to train, and is inferior in visual quality and cost for 3DGP; KL and L2 perform similarly, while CLIP guidance overtakes ResNet later.The comparison table covers ImageNet and SDIP Dogs40k; Figure 10 explores distillation objectives for conditional ImageNet generation at 1282 resolution.Source: C KNOWLEDGE DISTILLATION · p. 18, Table 4 · p. 19, Figure 10 · p. 19

D FAILED EXPERIMENTS

• Depth-adaptor gradient penalties did not enforce good geometry, even with large loss weights and a shallow adaptor.Source: D FAILED EXPERIMENTS · p. 19
• Projected D produced significantly inferior learned geometry, which was often completely flat, across the tested discriminator setups.Source: D FAILED EXPERIMENTS · p. 19
• MLP representations consistently yielded significantly inferior visual quality; the authors hypothesize they lack capacity for large-scale datasets.Source: D FAILED EXPERIMENTS · pp. 19–20
• InfoNeRF ray-entropy regularization always diverged; the authors hypothesize sharp object densities can trap geometry learning in local minima.Source: D FAILED EXPERIMENTS · p. 20
• Normal supervision was inferior to depth supervision, and the generator could more easily fake normals.Source: D FAILED EXPERIMENTS · p. 20
• Variance, entropy, and moment regularizers let the generator cheat with delta or mixtures of delta distributions, producing flat images and geometry.Source: D FAILED EXPERIMENTS · p. 20

F NON-FLATNESS SCORE DETAILS

• NFS averages depth-histogram entropy over 256 sampled radiance fields using 64 bins, as a proxy for flatness rather than overall geometry quality.It also gives low scores to repetitive EG3D geometry despite the expectation that a wide camera distribution would spread shapes along rays.Source: F NON-FLATNESS SCORE DETAILS · p. 20

G INVESTIGATING THE POTENTIAL DATA LEAK FROM DEPTH ESTIMATOR

• LeReS combines parts of four datasets spanning outdoor scenes, in-the-wild imagery, and indoor scenes; its pretraining data contains almost no animal images.Holopix50k is described as the closest of these datasets to ImageNet in underlying data distribution.Source: G INVESTIGATING THE POTENTIAL DATA… · pp. 24–25
• LeReS pretraining contains 3.79% animal images versus 48.3% in ImageNet, and animal-subset FID follows the same trend as full-ImageNet FID.The authors conclude that adversarial depth supervision does not give the generator an unfair advantage through implicit access to additional animal data.Source: G INVESTIGATING THE POTENTIAL DATA… · p. 25, Table 5 · p. 25, Table 6 · p. 25

H GEOMETRY EVALUATION ON SHAPENET

• The ShapeNet study examines whether adversarial depth supervision improves geometry quality.The authors conclude that the depth estimator’s training data does not help through data leakage, but provides geometric guidance.Source: G INVESTIGATING THE POTENTIAL DATA… · p. 25, H GEOMETRY EVALUATION ON SHAPENET · p. 26

H.1 RENDERING DETAILS

• The ShapeNet rendering uses 51,209 single-view training images from random frontal camera positions, with transparent mesh elements removed to avoid depth-map aliasing.The dataset comprises 50,000 models from 54 classes before broken models are discarded.Source: §H.1 · p. 26

H.2 EXPERIMENTS

• The study trains EG3D and EpiGRAF with ground-truth camera parameters, and tests 3DGP with default or blurred adversarial depth supervision.Camera-distribution learning and knowledge distillation are disabled to isolate adversarial depth supervision; StyleGAN2 serves as an FID lower bound.Source: §H.2 · p. 26
• Evaluation uses FID for images and Frechet Pointcloud Distance (FPD) for point-cloud distributions, with PointNet features for FPD.The authors instead extract FPD features with pretrained PointNet++ after observing unusually high scores from the original custom PointNet implementation.Source: §H.2 · pp. 26–27
• The point-cloud pipeline extracts surfaces with marching cubes from 256^3 density fields, then uniformly samples 2,048 surface points per mesh.The marching-cubes surface threshold is density σ = 10.Source: §H.2 · p. 27

H.3 RESULTS

• 3DGP learns proper geometry and drops the background, while blurred depth maps deteriorate both geometry and texture quality.EG3D produces inverted, hollow shapes; EpiGRAF recovers shapes but models the white background as an additional sphere.Source: §H.3 · p. 27, Table 7 · p. 27

I DETAILS ON THE EXPERIMENTS WITH 3D PHOTO INPAINTING

• The experiments pair StyleGAN-XL with 3D Photo, generating 10,000 image variations using narrow yaw and pitch distributions.The authors first generated 50,000 StyleGAN-XL images and measured FID 2.51, compared with the authors’ reported 2.26.Source: I DETAILS ON THE EXPERIMENTS WITH 3D… · p. 27
• For negligible camera variations, 3D Photo produces excellent results, but larger movements quickly bring inpainting artifacts and unfilled outpainting holes.The approach inherits the 2D generator’s image quality but is not perfectly view consistent.Source: I DETAILS ON THE EXPERIMENTS WITH 3D… · p. 29

J ENTROPY REGULARIZATION

• The entropy-style camera regularizer minimizes each camera parameter’s distribution distance from a uniform distribution over its minimum-to-maximum range.The Earth Mover’s Distance is approximated with 64 samples.Source: J ENTROPY REGULARIZATION · p. 29