Source-linked AI summary
CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
Peng Zhou, Lingxi Xie, Bingbing Ni, Qi Tian
TL;DR
Existing GANs lack precise camera-pose control, while NeRF-based GANs have struggled to produce high-quality images. CIPS-3D combines shallow NeRF with deep INR for independent pixel synthesis, addresses mirror symmetry with an auxiliary discriminator, and achieves strong high-resolution 3D-aware generation results.
Problem
Existing GANs lack explicit camera-pose control, while prior NeRF-based GANs are limited in image quality or affected by artifacts.
Method
CIPS-3D combines a shallow NeRF for pose control with a deep INR that synthesizes pixels independently without spatial convolution or upsampling, using an auxiliary discriminator to address mirror symmetry.
Results
6.97 FID at 256^2 and 12.26 FID at 1024^2 on FFHQ set state-of-the-art 3D-aware synthesis results, surpassing StyleNeRF.
Takeaways & Limitations
CIPS-3D supports high-resolution 3D-aware synthesis and demonstrates transfer learning and 3D-aware face stylization.
Takeaways & Limitations
Current stylization is limited to randomly generated images because editing real images requires 3D-aware GAN inversion, which is left for future work.
Abstract
from arXiv · showhide
The style-based GAN (StyleGAN) architecture achieved state-of-the-art results for generating high-quality images, but it lacks explicit and precise control over camera poses. The recently proposed NeRF-based GANs made great progress towards 3D-aware generators, but they are unable to generate high-quality images yet. This paper presents CIPS-3D, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation (INR) network. The generator synthesizes each pixel value independently without any spatial convolution or upsampling operation. In addition, we diagnose the problem of mirror symmetry that implies a suboptimal solution and solve it by introducing an auxiliary discriminator. Trained on raw, single-view images, CIPS-3D sets new records for 3D-aware image synthesis with an impressive FID of 6.97 for images at the $256\times256$ resolution on FFHQ. We also demonstrate several interesting directions for CIPS-3D such as transfer learning and 3D-aware face stylization. The synthesis results are best viewed as videos, so we recommend the readers to check our github project at https://github.com/PeterouZh/CIPS-3D
1. Introduction
CIPS-3D addresses the limited camera control and low image quality of existing GAN-based approaches with a 3D-aware generator that combines shallow NeRF and deep INR networks. It also resolves mirror symmetry and achieves strong high-resolution synthesis results across several datasets.
- Existing GANs generate high-fidelity images but lack explicit viewpoint control, while latent editing changes pose only roughly and cannot render arbitrary camera views.
- Previous 3D-aware generators provide camera-pose control but are often restricted to low-resolution images with artifacts.Pure NeRF generators can produce blurred images because memory demands limit generator depth, while NeRF-plus-CNN designs can suffer aliasing from upsampling filters.
- CIPS-3D combines a shallow NeRF for explicit camera-pose control with a deep INR that synthesizes each pixel independently without spatial convolution or upsampling.The architecture assigns pose to early NeRF layers and semantic attributes and color to later INR layers.
- An auxiliary discriminator solves CIPS-3D’s mirror symmetry problem, while partial gradient backpropagation enables high-resolution training and an efficient ModFC implementation speeds training.
- 6.97 FID at 256^2 and 12.26 FID at 1024^2 on FFHQ establish state-of-the-art 3D-aware synthesis results, surpassing StyleNeRF.The method was evaluated on FFHQ, MetFaces, BitmojiFaces, CartoonFaces, and AFHQ, and also demonstrated transfer learning and 3D-aware face stylization.
2. Related Work
Related 3D-aware generators combine GANs with NeRF, but CIPS-3D avoids their low-resolution feature-map upsampling design. It separates 3D shape from appearance using NeRF and INR networks, supporting transfer learning.
- Implicit neural representations model scenes continuously and memory-efficiently compared with mesh- and voxel-based representations.They are commonly implemented with multilayer perceptrons and have been used in both 3D and 2D tasks.
- Unlike GIRAFFE and StyleNeRF, CIPS-3D synthesizes every pixel independently without upsampling or spatial convolution operations.
- CIPS-3D represents 3D shape with NeRF and appearance with INR, a decomposition described as convenient for transfer learning.
3. Method
CIPS-3D combines a shallow NeRF network for 3D shape with a deep INR network for appearance, synthesizing pixels independently without spatial convolution or upsampling. It also uses partial gradient backpropagation and an auxiliary discriminator to enable high-resolution training and address mirror symmetry.
- NeRF Network for 3D Shape: The NeRF network represents 3D shape while the deep INR network converts per-ray features into pixel appearance.The NeRF is shallow to reduce memory, whereas the INR is deep to increase generator capacity.
- NeRF Network for 3D Shape: A conditional shape code modulates the NeRF network so different sampled codes produce different 3D shapes.The shape code is shared across all pixels in a generated image.
- INR Network for Appearance: Each pixel is synthesized independently from a volume-rendered feature vector using the deep INR network, without spatial convolution or upsampling.The INR contains nine blocks with modulated fully connected layers and produces RGB values from per-ray features.
- INR Network for Appearance: Partial gradient backpropagation enables high-resolution training by computing gradients for randomly sampled rays while disabling them for the remaining rays.All ray outputs are combined into a high-resolution image for the discriminator, while only the sampled rays retain gradient computation.
- Overcoming Mirror Symmetry Issue with Auxiliary Discriminator: The auxiliary discriminator supervises NeRF outputs directly, eliminating mirror symmetry that persists when a 2D network lies between NeRF and the main discriminator.The problem is linked to symmetry in input coordinates and can be reinforced by non-distance-preserving positional encoding.
4. Experiments
Experiments show that CIPS-3D achieves strong image quality, training efficiency, transferability, and explicit pose control. Ablations support the auxiliary discriminator, partial gradient strategy, and NeRF–INR division of labor.
- Comparison with SOTA: 6.97 FID at 256^2 and 12.26 FID at 1024^2 establish state-of-the-art 3D-aware synthesis on FFHQ.At 256^2, CIPS-3D also surpasses StyleNeRF in FID and KID.
- Progressively Growing Training: CIPS-3D remains better than pi-GAN during progressive training, especially at higher resolution.The pure-NeRF pi-GAN generator is memory-intensive, deteriorates at 128^2, and is sensitive to hyperparameters.
- Efficiency Comparison: CIPS-3D trains faster than pi-GAN despite having more parameters, while bmm increases ModFC speed from 15.18 to 27.78 batches/second.The efficiency advantage is attributed to the 2D INR network being more efficient to train than the 3D NeRF network.
- Ablation Studies: The auxiliary discriminator solves mirror symmetry and improves FID, while the final method uses coordinates, learnable positional encoding, and the auxiliary discriminator.The learnable positional encoding alone deteriorates FID relative to fixed encoding, but the auxiliary discriminator reverses that outcome.
- Ablation Studies: At 128^2, using 96^2 gradient pixels performs comparably to using all 128^2 pixels; fewer pixels can retain performance only with more iterations.Too few pixels during partial gradient backpropagation harms performance.
- Transfer Learning: The NeRF network preserves pose while the INR network adds richer textures, supporting their respective roles in shape and appearance.Freezing the NeRF network enables transfer learning by fine-tuning only the INR network.
- 3D-aware Face Stylization: CIPS-3D transfers across four face and animal datasets and explicitly controls the pose of generated faces.Interpolating INR layers produces domain blends or combines content structure with style characteristics.
5. Limitation and Future Work
CIPS-3D currently generates images from noise rather than editing real images. Real-image editing requires 3D-aware GAN inversion, which the paper leaves for future work.
- CIPS-3D stylization is limited to randomly generated images because the model operates in a noise-to-image manner.Editing a real image requires projecting it into the generator’s latent space through 3D-aware GAN inversion.
6. Conclusion
CIPS-3D is a style-based 3D-aware generator that independently synthesizes pixels without spatial convolution or upsampling. It addresses coordinate-induced mirror symmetry with an auxiliary discriminator.
- CIPS-3D independently synthesizes pixel values without spatial convolution or upsampling and uses an auxiliary discriminator to solve mirror symmetry.The paper attributes mirror symmetry to the symmetry of the input coordinates.