PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt

arXiv:2110.08985v1cs.CVstat.ML

TL;DR

High-resolution, multi-view-consistent image synthesis from unstructured 2D images remains difficult, especially with explicit camera and style control. StyleNeRF integrates NeRF into a style-based generator and achieves photo-realistic high-resolution synthesis at interactive rates while supporting camera and style control.

  • Problem

    Existing image generators lack 3D understanding and multi-view consistency, while NeRF-based methods are computationally costly for high-resolution synthesis and often lack style control.

  • Method

    StyleNeRF combines NeRF with a style-based generator, rendering low-resolution features before progressive 2D upsampling with consistency-preserving designs and progressive training.

  • Results

    StyleNeRF synthesizes photo-realistic 1024^2 images at interactive rates with high multi-view consistency, outperforming prior 3D-aware methods across image-quality metrics and supporting camera and style control.

  • Takeaways & Limitations

    The model supports high-resolution view-controlled generation and applications including style mixing, interpolation, inversion, and semantic editing.

  • Takeaways & Limitations

    StyleNeRF recovers coarser geometry with fewer details than pure NeRF methods and empirically preserves, rather than strictly guarantees, multi-view consistency.

Abstract

from arXiv · show

We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks, including zoom-in and-out, style mixing, inversion, and semantic editing.

1 INTRODUCTION

StyleNeRF is a 3D-aware, style-based generative model for photo-realistic high-resolution image synthesis from unstructured 2D images. It uses low-resolution NeRF feature rendering with progressive 2D upsampling to achieve interactive rates, high multi-view consistency, camera-pose and style control.

  • Motivation: Traditional graphics pipelines require production-quality 3D models, expensive rendering, and manual work, whereas GANs can train on large collections of unstructured images.These limitations motivate learning-based 3D-aware generation from images.
  • Proposed model: NeRF produces only a low-resolution feature map, which StyleNeRF progressively upsamples to avoid expensive direct high-resolution color rendering.This design addresses NeRF’s computationally expensive rendering process.
  • Results: StyleNeRF synthesizes photo-realistic 1024^2 images at interactive rates while achieving high multi-view consistency.The paper states that existing methods do not achieve both characteristics simultaneously.
  • Applications: The model controls styles and 3D camera poses, including poses starkly different from training, and supports style mixing, interpolation, inversion, and semantic editing.These capabilities are evaluated on various challenging datasets.

2 RELATED WORK

Related work spans neural implicit fields, GAN-based image synthesis, and 3D-aware GANs. Prior methods provide 3D representations, style control, or camera control, but face limitations involving detail, consistency, or high-resolution training.

  • Neural Implicit Fields: Neural implicit-field methods represent 3D scenes with learned fields, using 3D supervision or differentiable renderers for implicit surfaces.The passage also introduces NeRF and its variants, but the supplied text is truncated before describing them further.
  • Image Synthesis with GANs: GANs have demonstrated high-quality image synthesis, while StyleGANs achieve SOTA quality and support control over different style levels.The passage also notes work addressing the “texture sticking” problem in 2D GAN animations with 2D transformations.
  • 3D-Aware GANs: 3D-aware GANs integrate neural scene representations into 2D generators for direct camera control, but voxel methods lack fine details and radiance-field methods struggle with high-resolution training.Radiance-field methods are described as having higher quality and better 3D consistency, while expensive rendering makes training difficult for 5122 and beyond.

3 METHOD

StyleNeRF conditions a NeRF on StyleGAN2-like style vectors and renders high-resolution images efficiently by aggregating features early, volume-rendering at low resolution, and progressively upsampling in 2D. It addresses the resulting multi-view inconsistency with a specialized upsampler, NeRF path regularization, and progressive training.

  • Style-based Generative Neural Radiance Field: StyleNeRF maps Gaussian noise to style vectors that modulate NeRF MLP layers, whose features predict density and view-dependent color.The first min(nσ, nc) layers are shared, with nc > nσ because visual appearance generally requires more capacity than geometry.
  • Efficient high-resolution rendering: StyleNeRF early-aggregates ray features, volume-renders a low-resolution feature map, and recursively upsamples it in 2D to synthesize high-resolution images efficiently.Using fewer channels at higher resolutions further reduces computation.
  • Upsampler design: Because 2D aggregation and upsampling can introduce multi-view inconsistency, StyleNeRF designs the upsampler to avoid coordinate-dependent chessboard, texture-sticking, and bubble artifacts.Pixel-wise learnable upsamplers can create chessboard or texture-sticking artifacts, while fixed low-pass interpolation can produce bubble artifacts.
  • NeRF path regularization: NeRF path regularization enforces 3D consistency by comparing randomly subsampled StyleNeRF output pixels with corresponding pixels generated along the original NeRF path.The comparison uses corresponding low-resolution and high-resolution rays.
  • Progressive Training: StyleNeRF trains progressively from low to high resolution because directly training at the highest resolution can fail to capture object geometry.The method uses a new three-stage progressive training strategy because the accelerated rendering equations approximate the original NeRF.

4 EXPERIMENTS

StyleNeRF is evaluated against voxel-, radiance-field-, and 2D-GAN baselines on four high-resolution image datasets, with comparisons showing strong image quality, high-resolution synthesis, camera control, and style manipulation. Ablations and limitations reveal dependencies for multi-view consistency and reduced geometric detail relative to pure NeRF methods.

  • Evaluation setup: StyleNeRF is evaluated on FFHQ, MetFaces, AFHQ, and CompCars against HoloGAN, GRAF, π-GAN, GIRAFFE, and StyleGAN2, with comparisons made at 256^2 pixels.StyleGAN2 is included for reference, while most baselines are restricted to low resolutions.
  • Quantitative comparison: Across FFHQ, AFHQ, and CompCars, StyleNeRF consistently outperforms the baselines on FID and KID and narrows the gap between 3D-aware GANs and StyleGAN2.The quantitative comparison uses FID and KID to measure image-generation visual quality.
  • High-resolution synthesis: StyleNeRF generates high-resolution images at 512^2 and beyond, with Figure 5 showing uncurated samples from three datasets rendered from randomly sampled cameras.The reported high-resolution capability distinguishes StyleNeRF from the baseline models in the comparison.
  • Explicit camera control: StyleNeRF supports direct camera control, including zooming and steep view angles, while maintaining consistency for views that differ strongly from the training camera distribution.The results include extreme camera poses and novel views.
  • Style mixing and editing: Style mixing before 2D aggregation changes geometry-related attributes, whereas mixing afterward changes appearance, indicating disentangled geometry and appearance styles.Style interpolation also produces smooth transitions between different styles, and inversion enables semantic editing after estimating camera pose and optimizing styles.
  • Ablations and limitations: Ablations show that removing progressive training, NeRF-path regularization, or view-direction conditioning can produce incorrect or flat geometry and severe 3D-inconsistent artifacts.StyleNeRF recovers coarse geometry but less detail than pure NeRF methods such as π-GAN, and its multi-view consistency is empirical rather than strictly guaranteed.

5 CONCLUSION

StyleNeRF is a 3D-aware generative model for efficient, high-resolution image generation with high 3D consistency. It supports explicit 3D camera-pose and style-attribute control, synthesizing photo-realistic 1024^2 images at interactive rates while outperforming prior 3D-aware methods.

  • 5 CONCLUSION: StyleNeRF generates high-resolution images efficiently while maintaining high 3D consistency.The model is designed for 3D-aware image generation.
  • 5 CONCLUSION: The model provides control over explicit 3D camera poses and style attributes.
  • 5 CONCLUSION: 1024^2 photo-realistic images are synthesized at interactive rates, outperforming previous 3D-aware generative methods.

ETHICS STATEMENT

StyleNeRF is intended for technical and creative applications without subject-specific bias, but its realistic image synthesis raises disinformation and privacy concerns. The authors therefore advocate synthetic labeling, privacy preservation, authenticity assessment, detection systems, and public regulation.

  • Applications and fairness: StyleNeRF targets technical and creative uses, including movie post-production, gaming, artist assistance, and synthetic-data generation, without bias toward specific demographic groups.The approach is described as working equally well across differences in gender, race, region, social class, and subjects.
  • Risks and safeguards: Realistic synthesis that can be difficult to distinguish from source images raises disinformation risks, including fake image and video generation.The authors state that synthesized images should present themselves as synthetic and call for privacy-preserving techniques and large-scale authenticity assessment.
  • Detection and governance: The authors hope StyleNeRF’s high-quality outputs will support forgery detection and verification, alongside public discussion of regulations that reduce misuse while preserving technological benefits.They identify robust public conversation as essential to creating appropriate regulations and laws.

A ADDITIONAL IMPLEMENTATION DETAILS … A.3 DETAILS ABOUT THE NOISE INJECTION

The implementation details specify NeRF++ background modeling, hierarchical foreground sampling, camera parameterization, and a geometry-aware noise-injection strategy designed to preserve multi-view consistency while improving visual quality. The method also addresses the limitations of removing or independently injecting noise across views.

  • A.1 IMPLICIT FIELDS MODELING: Complex, unbounded backgrounds are partitioned into foreground and background, with the background modeled by an additional network following NeRF++.This avoids modeling the entire embedded scene within a fixed bounding box, which is inefficient for such images.
  • A.1 IMPLICIT FIELDS MODELING: Background points are uniformly sampled in the inverse-depth range [1/R, 0), with R = 2.0 defining where the background starts.The sampling range represents background locations in the scene.
  • A.1 IMPLICIT FIELDS MODELING: Foreground learning samples N points uniformly between near and far planes, then importance-samples M points from the estimated coarse density distribution.Unlike NeRF, a single network predicts color and density for both coarse and fine sampling.
  • A.2 CAMERA: Each image uses a camera with fixed intrinsics and a pose sampled from a predefined distribution, parameterized by pitch θ and yaw φ on the unit sphere.Intrinsics are determined by field-of-view and resolution, then normalized to [−1, 1].
  • A.3 DETAILS ABOUT THE NOISE INJECTION: Naive per-pixel noise is unsuitable because it remains fixed in pixel space across viewpoint changes, while independently sampled frame noise produces noisy outputs.The implementation therefore removes noise injection by default, similarly to Karras et al. (2021).
  • A.3 DETAILS ABOUT THE NOISE INJECTION: Removing noise injection works reasonably but reduces the ability to model local stochastic variations, resulting in lower FID, KID, and related visual-quality metrics.The limitation motivates an alternative noise-injection design.
  • A.3 DETAILS ABOUT THE NOISE INJECTION: The geometry-aware noise method extracts geometry with marching cubes, assigns independent Gaussian noise to mesh vertices, and rasterizes the noise map from the matching viewpoint.For each feature layer at resolution N^2, the volume resolution N^3 is set accordingly before noise rendering.

A.4 PROGRESSIVE TRAINING · A.5 HYPERPARAMETERS

StyleNeRF is trained progressively from low to high resolutions to learn good geometry, using either progressive growing or progressive up-sampling; the reported results use progressive growing because it is faster for low-resolution images. The appendix also specifies StyleGAN2-based defaults and field-network hyperparameters.

  • A.4 PROGRESSIVE TRAINING: Progressive training from low to high resolutions is essential for StyleNeRF to learn good geometry.The paper considers two progressive-training variants.
  • A.4 PROGRESSIVE TRAINING: Progressive growing starts with a shallow low-resolution network and fades in high-resolution layers using linear interpolation to stabilize training.The interpolation combines the low-resolution output with newly added layers whose parameters are random.
  • A.4 PROGRESSIVE TRAINING: Progressive up-sampling inserts upsample operations during training instead of growing new layers, so linear layer fading is unnecessary.This follows the progressive insertion strategy described for StyleNeRF.
  • A.4 PROGRESSIVE TRAINING: Both the generator and discriminator grow progressively from low to high resolutions, and both training methods achieve similar visual quality.The discriminator correspondingly grows across resolutions in both variants.
  • A.4 PROGRESSIVE TRAINING: Progressive growing provides a better speed advantage for low-resolution images, so the reported results use progressive growing.Figure 11 illustrates generator progressive growing, progressive upsampler insertion, and the corresponding discriminator.
  • A.5 HYPERPARAMETERS: The mapping network and discriminator reuse StyleGAN2’s architecture and default parameters, including 8 fully connected mapping layers and a 100× lower learning rate.The passage states these defaults are reused for both the mapping network and discriminator.
  • A.5 HYPERPARAMETERS: The latent and style dimensions are both 512, while the foreground and background style-based MLPs use 256 and 128 hidden units, respectively.The hidden-unit counts apply before density prediction in the foreground and background fields.
  • A.5 HYPERPARAMETERS: Both foreground and background fields use Fourier feature dimension L = 10.This setting is specified with reference to Equation (1).

A.6 IMPLEMENTATION

StyleNeRF is implemented from the official PyTorch StyleGAN2-ADA implementation and trained for 25,000k images with minibatch 64 on eight Tesla V100 GPUs for about three days.

  • A.6 IMPLEMENTATION: The model follows the official PyTorch implementation of StyleGAN2-ADA and trains for 25,000k images with minibatch 64.Training uses eight Tesla V100 GPUs and takes about three days.

B DATASET DETAILS · C BASELINE DETAILS

The experiments use four datasets spanning human, animal, artwork, and car imagery, with dataset-specific resolutions and camera-sampling schemes. Baselines are implemented or retrained using official code, pretrained models, or adaptations for higher-resolution synthesis.

  • B DATASET DETAILS: FFHQ contains 70k real human-face images at 1024^2 resolution, with faces assumed at the origin and pitch and yaw sampled from Gaussian distributions.Training samples camera pitch and yaw from Gaussian distributions.
  • B DATASET DETAILS: AFHQ contains 15k cat, dog, and wild-animal images at 512^2, merged without labels while sampling pitch and yaw from Gaussian distributions.All training images are merged directly without using category labels.
  • B DATASET DETAILS: MetFaces contains 1,336 artwork-derived face images, resized from 1024^2 to 512^2, and StyleNeRF is finetuned from the FFHQ checkpoint because the dataset is small.The original resolution is 1024^2, while the main experiments use 512^2.
  • B DATASET DETAILS: CompCars contains 136,726 whole-car images, center-cropped and resized to 256^2, with camera viewpoints sampled uniformly across 360 degrees.The preprocessing resolves the original dataset’s varying aspect ratios.
  • C BASELINE DETAILS: HoloGAN is trained from its official implementation at 256^2 for 50 epochs using Adam with an initial learning rate of 5e-5 and linear decay from epoch 25 to 50.Because original HoloGAN supports only 64^2 and 128^2 training, the 256^2 setup follows the adaptation scheme used in GIRAFFE.
  • C BASELINE DETAILS: GRAF models are retrained with official code on FFHQ, AFHQ, and preprocessed CompCars at 256^2 using target-resolution training and a patch-based discriminator.The setup follows GRAF’s default configuration.
  • C BASELINE DETAILS: π-GAN models are retrained on FFHQ, AFHQ, and CompCars using a 32^2-to-128^2 progressive pipeline, with final 256^2 outputs produced by sampling more pixels.The progressive pipeline is used because high-resolution π-GAN execution is costly.
  • C BASELINE DETAILS: For GIRAFFE, pretrained 256^2 FFHQ models are used directly, while AFHQ and CompCars models are retrained officially with center-crop preprocessing.The default random-crop preprocessing is changed to center crop for comparison with other approaches.

D ADDITIONAL RESULTS

Additional results demonstrate StyleNeRF’s image quality and 3D consistency through quantitative comparisons, COLMAP reconstruction, and visualizations across camera views, datasets, and resolutions.

  • Comparison to StyleGAN2: StyleNeRF is quantitatively compared with StyleGAN2 and a comparable-setting variant for high-resolution image synthesis, but remains behind the best 2D-GAN models.The paper attributes the quality gap mainly to less powerful architecture, including no noise injection and no 3 × 3 convolutions.
  • Comparison to StyleGAN2: The results indicate that StyleNeRF achieves similar or better performance than 2D models with similar architectures.This comparison supports the claim that the remaining quality drop is primarily architectural rather than inherent to the proposed 3D-aware design.
  • 3D Reconstruction: On FFHQ, COLMAP reconstructs point clouds from 36 camera-posed images generated with fixed style, default parameters, and no known camera poses.This reconstruction example is intended to validate the 3D consistency of StyleNeRF outputs.
  • Additional Visual Results: Additional visual results show generated images across different camera views and provide further examples in the supplemental video.The figures include FFHQ 10242, FFHQ 5122, AFHQ 5122, MetFaces 5122, and CompCars 2562, including explicit and random camera poses.
Loading 2110.08985v1…