StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

Roy Or-El; Xuan Luo; Mengyi Shan; Eli Shechtman; Jeong Joon Park; Ira Kemelmacher-Shlizerman

Source-linked AI summary

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, Ira Kemelmacher-Shlizerman

arXiv:2112.11427v2Dec 21, 2021cs.CVcs.AIcs.GRcs.LG

TL;DR

StyleSDF targets the low resolution and geometric inconsistency of single-view-supervised 3D-aware GANs. It combines an SDF-based 3D renderer with a StyleGAN-based 2D generator to produce high-resolution, view-consistent images and detailed geometry, achieving higher visual and geometric quality than state-of-the-art methods.

Problem
Existing 3D-aware GANs struggle to generate high-resolution view-consistent images and well-defined geometry from single-view RGB data.
Method
StyleSDF renders low-resolution feature maps with a latent-conditioned SDF and radiance-field network, then uses a StyleGAN generator to produce high-resolution images.
Results
StyleSDF outperforms state-of-the-art 3D-aware methods on generated-image quality, surface quality, and view consistency on FFHQ and AFHQ.
Takeaways & Limitations
The method demonstrates that detailed 3D shape and view-consistent 1024×1024 images can be learned from single-view RGB images without 3D or multiview supervision.
Takeaways & Limitations
View-dependent color can overfit dataset biases, so inference fixes the view direction to the frontal view; strong lighting effects can also introduce depth dents.

Abstract

from arXiv · show

We introduce a high resolution, 3D-consistent image and shape generation technique which we call StyleSDF. Our method is trained on single-view RGB data only, and stands on the shoulders of StyleGAN2 for image generation, while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape. We achieve this by merging a SDF-based 3D representation with a style-based 2D generator. Our 3D implicit network renders low-resolution feature maps, from which the style-based network generates view-consistent, 1024x1024 images. Notably, our SDF-based 3D modeling defines detailed 3D surfaces, leading to consistent volume rendering. Our method shows higher quality results compared to state of the art in terms of visual and geometric quality.

1. Introduction

StyleSDF addresses the limited resolution and geometric inconsistency of existing 3D-aware GANs by combining SDF-based 3D modeling with StyleGAN generation using only single-view RGB supervision. It produces 1024×1024 view-consistent images and detailed geometry, outperforming prior 3D-aware methods on visual, geometric, and view-consistency measures.

StyleGAN produces high-quality RGB images but is designed for single views rather than 3D content.
StyleSDF generates 1024×1024 RGB images and geometry from single-view RGB training data.
Existing 3D generative models often require difficult-to-collect 3D or multiview supervision or remain limited to low-resolution rendering.Opacity fields used without multiview supervision can yield poorly defined surfaces and inconsistent depth across views.
StyleSDF combines an SDF-based 3D volume renderer with a 2D StyleGAN generator to transform low-resolution feature maps into high-resolution images.Adversarial and Eikonal losses support realistic images across viewpoints and proper SDF modeling without 3D or multiview supervision.
StyleSDF outperforms state-of-the-art 3D-aware methods in generated-image quality, surface quality, and view consistency on FFHQ and AFHQ.

2. Related Work

Related work progresses from high-resolution 2D synthesis and neural implicit representations toward single-view-supervised 3D-aware GANs. However, existing approaches remain constrained by volume-rendering cost, low output resolution, and imperfect 3D consistency.

2D image synthesis: GANs can synthesize high-resolution RGB images and support manipulation through latent directions, contrastive learning, inverse graphics, exemplars, and multiple views.
Neural implicit representations: Neural implicit representations learn shape and appearance from explicit 3D supervision or reconstruct structure from multiview 2D supervision with differentiable rendering.
Neural implicit representations: NeRF reconstructs scenes with neural radiance and density fields for novel-view synthesis, while SIREN uses sinusoidal activations for single-scene fitting.
Single-view-supervised 3D-aware GANs: Single-view-supervised 3D-aware GANs optimize 3D representations to render realistic images from randomly sampled viewpoints using adversarial loss.
Single-view-supervised 3D-aware GANs: GRAF and Pi-GAN are limited to low-resolution outputs because volume rendering is computationally expensive, while density fields can permit view-inconsistent geometry.
Single-view-supervised 3D-aware GANs: ShadeGAN and GOF improve surface quality, but ShadeGAN remains limited to 128×128 outputs; GIRAFFE reaches 256×256 yet lacks view consistency.

3. Algorithm

StyleSDF uses a conditional SDF volume renderer to produce low-resolution spatial features and a style-based 2D generator to create high-resolution images. Its SDF representation supports explicit surfaces and view-consistent geometry, while the training design addresses the computational cost and viewpoint biases of volume rendering.

3.1. Overview: The framework has two main components: a conditional SDF volume renderer and a 2D style-based generator, each paired with a mapping network.
3.1. Overview: The renderer takes latent and camera inputs, queries 3D points and view directions, and projects surface features into a 2D view for the styled generator.
3.2. SDF-based Volume Rendering: For each queried point, the renderer outputs an SDF value, view-dependent color, and feature vector conditioned on the latent code.
3.2. SDF-based Volume Rendering: The SDF is converted into a density field for volume rendering, with learned α controlling boundary sharpness from solid and sharp to more fluffy.
3.2. SDF-based Volume Rendering: The renderer produces 64×64 feature maps and color images by querying points along camera rays and approximating rendering through discrete sampling.
3.2. SDF-based Volume Rendering: SDFs define extractable surfaces and, with the associated losses, improve geometry expressiveness and view consistency even with simplified sampling.
3.2. SDF-based Volume Rendering: View-dependent color is retained during training but fixed to the frontal view at inference because it can overfit dataset biases and change expressions across viewpoints.
3.3. High-Resolution Image Generation: Pure volume rendering is impractical for 1024×1024 images because it requires sampling over 10^7 queries per image.

4. Experiments

Experiments evaluate StyleSDF on FFHQ and AFHQ against 3D-aware GAN baselines for image quality, geometry, novel-view synthesis, and depth consistency. The results report high-quality 3D shapes, realistic out-of-distribution views, and stronger consistency from the SDF-based representation.

Experimental Setup: StyleSDF is evaluated on FFHQ and AFHQ against GIRAFFE, PiGAN, GRAF, and HoloGAN using image, shape, and rendering-consistency measures.FFHQ contains 70,000 aligned 1024 × 1024 face images, while AFHQ contains 15,630 unaligned 512 × 512 animal images.
Qualitative Evaluations: StyleSDF generates superior images and more detailed 3D shapes than the compared baselines when rendering the same identity from four viewpoints.HoloGAN and GIRAFFE do not generate meshes; PiGAN meshes contain noticeable artifacts, while GIRAFFE exhibits viewpoint-dependent identity changes.
Image Quality: StyleSDF performs consistently better than all baselines in visual quality under the reported FID and KID evaluations.The comparison evaluates 256 × 256 outputs, with PiGAN trained at 128 × 128 and rendered at 256 × 256 during inference.
Novel View Synthesis: StyleSDF remains realistic when field of view changes by ±25% and camera angles reach 1.5 standard deviations beyond the training distribution.The model was trained with a fixed field of view, yet both increased and decreased FOV settings remain visually plausible.
Depth Consistency: The SDF representation dramatically improves depth consistency compared with PiGAN, whose depth maps show noise and spurious concave regions.Depth consistency is measured using average modified Chamfer distance over 1,000 frontal-side depth-map pairs, with distances normalized by the volume sampling bin size.
RGB View Consistency: Consistent volume rendering makes side-view RGB reprojections closely match frontal renderings, with remaining errors concentrated in high-frequency textures and geometry.The reprojection comparison ignores occluded pixels and measures mean absolute RGB pixel differences on a 0–255 scale.

5. Limitations & Future Work

StyleSDF has artifact and modeling limitations, including aliasing, lighting-induced depth dents, abrupt foreground-background transitions, and computational trade-offs in proposed improvements.

Limitations: Aliasing and flickering can appear in high-frequency regions such as teeth, while specularities and shadows may introduce artifacts or depth dents.The authors leave aliasing corrections and lighting-aware loss adjustments for future work.
Limitations: Using a single SDF for the whole image can make foreground-to-background transitions too abrupt, potentially reducing photorealism.The authors suggest adding an additional volume renderer for the background.
Future Work: End-to-end training could refine geometry but would increase GPU memory consumption, potentially requiring smaller batches or longer training.The proposed alternative would send StyleGAN2 discriminator gradients back to the volume renderer.
Future Work: Surface-boundary sampling could reduce query points and remove the 2D upsampling CNN, directly linking 3D geometry to high-resolution images.This is presented as a possible algorithmic improvement rather than a demonstrated result.

6. Conclusions

StyleSDF combines SDF-based volume rendering with a 2D StyleGAN network to generate high-resolution, view-consistent images and detailed geometry from single-view RGB data. Experiments report improved color consistency and strong view consistency, while the paper also notes societal risks and dataset bias concerns.

Conclusions: StyleSDF renders 1024x1024 view-consistent images together with detailed underlying geometry using only single-view RGB supervision.Its architecture combines an SDF-based volume renderer with a 2D StyleGAN network.
Societal Impacts: The technology may enable more realistic fake videos and can inherit ethnic, gender, or other biases from its training data.The paper suggests content detection and more balanced datasets as possible mitigation strategies.
Conclusions: StyleSDF reports significantly improved color consistency over PiGAN, with reprojection errors concentrated mostly in occluded regions.The comparison uses median per-pixel mean absolute RGB differences over 1,000 samples per dataset.
Conclusions: High-resolution reprojections closely replicate frontal identities and structures, while remaining errors are concentrated in StyleGAN high-frequency details such as hair, ears, and eyes.The reported errors are associated with aliasing in high-frequency components.

C. Qualitative 3D results

Qualitative results show that StyleSDF produces consistent depth maps and plausible SDF-extracted meshes across views, while reprojection errors are concentrated mainly near occlusion boundaries.

C. Qualitative 3D results: Reprojected side-view RGB renderings align well with frontal renderings, with errors mostly occurring at occlusion boundaries.The comparison uses depth maps to warp side-view RGB images into the frontal view.
C. Qualitative 3D results: Frontal and side-view depth-map point clouds show high consistency when overlaid.The visualization uses two different colors to distinguish the point clouds.
C. Qualitative 3D results: Meshes extracted from the SDF with marching cubes remain visually plausible when viewed from extreme angles.This qualitative inspection evaluates the underlying 3D models rather than only rendered RGB images.

D. Video Results

Video results use view-consistent RGB and depth sequences, while a 3D-attached noise strategy is introduced to reduce frame-to-frame flickering of high-frequency details.

D. Video Results: The supplementary material provides 24 RGB and depth video sequences across two datasets and two camera trajectories.Each identity receives one RGB-rendering video and one depth-rendering video.
D. Video Results: Naively applying StyleGAN2’s random 2D noise can cause severe flickering of high-frequency details between frames, especially for AFHQ fur textures.The problem is attributed to frame-varying high-frequency noise.
D. Video Results: High-resolution reprojection visualizations compare warped non-frontal pixels with frontal renderings and their error maps.The comparison identifies alignment and pixel-error patterns across viewpoints.
D. Video Results: The proposed mitigation attaches unit Gaussian noise to vertices of an SDF-extracted mesh and renders it using vertex coloring.The mesh is extracted at a 128-resolution grid, with triangle subdivision supporting higher-resolution intermediate features.

E.1. Dataset Details

The experiments use FFHQ and AFHQ with specified camera, sampling, and training settings, while the method uses SDF-based sampling and initialization choices to improve rendering and geometry.

Dataset and training settings: FFHQ uses a 12° field of view, Gaussian camera angles, 24 samples per ray, and separate 200k/300k iteration schedules for the volume renderer and styled generator.The azimuth and elevation standard deviations are 0.3 and 0.15, respectively.
Dataset and training settings: AFHQ combines three animal classes into one training set and uses a 12° field of view with Gaussian camera angles of standard deviation 0.15.Its near and far fields and samples per ray match the FFHQ setup.
Initialization: Sphere initialization fits the SDF MLP to analytically computed distances from an origin-centered sphere before main training.This procedure runs for 10k iterations and is intended to avoid concave-surface local minima.
Training strategy: Training first optimizes a 64×64 SDF volume renderer, then freezes it while training the 2D styled generator with StyleGAN2 settings.The first stage uses gradient accumulation under GPU memory constraints; the second stage trains on 8 Tesla V100 GPUs for 7 days.
Sampling: SDF-based sampling uses fewer samples without sacrificing surface quality, reducing memory footprints and simplifying implementation.Equal-length integration intervals also remove sampling noise and produce smoother volume-rendering outputs.

G. Ablation studies

The ablations show that minimal surface loss promotes smooth, less ambiguous geometry, while sphere initialization prevents concave solutions; high-frequency rendering and eye geometry remain limitations.

Minimal surface loss: Minimal surface loss prevents spurious non-visible surfaces and helps disambiguate shape from radiance by encouraging smooth SDFs.Penalizing values close to zero minimizes surface area and makes the network prefer smooth SDFs.
Sphere initialization: Sphere initialization breaks the concave/convex ambiguity and prevents optimization from settling at concave surfaces.Concave mirror surfaces can explain multi-view images while rendering views at the opposite azimuth, which the discriminator cannot detect as fake.
Limitations: The StyleGAN2-based high-resolution network may exhibit aliasing and flickering in high-frequency regions, while reconstructed human eyes can contain concave eyeball artifacts and view-dependent gaze changes.Fixed frontal view directions for RGB queries also prevent moving specular highlights from being expressed with camera changes.

I. Additional Results

The paper presents an uncurated set of randomly generated, high-resolution RGB images produced by StyleSDF networks.

StyleSDF networks generate an uncurated set of images.
The displayed samples are randomly generated rather than selected examples.
The samples are high-resolution RGB images.