PaperlayerView exact revision on arXiv

Source-linked AI summary

PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$

Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Ogras, Linjie Luo

arXiv:2303.13071v1cs.CV

TL;DR

Existing 3D GANs remain limited in view range, while full-head synthesis also faces foreground-background entanglement, tri-plane mirroring, and alignment challenges. PanoHead combines foreground-aware discrimination, tri-grid representation, and self-adaptive alignment; the authors report strong qualitative and quantitative comparisons and single-view reconstruction. They also identify data bias and rendering artifacts as limitations.

  • Problem

    Full-head 3D synthesis is challenged by foreground-background entanglement and tri-plane mirroring, while back-head images can have noisy camera labels and inconsistent crops.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Method

    PanoHead combines foreground-aware tri-discrimination, a depth-augmented tri-grid representation, and self-adaptive image alignment for 3D head synthesis from in-the-wild images.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    PanoHead presents compelling qualitative and quantitative results against state-of-the-art 3D GANs and demonstrates 360-degree reconstruction with detailed geometry from single-view portraits.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The system supports view-consistent 360° full-head synthesis and reconstruction of full 3D heads from single input images.

    Source: §6 · p. 8

    Training on only a combination of several datasets still leaves PanoHead subject to some data bias, despite the authors’ data-collection effort.

    Source: §5 · p. 8

Abstract

from arXiv · show

Synthesis and reconstruction of 3D human head has gained increasing interests in computer vision and computer graphics recently. Existing state-of-the-art 3D generative adversarial networks (GANs) for 3D human head synthesis are either limited to near-frontal views or hard to preserve 3D consistency in large view angles. We propose PanoHead, the first 3D-aware generative model that enables high-quality view-consistent image synthesis of full heads in $360^\circ$ with diverse appearance and detailed geometry using only in-the-wild unstructured images for training. At its core, we lift up the representation power of recent 3D GANs and bridge the data alignment gap when training from in-the-wild images with widely distributed views. Specifically, we propose a novel two-stage self-adaptive image alignment for robust 3D GAN training. We further introduce a tri-grid neural volume representation that effectively addresses front-face and back-head feature entanglement rooted in the widely-adopted tri-plane formulation. Our method instills prior knowledge of 2D image segmentation in adversarial learning of 3D neural scene structures, enabling compositable head synthesis in diverse backgrounds. Benefiting from these designs, our method significantly outperforms previous 3D GANs, generating high-quality 3D heads with accurate geometry and diverse appearances, even with long wavy and afro hairstyles, renderable from arbitrary poses. Furthermore, we show that our system can reconstruct full 3D heads from single input images for personalized realistic 3D avatars.

1. Introduction

Existing 3D GANs can synthesize view-consistent heads from in-the-wild images but remain limited to near-frontal views; extending them to full-head synthesis raises geometry, foreground-background, and alignment challenges. PanoHead combines a tri-grid representation, foreground-aware discrimination, and self-adaptive image alignment to synthesize full heads in 360° and support monocular reconstruction.

1. Introduction

  • Near-frontal limits in existing 3D GANs, foreground-background entanglement, tri-plane projection ambiguity, and misaligned back-head images complicate full-head synthesis.Tri-plane ambiguity can produce mirrored faces, while an alignment gap between back-head and frontal images can lead to noisy appearance and unappealing geometry.Source: §1 · p. 2
  • The resulting framework generates high-fidelity 360° RGB images and geometry, achieves better quantitative metrics than state-of-the-art methods, and supports single-image 3D head reconstruction.The introduction presents reconstruction from a single monocular-view image as a route to accessible 3D portrait creation.Source: §1 · p. 2
  • PanoHead is presented as the first 3D GAN framework for view-consistent, high-fidelity full-head synthesis with detailed geometry in 360°, and it also demonstrates monocular 3D head reconstruction from in-the-wild images.The framework is trained from in-the-wild unstructured images.Source: §1 · p. 2
  • Tri-grid representation addresses tri-plane feature entanglement, while foreground-aware tri-discrimination separates 3D head modeling from 2D background synthesis.The tri-grid balances efficiency and expressiveness; the discriminator uses 2D segmentation prior knowledge.Source: §1 · p. 2
  • Two-stage self-adaptive image alignment accommodates imperfect camera poses and misaligned crops, supporting training on in-the-wild images with widely distributed views.Camera self-adaptation dynamically adjusts rendering-camera positions to accommodate back-head alignment drift.Source: §1 · p. 2

2. Related Work

Related work spans scan-trained parametric head models, neural implicit representations, and 3D-aware GANs trained from 2D images. PanoHead builds on EG3D’s efficient tri-plane representation while extending it toward full-head synthesis.

2. Related Work

  • Parametric textured meshes such as 3DMM and FLAME are learned from scans but do not model photorealistic appearance and geometry beyond the front face or skull.Neural radiance fields offer continuous scene representations and multiview consistency, while PanoHead learns a generative NeRF from unstructured monocular images.Source: §2 · p. 2
  • The framework diagram traces latent code and conditioned camera pose through mapping, tri-grid generation, and neural rendering, then shows a discriminator evaluating image and mask outputs.Its data-processing side depicts crop and scale/location mismatch alongside camera self-adaptation.Source: Figure 2 · p. 3
  • EG3D uses a StyleGAN2 backbone to generate three feature planes that are decoded and volume-rendered into an image before super-resolution.Its tri-plane representation scales efficiently with resolution; tri-grid transforms it into a more expressive space for unconditional 3D head synthesis.Source: §2 · p. 3
  • Earlier 3D GAN approaches include vertex-position-map meshes, StyleGAN conversion, and NeRF-based models, whose performance can be limited by dependence on a base 2D GAN or high computation cost.EG3D combines a 2D GAN backbone with tri-plane representation for efficient 3D generation.Source: §2 · p. 3

3. Methodology

PanoHead builds on EG3D to address foreground-background entanglement, tri-plane projection ambiguity, and camera-alignment problems in training on widely distributed in-the-wild views. It combines foreground-aware discrimination, a tri-grid representation, and two-stage self-adaptive alignment for 360° head synthesis.

3.1. PanoHead Overview

  • PanoHead builds on EG3D’s StyleGAN2 backbone, tri-plane scene features, volume rendering, super-resolution, and dual discrimination, then adds components to address three barriers to full-head synthesis.The identified barriers are foreground-background entanglement, tri-plane mirroring artifacts, and noisy camera labels with inconsistent back-head cropping.Source: §3.1 · p. 3

3.2. Foreground-Aware Tri-Discrimination

  • A foreground-aware tri-discriminator supervises RGB images and the foreground mask, propagating 2D segmentation priors into neural-field density for background-free geometry and compositable full-head images.A background generator supplies the background, while the mask supports synthesis with different backgrounds.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Feature Disentanglement in Tri-Grid

  • Tri-grid adds a depth dimension to tri-plane features, so points at different depths can draw on distinct planes and reduce front-face feature borrowing that causes mirrored back heads.Features are retrieved by tri-linear interpolation; a small depth setting such as D = 3 can retain efficiency while disentangling features.Source: §3.3 · p. 5, Figure 5 · p. 5

3.4. Self-Adaptive Camera Alignment

  • Two-stage alignment uses landmark-based processing for detectable faces and pose estimation plus head-centered cropping for large-pose images, then learns residual camera transformations during adversarial training.The learned transformation is mapped from the latent code and camera and regularized with an L2 norm to refine image alignment.Source: §3.4 · pp. 5–6, Figure 6 · p. 5

4. Experiments

Experiments train PanoHead and the baselines on FFHQ-F, a mixture of frontal and large-pose head images. PanoHead performs well across image-quality, identity-consistency, and segmentation metrics, generates realistic full-head images and geometry across views, and supports single-view reconstruction.

4.1. Datasets and Baselines

  • The experiments use FFHQ-F, combining FFHQ with K-hairstyle back-head images and in-house large-pose images, and retrain the baselines on the same data.The combination includes 70K FFHQ images, 4K K-hairstyle back-head images, and 15K in-house images spanning 60° to 180° yaw.Source: §4.1 · p. 6

4.3. Quantitative Results

  • PanoHead outperforms the baselines across image quality, fidelity, diversity, multi-view consistency, and segmentation metrics.The evaluation uses FID, identity similarity, and segmentation MSE; Table 1 compares these metrics across methods.Source: Table 1 · p. 7, §4.3 · p. 7

4.2. Qualitative Comparisons

  • PanoHead generates photo-realistic, multi-view-consistent head images across camera poses, while several baselines degrade at larger views or show back-head mirroring.The comparison covers views from frontal to back; PanoHead also handles diverse appearances, including shaved heads and long curly hairstyles.Source: §4.2 · p. 7
  • PanoHead produces high-fidelity, background-free full-head geometry, unlike the noisy or broken geometry and hollowed or background-entangled heads shown for StyleSDF and EG3D.The geometry comparison uses shapes extracted with Marching Cubes.Source: §4.2 · p. 7, Figure 8 · p. 8

4.3. Quantitative Results

  • With tri-discrimination, tri-grid, and camera self-adaptation together, PanoHead achieves the lowest FID-back and highest IS for back-head generation.Adding foreground-aware discrimination boosts quality, and tri-grid further improves image quality; the full combination adds only minor computation overhead.Source: Table 2 · p. 7, §4.3 · p. 8

4.4. Single-view GAN Inversion

  • PanoHead’s generative latent space reconstructs a full 3D head from one portrait and renders novel views, including large poses and the back of the head.The inversion optimizes a latent code with pixel-wise L2 and image-level LPIPS losses, then applies pivotal tuning inversion with the code fixed.Source: §4.4 · p. 8

5. Discussion

PanoHead retains some visual artifacts and geometric detail limitations. Its training data also leaves some data bias, and the authors identify possible misuse of single-view reconstruction as a social threat.

Limitations and Future Work

  • PanoHead retains minor teeth-area artifacts and texture flickering, and lacks finer high-frequency geometric details such as hair tips.The authors note more noticeable flickering with a higher conditional-camera swapping probability, set to 70% to improve 360° rendering quality.Source: §5 · p. 8
  • Training on only several dataset combinations leaves PanoHead susceptible to some data bias; large-scale annotated full-head training images remain a key need.The authors also leave quantitative evaluation of geometry, such as with depth maps, for future work.Source: §5 · p. 8

Ethical considerations

  • The authors caution that single-view portrait reconstruction could be manipulated in ways that pose a social threat.They do not encourage using the method to violate others’ rights.Source: §5 · p. 8

6. Conclusion

PanoHead is presented as the first 3D GAN framework to synthesize view-consistent full-head images from single-view images. Its design supports 360° synthesis and single-view reconstruction, with qualitative and quantitative results compared with state-of-the-art 3D GANs.

6. Conclusion

  • PanoHead is presented as the first 3D GAN framework for view-consistent full-head synthesis using only single-view images.Its design combines foreground-aware tri-discrimination, a 3D tri-grid representation, and self-adaptive image alignment for 360° synthesis.Source: §6 · p. 8
  • The paper reports compelling qualitative and quantitative results against state-of-the-art 3D GANs and photo-realistic 360° reconstruction from single-view portraits.The authors describe the method as a promising direction for 3D portrait creation and potential downstream tasks.Source: §6 · p. 8
Loading 2303.13071v1…