PaperlayerView exact revision on arXiv

Source-linked AI summary

In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing

Yiran Xu, Zhixin Shu, Cameron Smith, Seoung Wug Oh, Jia-Bin Huang

arXiv:2302.04871v4cs.CV

TL;DR

Natural-face-pretrained GANs struggle with OOD objects, making faithful reconstruction and editability difficult to balance. This paper models face and OOD content in separate radiance fields and composites them, reporting a better reconstruction-editability balance than other baselines.

  • Problem

    Natural-face-pretrained GANs often cannot model OOD objects well, and existing inversion methods may fail to reconstruct them or reduce editability when fine-tuning the generator.

    Source: §1 · p. 2

  • Method

    The method represents InD face and OOD content with separate radiance fields, composites them for reconstruction, and edits the InD latent code while leaving OOD content unchanged.

    Source: §1 · p. 2

  • Results

    The method improves reconstruction and editing quality over previous state-of-the-art GAN inversion work on challenging in-the-wild face images and videos.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The method achieves a better reconstruction-editability balance than other baselines by isolating OOD objects from the InD face representation.

    Source: §7 · p. 9

    When editing OOD regions, high blending weights can make objects such as added eyeglasses hard to introduce, and adding eyeglasses when the OOD object is already glasses can produce duplicates.

    Source: §6 · p. 8

Abstract

from arXiv · show

3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. However, a model pre-trained on a particular dataset (e.g., FFHQ) often has difficulty reconstructing images with out-of-distribution (OOD) objects such as faces with heavy make-up or occluding objects. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs. Our core idea is to represent the image using two individual neural radiance fields: one for the in-distribution content and the other for the out-of-distribution object. The final reconstruction is achieved by optimizing the composition of these two radiance fields with carefully designed regularization. We demonstrate that our explicit decomposition alleviates the inherent trade-off between reconstruction fidelity and editability. We evaluate reconstruction accuracy and editability of our method on challenging real face images and videos and showcase favorable results against other baselines.

1. Introduction

3D-aware GAN inversion supports view control and editing, but OOD elements such as heavy makeup or occlusions challenge reconstruction and editability. The paper separates InD faces from OOD content in two radiance fields and reports improved reconstruction and editing quality on challenging face images and videos.

1. Introduction

  • 3D-aware GANs offer camera-view control and geometric consistency while retaining generative capacity and editability.The paper situates these capabilities in applications such as novel-view synthesis and image or video editing.Source: §1 · p. 2
  • OOD objects are poorly modeled by natural-face-pretrained GANs, creating a reconstruction-editability trade-off when both face and object must fit one latent code.Existing methods either fail to reconstruct OOD content or fine-tune the generator at the cost of editability.Source: §1 · p. 2
  • The method decomposes images into InD and OOD radiance fields, composes them for reconstruction, and edits only the InD latent code.It uses EG3D and an additional tri-plane for OOD content.Source: §1 · p. 2
  • The method improves reconstruction and editing quality over previous state-of-the-art GAN inversion work on challenging in-the-wild face images and videos.The paper also demonstrates semantic editing, novel view synthesis, and OOD object removal.Source: §1 · pp. 2–3
  • The approach targets single images and monocular videos with OOD objects, including accessories and heavy makeup, and reconstructs their 3D face shapes.The paper presents composite volume rendering as part of the 3D-aware inversion approach.Source: §1 · p. 3

2. Related Work

Prior inversion work spans encoder-based, optimization-based, and hybrid approaches, while 3D-aware GANs extend inversion with view synthesis. For OOD content, this paper contrasts 2D segment composition with a volumetric 3D decomposition rendered by compositing radiance fields.

2. Related Work

  • 3D-aware GANs use 3D representations and commonly render low-resolution images before a super-resolution stage, with EG3D serving as this paper’s generator architecture.These models support view synthesis and other 3D-aware applications.Source: §2 · p. 3
  • GAN inversion methods include encoder-based, optimization-based, and hybrid approaches, with recent work addressing single-image and video inversion for 3D-aware GANs.The paper notes that prior approaches have difficulty with challenging OOD cases.Source: §2 · p. 3
  • Prior OOD inversion work includes extended W+ projection, generator fine-tuning, StyleSpace inversion, and ChunkyGAN’s composition of generated images using segmentation masks.ChunkyGAN represents images as 2D segments, whereas this paper uses volumetric 3D representation and composite rendering.Source: §2 · p. 3
  • Composite NeRFs provide a way to combine multiple radiance fields, and this paper applies that idea by splitting InD and OOD content in EG3D’s tri-plane representation.The two parts are composed during volume rendering.Source: §2 · p. 3

3. 3D-aware GAN: EG3D

EG3D generates a low-resolution image through tri-plane features, a decoder, and volume rendering, then uses a super-resolution module for high-resolution output. The method builds on this architecture and adds a separate OOD radiance field for composite rendering.

3. 3D-aware GAN: EG3D

  • EG3D consists of a tri-plane representation and a super-resolution module and is the 3D-aware GAN used by the method.The paper builds its approach on this generator architecture.Source: §3 · p. 3
  • EG3D generates a tri-plane from a latent code and camera parameters, samples points along pixel rays, and decodes aggregated plane features into color and density for volume rendering.The result is a low-resolution raw RGB image with a 32-channel feature.Source: §3 · p. 3
  • EG3D’s super-resolution module takes the raw image and 32-channel feature to produce a high-resolution RGB image.The paper selects EG3D in part for its rendering efficiency compared with other alternatives.Source: §3 · p. 3
  • The method uses two radiance fields for the InD face and OOD object, with EG3D’s pretrained generator and InD decoder kept frozen.It optimizes an additional OOD tri-plane, per-frame latent code, and decoder, then composites the fields and fine-tunes the super-resolution module.Source: Figure 3 · p. 4

4. Method

The method inverts aligned face images or monocular videos by decomposing them into in-distribution face and out-of-distribution object radiance fields, then composing them for reconstruction. It finetunes EG3D’s super-resolution module before enabling face or video editing.

4. Method

  • The pipeline represents an input face image or monocular video with separate in-distribution face and out-of-distribution object radiance fields, composes them for low-resolution reconstruction, then finetunes EG3D’s super-resolution module before editing.The method uses tri-plane representations for both fields.Source: §4 · p. 4

4.1. In-distribution GAN inversion

  • For in-distribution content, the method optimizes latent code wt to reconstruct each frame, jointly inverting video frames and obtaining camera parameters from an off-the-shelf pose detector.Regularizers keep wt near the pretrained latent distribution and constrain variation among its style vectors to preserve editability.Source: §4.1 · pp. 4–5

4.2. Modeling out-of-distribution contents

  • For OOD content, an additional tri-plane represents the object, while a per-frame latent code models changes across video frames.The OOD decoder maps tri-plane features and the frame code to color, density, and a blending weight.Source: §4.2 · p. 5

4.3. Composite volume rendering

  • Composite volume rendering combines the two fields using blending weight b and its complement, with an entropy penalty encouraging cleaner separation into InD or OOD content.For single-image input, a depth regularizer aligns the composite-rendered depth map with a rescaled MiDaS depth map.Source: §4.3 · p. 5

4.4. Low-resolution reconstruction

  • Low-resolution reconstruction jointly optimizes the InD latent code, OOD tri-plane, decoder, and per-frame code using composite-rendering losses and latent regularizers.The total loss includes the depth penalty only for single-image input.Source: §4.4 · p. 5

4.5. Super-Resolution

  • Because the pretrained super-resolution module produces unsatisfactory high-resolution output with the new OOD tri-plane, the method finetunes that module to produce 512 × 512 output.Its objective combines squared reconstruction error and LPIPS loss.Source: §4.5 · p. 6

4.6. Editing

  • After reconstruction, the method edits the latent code with existing GAN-based approaches such as InterfaceGAN and StyleCLIP, while explicit decomposition keeps OOD contents from interfering with InD semantic editing.Source: §4.6 · p. 6

5. Experimental Results

Across in-the-wild face images and videos, the experiments evaluate reconstruction accuracy and editability, alongside qualitative editing and 3D-aware applications. The results indicate stronger reconstruction and identity preservation in most editing cases, with improved reconstruction–editability balance at greater optimization cost.

5.1. Experimental Setup

  • The evaluation uses 20 challenging videos featuring heavy makeup or occluding objects, with first frames for single-image inversion and all frames for video inversion.Baselines include optimization-based methods and encoder-based methods; W+ optimization serves as an ablation without the OOD triplane.Source: §5.1 · p. 6

5.2. Quantitative results

  • The method outperforms other approaches on all reconstruction metrics and preserves identity better in most editing cases.Table 1 evaluates reconstruction quality; Table 2 evaluates identity preservation, where higher numbers are better.Source: §5.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

5.3. Qualitative results

  • Video reconstructions show higher fidelity, particularly in OOD regions, than encoder- and optimization-based baselines; qualitative editing better preserves appearance details.The video comparison includes Figure 5, and the editing comparison includes Figure 6.Source: §5.3 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 8

5.4. Other Applications

  • After inversion, the 3D representation supports novel-view synthesis, and setting OOD-object blending weights to zero removes those objects.Figure 7 shows different views of the same video frame.Source: §5.4 · p. 7, Figure 7 · p. 8

5.5. Ablation Study

  • Removing blending-weight regularization Lb or latent-code regularization Lw improves reconstruction accuracy but reduces editability.Without Lb, OOD blending can produce duplicate eyes during eyeglasses editing; without Lw, eyebrows can become unnatural.Source: §5.5 · pp. 7–8, Table 3 · p. 7, Figure 8 · p. 8

5.6. Speed

  • The method takes more optimization time than baselines while significantly improving the reconstruction–editability trade-off.The speed comparison uses 200 frames on one NVIDIA RTX A6000 GPU.Source: §5.6 · p. 8

6. Limitations

The authors report limitations involving editing OOD regions, duplicate objects, extreme poses, slight object movement, and temporal consistency.

  • Adding eyeglasses to an OOD region such as heavy makeup is difficult because blending weights there are closer to 1.Source: §6 · p. 8
  • When the OOD object is already glasses, adding eyeglasses introduces duplicates because the OOD radiance field lacks GAN and face priors.Source: §6 · p. 8
  • Editing fails when the subject has an extreme pose, such as a side view.Source: §6 · p. 8
  • Slight movement of an OOD object can impair radiance-field reconstruction and introduce floater artifacts in novel views.Source: §6 · p. 8
  • Video editing results may have temporal inconsistency; temporal constraints and finetuning could improve this aspect.Source: §6 · p. 8

7. Conclusions

The method isolates OOD objects from the in-distribution face, composes two radiance fields for reconstruction, and modifies the in-distribution latent code for editing.

  • The method achieves a better reconstruction-editability balance than other baselines while producing accurate reconstructions and faithful edits through separate radiance fields.Source: §7 · p. 9
Loading 2302.04871v4…