PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperInverter: Improving StyleGAN Inversion via Hypernetwork

Tan M. Dinh, Anh Tuan Tran, Rang Nguyen, Binh-Son Hua

arXiv:2112.00719v2cs.CV

TL;DR

GAN inversion must balance reconstruction fidelity, editability, and inference speed. HyperInverter maps images to W-space with an encoder, then uses hypernetworks to refine generator weights; the authors report strong reconstruction results relative to encoder-based methods while retaining fast inference.

  • Problem

    GAN inversion methods face a reconstruction-editing trade-off: W-space supports editability but reconstructs poorly, while W+ improves reconstruction at an editing cost.

    Source: §1 · p. 2

  • Method

    An encoder first maps the image to W-space, and hypernetworks then predict residual generator weights to recover details without per-image optimization or fine-tuning.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    The method significantly outperforms other encoder-based methods on reconstruction across both domains and has comparable performance to optimization or fine-tuning methods.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    The fully encoder-based method combines high-fidelity reconstruction and editability with almost real-time inference, according to the authors.

    Source: §1 · p. 2, §5 · p. 8

    The method does not completely surpass per-image optimization methods such as SG2 W+ and PTI in metrics.

    Source: §5 · p. 8

Abstract

from arXiv · show

Real-world image manipulation has achieved fantastic progress in recent years as a result of the exploration and utilization of GAN latent spaces. GAN inversion is the first step in this pipeline, which aims to map the real image to the latent code faithfully. Unfortunately, the majority of existing GAN inversion methods fail to meet at least one of the three requirements listed below: high reconstruction quality, editability, and fast inference. We present a novel two-phase strategy in this research that fits all requirements at the same time. In the first phase, we train an encoder to map the input image to StyleGAN2 $\mathcal{W}$-space, which was proven to have excellent editability but lower reconstruction quality. In the second phase, we supplement the reconstruction ability in the initial phase by leveraging a series of hypernetworks to recover the missing information during inversion. These two steps complement each other to yield high reconstruction quality thanks to the hypernetwork branch and excellent editability due to the inversion done in the $\mathcal{W}$-space. Our method is entirely encoder-based, resulting in extremely fast inference. Extensive experiments on two challenging datasets demonstrate the superiority of our method.

1. Introduction

GAN inversion must balance faithful reconstruction, editability, and inference speed. The paper proposes a two-phase encoder method that uses W-space codes and hypernetworks to address these competing goals.

1. Introduction

  • The method significantly outperforms pSp [37], e4e [44], and ReStyle [3], matches per-image optimization quality while running 3000× faster, and is surpassed in reconstruction quality only by PTI [38].PTI takes 1100× longer because it fine-tunes the generator per image during inference.Source: Figure 1 · p. 1
  • GAN inversion maps real photographs to latent codes for reconstruction and later attribute editing, but optimization-based methods trade greater reconstruction accuracy for much longer computation.That computation cost hinders interactive editing.Source: §1 · p. 1
  • The reconstruction-editing trade-off depends on the embedding space: W-space favors editability but weak reconstruction, whereas W+ improves reconstruction at the cost of editing ability.Two-stage alternatives such as PIE and PTI still require per-image optimization or fine-tuning at inference.Source: §1 · p. 2
  • The proposed two-phase method maps images to W-space with an encoder, then uses hypernetworks to predict residual generator weights that recover lost details without inference-time optimization or fine-tuning.The authors describe the approach as fully encoder-based, combining reconstruction, editability, and fast inference.Source: §1 · p. 2
  • The paper also presents an extensive benchmark and a real-world image interpolation method that interpolates both latent codes and generator weights.The benchmark is reported to demonstrate superior performance over existing work.Source: §1 · p. 2

2. Related Work

Prior GAN inversion methods balance reconstruction quality against inference speed, while latent-space research also studies interpretable directions for image editing. HyperInverter uses encoder-based stages and hypernetworks to refine generator weights rather than fine-tuning the generator per image.

2. Related Work

  • GAN latent-space studies use supervised labels, unsupervised exploration, or contrastive learning to identify semantic directions for image editing.Supervised approaches require attribute labels or a trained classifier, limiting them to known attributes.Source: §2 · p. 2
  • Optimization-based inversion often yields high-quality reconstructions but is slow; encoder-based approaches infer latent codes quickly but usually reconstruct less accurately.Hybrid approaches combine an encoder initialization with later optimization, while PIE and PTI use other two-stage combinations.Source: §2 · p. 2
  • HyperInverter is a two-stage method that remains encoder-based in both phases, unlike hybrid approaches that follow encoding with optimization.Its second phase refines the generator to preserve image details while retaining editability.Source: §2 · p. 2, Figure 2 · p. 3
  • A hypernetwork produces weights for a primary network; here, hypernetworks predict residuals that update a pretrained GAN generator instead of using slow per-image fine-tuning.A concurrent method, HyperStyle, also applies hypernetworks to StyleGAN inversion.Source: §2 · pp. 2–3

3. Method

The method uses a two-phase encoder-based inversion: map an image to StyleGAN W-space, then refine generator weights to recover details missing from the initial reconstruction. Reconstruction losses train the phases, with an adversarial loss added in Phase II.

3.1. Phase I: From image to content code

  • Phase I regresses the input image to an editable W-space content code, but its lower-dimensional representation can leave image details unreconstructed.Source: §3.1 · p. 3

3.2. Phase II: Generator refinement via hypernetworks from appearance code.

  • Phase II encodes the input and initial reconstruction into style-layer-aligned features, fuses them as an appearance code, and uses per-layer hypernetworks to predict residual convolution weights.The residuals update the pretrained generator's convolutional weights; biases and other layers are not predicted.Source: §3.2 · p. 4

3.3. Loss functions

  • Training uses weighted reconstruction losses, including perceptual and identity terms, and adds a non-saturating GAN loss in Phase II to support image realism.Source: §3.3 · p. 4

4. Experiments

Across face and church experiments, HyperInverter combines competitive reconstruction with fast inference and retains editing ability. User evaluation and ablations assess these properties, while joint latent-code and generator-weight interpolation produces smooth real-world image transitions.

4.1. Experimental Settings

  • Experiments cover human faces (FFHQ-trained, CelebA-HQ-tested) and LSUN Church, against optimization-, encoder-, and two-stage inversion baselines.Training adds adversarial loss and R1 regularization after 200,000 iterations.Source: §4.1 · p. 5

4.2. Reconstruction Results

  • HyperInverter outperforms other encoder methods on reconstruction in both domains, is comparable to optimization methods, and runs 3000× and 1100× faster than SG2 W+ and PTI, respectively.Qualitatively, church reconstructions outperform other methods in distortion and perception, while PTI images are not completely realistic.Source: §4.2 · pp. 5–6, Figure 4 · p. 5, Table 1 · p. 6

4.3. Editing Results

  • At equal editing magnitude, Phase I has the strongest age and pose effects, while Phase II retains editing ability and outperforms prior methods; qualitative edits preserve background and yield fewer artifacts than SG2 W+.Qualitative comparisons also report that PTI edits well on faces but less impressively on churches because its reconstructions are not realistic.Source: §4.3 · p. 6, Table 2 · p. 6, Figure 5 · p. 7

4.4. User survey

  • 71.8% of reconstruction tests favored HyperInverter—2.5 times e4e and ReStyle combined—and it outperformed them by a large margin on both reconstruction and editing quality.The survey compared human perceptual rankings of reconstruction and editing quality.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5. Ablation Study

  • Phase II materially improves reconstruction; layer-wise appearance codes using both input and initial-image features outperform simplified variants, and D=256 performs best.Residual changes concentrate in main-generator convolutions and at 1024×1024, consistent with refining fine details.Source: §4.5 · pp. 7–8, Figure 7 · p. 7, Table 3 · p. 7

4.6. Application: Real-world Image Interpolation

  • Interpolating both latent codes and generator weights yields smooth intermediate images while retaining fine details such as hats, backgrounds, and hands, compared with latent-only interpolation.Source: §4.6 · p. 8, Figure 9 · p. 8

5. Conclusion

The paper concludes that its two-phase, encoder-based StyleGAN inversion combines high-fidelity reconstruction and editability with almost real-time inference. It also notes a remaining metric gap against per-image optimization methods.

5. Conclusion

  • The two-phase encoder-based method uses hypernetworks to refine generator weights and achieves high-fidelity reconstruction, excellent editability, and almost real-time inference.The hypernetworks refine the generator’s weights.Source: §5 · p. 8
  • On metrics, the encoder-based method is not completely better than per-image optimization approaches such as SG2 W+ and PTI.The authors also show cases where SG2 W+ and PTI yield inaccurate reconstructions.Source: §5 · p. 8
  • The authors identify iterative refinement, multilayer identity loss, and extension to StyleGAN3 as possible improvements.Source: §5 · p. 8

Supplementary Material

The supplementary material reports a churches-domain user study, discusses potential societal harms, and provides additional analyses and reproducibility details.

Supplementary Material

  • The supplement also includes residual visualizations, dataset and implementation details, and further qualitative examples.Source: Supplementary Material · p. 11

A. Discussion on Negative Societal Impacts

  • The authors warn that realistic reconstructions and fast predictions could be misused to create deepfakes, including real-time videos.They state that generated faces may be indistinguishable from real faces.Source: A. Discussion on Negative Societal… · p. 11

B.1. The Churches Domain

  • Our method was favored in 84.3% of churches reconstruction tests and 77.2% of editing tests, about 5.3× and 3.3× better than e4e and ReStyle combined, respectively.Source: §B.1 · p. 11

B.2. Details of User Study

The user study asks participants to rank reconstructed and edited images separately, using domain-specific criteria. It covers human faces and churches with 30 input images per domain.

B.2. Details of User Study

  • Participants ranked images from first (best) to third (worst) in separate reconstruction and editability tests.Source: §B.2 · p. 11
  • For churches, testers significantly preferred our method over e4e and ReStyle on reconstruction and editing quality.Source: Figure 10 · p. 11
  • The face study used 30 images and collected 1,140 reconstruction votes from 38 participants and 900 editing votes from 30 participants.Reconstruction criteria included identity, image details, and aesthetics; editing criteria emphasized identity and preserving non-edited details.Source: §B.2 · p. 11
  • The churches study used 30 LSUN Church test images and collected 1,050 reconstruction votes and 780 editing votes.The criteria emphasized restoring image details and aesthetics for reconstruction, and preserving non-edited details for editing.Source: §B.2 · p. 11

C. Visualization and Analysis

The churches analysis finds that Phase II updates residual weights across more layers than in face images, consistent with the domain’s greater reconstruction diversity and challenge.

C. Visualization and Analysis

  • In churches, residual weight updates occur at most layers, unlike face updates that concentrate at the last layer.Source: §C.1 · p. 12
  • The main convolution weights contribute significantly relative to the toRGB block in both churches and faces.Source: §C.1 · p. 12
  • The broader church updates align with a need to recover both low- and high-frequency signals; for faces, Phase I is already close to the input and Phase II restores fine-grained details.Source: §C.1 · p. 12, Figure 11 · p. 12

C.2. Difference maps

The authors average Phase I-to-final image differences into heat maps to identify where Phase II changes reconstructions. Faces show concentrated refinement of detailed regions, while church sky regions often change little.

C.2. Difference maps

  • The heat map averages absolute pixel differences between Phase I outputs and final reconstructions across the test set, then converts the mean map to grayscale for visualization.The analysis uses 2,824 CelebA-HQ test images and 300 LSUN Church test images.Source: §C.2 · p. 12
  • Phase II focuses face refinements on fine-grained regions such as hair, cheek, beard, and eyes; church heat maps are more random, with sky regions often changing little.The top-left church region, often sky, is already well reconstructed after Phase I.Source: §C.2 · p. 12, Figure 12 · p. 12
  • Figures 13 and 14 provide per-image heat maps to show where Phase II changes individual face and church reconstructions.Blue denotes small changes and red denotes large changes.Source: §C.2 · p. 12, Figure 13 · p. 12, Figure 14 · p. 13

D. Additional Experimental Details

The experiments use separate training and test sets for aligned human faces and more structurally diverse church scenes.

D.1. Datasets

  • The face experiments train on 70,000 FFHQ images and test on 2,824 official CelebA-HQ images, with both datasets at 1024x1024 resolution.All face images are aligned to the center.Source: §D.1 · p. 13
  • The church experiments use 126,227 LSUN Church training images and 300 official test images at 256 × 256 resolution.Churches are described as more diverse than faces and therefore more challenging.Source: §D.1 · p. 13
  • The evaluation uses CelebA-HQ for face testing and LSUN Church for church testing, while training uses FFHQ for faces and LSUN Church for churches.Source: §D.1 · p. 13

D.2. Detailed architecture of E1 and E2 encoders.

The encoder design retains an established FPN-based backbone while adapting its output for the E2 encoder’s intermediate features.

D.2. Detailed architecture of E1 and E2 encoders

  • E1 uses the W encoder design from [37] [44] without modifications, while E2 adopts their W+ encoder design with modifications for intermediate features.Source: §D.2 · p. 13
  • The FPN-based E2 encoder modifies map2style to output a 512 × 8 × 8 feature tensor instead of the original 512-dimensional vector.Figure 15 depicts the FPN-based network and this map2style modification.Source: §D.2 · p. 13, Figure 15 · p. 13
  • The backbone follows the FPN-based encoder design of Richardson et al. [37], and the figure omits LeakyReLU activations after convolution layers for simplicity.Source: Figure 15 · p. 13, §D.2 · p. 13

E. Additional Qualitative Results

The supplementary figures provide additional visual comparisons of reconstruction, editing, and interpolation across faces and churches.

E. Additional Qualitative Results

Loading 2112.00719v2…