PaperlayerView exact revision on arXiv

Source-linked AI summary

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

Yuval Alaluf, Or Patashnik, Daniel Cohen-Or

arXiv:2104.02699v2cs.CV

TL;DR

Encoder-based GAN inversion is faster than per-image optimization but has a reconstruction-accuracy gap. ReStyle addresses this with iterative residual prediction using feedback from prior reconstructions, improving encoder reconstruction quality with a negligible inference-time increase while remaining much faster than optimization.

  • Problem

    Encoder-based inversion is faster and more editable than per-image optimization, but its reconstruction accuracy remains below optimization-based methods.

  • Method

    ReStyle repeatedly feeds the source image and current reconstruction to an encoder that predicts residual latent updates in StyleGAN’s W+ space.

  • Results

    ReStyle significantly improves reconstruction quality over standard feed-forward encoders with a negligible increase in inference time.

  • Takeaways & Limitations

    ReStyle offers a quality–time trade-off that is visually comparable to optimization while using substantially less inference time.

  • Takeaways & Limitations

    The reason codes from FFHQ latent space remain meaningful in toonify space is not immediately clear.

Abstract

from arXiv · show

Recently, the power of unconditional image synthesis has significantly advanced through the use of Generative Adversarial Networks (GANs). The task of inverting an image into its corresponding latent code of the trained GAN is of utmost importance as it allows for the manipulation of real images, leveraging the rich semantics learned by the network. Recognizing the limitations of current inversion approaches, in this work we present a novel inversion scheme that extends current encoder-based inversion methods by introducing an iterative refinement mechanism. Instead of directly predicting the latent code of a given real image using a single pass, the encoder is tasked with predicting a residual with respect to the current estimate of the inverted latent code in a self-correcting manner. Our residual-based encoder, named ReStyle, attains improved accuracy compared to current state-of-the-art encoder-based methods with a negligible increase in inference time. We analyze the behavior of ReStyle to gain valuable insights into its iterative nature. We then evaluate the performance of our residual encoder and analyze its robustness compared to optimization-based inversion and state-of-the-art encoders.

1. Introduction

ReStyle addresses the reconstruction gap of encoder-based GAN inversion by replacing single-pass prediction with iterative residual refinement. It improves reconstruction quality over standard encoders while retaining substantially faster inference than optimization.

  • Motivation: Encoder-based inversion is faster and more editable than per-image optimization but still has a significant reconstruction-accuracy gap.This gap leaves encoder design and training as open challenges.
  • Approach: ReStyle feeds the original image and the previous reconstruction to an encoder that predicts a latent residual at each iteration.The residual is added to the current latent estimate, enabling self-correcting convergence without subsequent per-image optimization.
  • Approach: ReStyle can be viewed as learning a small number of residual-based latent-space steps and can be applied to different encoder architectures and loss objectives.The paper describes examples using about 10 steps.
  • Results: ReStyle significantly improves reconstruction quality over standard feed-forward encoders with a negligible inference-time increase.Its residuals decrease in magnitude and converge after only a small number of steps, while refinements proceed coarse-to-fine.
  • Robustness: Across downstream analyses, the paper examines semantic latent manipulations and encoder bootstrapping for more faithful toonification.These experiments assess robustness and generalization beyond the core inversion task.

2. Background and Related Works

Prior work combines latent-space manipulation with inversion methods that trade reconstruction quality against speed. ReStyle extends iterative refinement, previously used in other vision tasks, to learned inversion of real images.

  • Iterative refinement: Iterative refinement has been explored for pose estimation, optical flow, object pose estimation, detection, and semantic segmentation.The paper positions its use for learned real-image inversion as novel.
  • GAN inversion and manipulation: GAN latent spaces encode semantic information that supports image manipulation, but real-image manipulation typically requires inversion first.Real images are commonly projected into StyleGAN’s W+ space before editing.
  • Inversion methods: Optimization-based inversion often achieves high reconstruction quality but can require several minutes per image.Encoder-based methods were developed partly to avoid this per-image cost.
  • ReStyle: ReStyle applies iterative refinement to learned inversion by repeatedly updating latent predictions and reconstructions.The scheme uses the previous output as feedback for the next prediction.

3. Preliminaries

Encoder-based inversion trains an encoder to map real images into StyleGAN’s latent space so the generator reconstructs the input. Training commonly combines reconstruction and perceptual objectives, with additional losses targeting identity or editability.

  • Encoder-based inversion: The inversion objective is to produce ˆy = G(E(x)) such that the generated reconstruction ˆy approximates the source image x.Conventional encoder-based inversion computes this reconstruction through one forward pass.
  • Training objectives: Encoder training commonly uses a weighted combination of pixel-wise L2 and perceptual losses.These objectives guide the encoder on the reconstruction task while the pretrained generator remains fixed.
  • Training objectives: Identity loss improves facial reconstruction, while additional regularization losses target editability of inverted latent codes.The cited extensions address different properties of the inversion output.

4. Method

ReStyle turns StyleGAN inversion into a multi-step encoder–generator loop: each step uses the current reconstruction to predict a residual latent update. Training applies losses and back-propagation at every step, with simpler encoder variants also explored.

  • Training: Training computes loss objectives at every forward pass and updates the encoder by back-propagation, while the pretrained generator remains fixed.A training iteration contains N steps on a batch, producing N back-propagation updates.
  • Inference: Inference uses the same multi-step process without loss computation, and convergence typically requires fewer than 10 steps.The procedure starts from the generator’s average style vector and corresponding synthesized image.
  • Iterative update: At each step, the encoder receives the source image concatenated with the current reconstruction and predicts a residual code relative to the previous latent prediction.This expanded input supplies feedback from the current inversion state.
  • Iterative update: The updated latent code is passed through the generator, and its reconstruction becomes the additional input for the next step.This closes the iterative feedback loop.
  • Method: The multi-step scheme relaxes the single-step constraint and learns efficient latent-space convergence in a self-correcting manner.The paper compares these learned steps conceptually with optimization steps.
  • Encoder architecture: Simplified encoder variants extract all style vectors from the final 16 × 16 feature map through k map2style blocks.The design replaces extraction from three intermediate feature levels.

5. Experiments

Experiments evaluate ReStyle across domains and inversion settings, showing improved reconstruction over standard encoders while preserving a favorable quality–time trade-off. Additional analyses examine iterative refinement, editability, and encoder bootstrapping.

  • 5.1. Settings: Evaluations span faces, cars, horses, churches, and wild animals, comparing encoder-based, optimization-based, and hybrid inversion methods.Faces use FFHQ for training and CelebA-HQ for evaluation; cars use Stanford Cars, with additional LSUN and AFHQ Wild evaluations.
  • 5.2. Comparison with Inversion Methods: ReStyle is visually comparable to optimization-based inversion while attaining significantly lower inference time, without claiming superior reconstruction quality over optimization.The comparison emphasizes the quality–time trade-off rather than optimization-level reconstruction superiority.
  • 5.2. Comparison with Inversion Methods: ReStyle improves reconstruction over pSp and e4e, particularly by preserving fine details such as collars, hair, car symbols, license plates, backgrounds, and poses.These improvements are reported across face, car, wild-animal, and horse comparisons.
  • 5.2. Comparison with Inversion Methods: On faces, optimization requires ≈20× more time to match ReStyle’s identity similarity, while ReStyle matches or surpasses conventional encoders across evaluated domains.ReStyle nearly matches optimization and hybrid reconstruction quality for churches, and its advantage over typical encoders is more pronounced for car L2 loss.
  • 5.3. ReStyle Analysis: ReStyle refines images coarse-to-fine: early steps adjust background and pose, later steps refine eyes and hair, while the largest changes occur initially and decrease thereafter.The iterative process progressively improves reconstructions of non-frontal faces, horse-rider posture, and skewed church structure.
  • 5.4. Editability via Latent Space Manipulations: ReStyle preserves editability comparable to conventional encoders, produces more plausible edits than optimization in shown examples, and bootstrapping better preserves input characteristics during toonification.The bootstrapping technique’s meaningful translation between FFHQ and toonify latent spaces is not immediately explained.

6. Conclusions

ReStyle improves encoder-based inversion by using multiple residual refinement passes, and the paper extends this scheme with encoder bootstrapping for toonification.

  • ReStyle replaces one-shot inversion with multiple forward passes that guide convergence toward the target inversion.The encoder is trained on original images and intermediate reconstructions.
  • The residual-based scheme is paired with bootstrapping to translate real face images into toonified versions.The paper presents this combination as a possible basis for additional tasks.
  • The appendix provides additional analysis, ablations, and full-resolution comparisons complementing the main manuscript.

A.1. The ReStyle Encoder Architecture

The appendix describes ReStyle’s simplified encoder architecture and the datasets and baseline encoders used for evaluation, alongside visualizations of iterative refinement.

  • The ReStyle Encoder Architecture: The original FPN encoder extracts StyleGAN style inputs from feature maps at 64 × 64, 32 × 32, and 16 × 16 resolutions.The architecture uses map2style blocks on three intermediate feature maps.
  • The ReStyle Encoder Architecture: ReStyle instead extracts every style input from the final 16 × 16 feature map, simplifying the encoder architecture.
  • Datasets: Evaluation covers human faces, cars, AFHQ Wild, horses, and churches using the listed training and evaluation datasets.The appendix specifies dataset sizes and test subsets for each domain.
  • Baselines: The comparisons include IDInvert, pSp, and e4e, using official models or retraining procedures across domains.
  • Iterative Refinement: Figure 14 visualizes intermediate outputs and globally normalized heatmaps, with red indicating larger pixel-space changes.

B. ReStyle Analysis

ReStyle refines images coarse-to-fine: early steps alter global structure, later steps adjust details, and the latent changes converge after a few iterations.

  • Where’s the focus?: Early iterations refine global features such as head pose or car shape, while later iterations adjust finer details.Figure 14 shows these changes for individual images using normalized heatmaps.
  • Latent-Space Analysis: ReStyle primarily changes coarse and medium-level latent inputs and converges after a few inference steps.
  • Latent-Space Analysis: The analysis measures squared latent-entry differences between consecutive iterations, averages them across test samples, and groups entries by StyleGAN scale.
  • Latent-Space Analysis: Coarse inputs change most early as the encoder refines background and pose, whereas fine styles change least and converge after only a few steps.The text gives color as an example of an aspect that may be easier to refine.
  • Ablation Study: Repeatedly feeding output images through pre-trained pSp or e4e deteriorates reconstructions, unlike the incrementally improving ReStyle scheme.

C. Ablation Study

The ablations validate dedicated residual training and show that ReStyle’s simplified encoder preserves reconstruction quality while reducing inference time.

  • The Iterative Training Scheme: Naively reusing a conventional encoder causes reconstructions to move away from the input at each additional step.This comparison is performed on human facial and cars domains with pSp and e4e encoders.
  • The Iterative Training Scheme: The dedicated ReStyle iterative, residual-based training scheme is needed instead of repeatedly applying readily available encoders.
  • The ReStyle Encoder Architecture: The simplified architecture has nearly identical L2 and LPIPS reconstruction quality to the FPN variant across five domains after five inference steps.
  • The ReStyle Encoder Architecture: The simplified architecture is more than 10% faster than the FPN-based architecture.

D. Analyzing the Toonify Latent Space

ReStyle uses an FFHQ-to-Toonify latent-space initialization strategy, supported by evidence that the two latent spaces are well-aligned. The same latent code produces semantically similar images in both generators.

  • Encoder bootstrapping: ReStyle initializes toonification by first inverting the real image into the FFHQ StyleGAN latent space.The inverted code and reconstructed image initialize the ReStyle toonify encoder.
  • Latent-space alignment: The same randomly sampled latent code is passed through both FFHQ and Toonify StyleGAN generators for comparison.This tests whether corresponding latent codes produce semantically similar synthesized images.
  • Latent-space alignment: The FFHQ and Toonify latent spaces are well-aligned because identical latent codes generate semantically similar images.This alignment motivates using FFHQ inversion to improve initialization for Toonify inversion.

E. Quantitative Results

The quantitative evaluation compares inversion methods by jointly examining reconstruction quality and inference time across five domains. Facial reconstructions additionally receive an identity-similarity evaluation.

  • Quality-time evaluation: Quality-time graphs show how reconstruction quality changes with inference time across inversion methods and five domains.The comparisons use L2 and LPIPS loss metrics in Figures 18 and 19.
  • Method comparison: The evaluation compares optimization-based, encoder-based, and hybrid inversion methods.The comparison framework is designed to assess both reconstruction quality and runtime during inference.
  • Quality-time evaluation: Facial reconstructions are additionally evaluated for identity similarity using the CurricularFace facial-recognition method.This supplements the L2 and LPIPS reconstruction metrics for the human facial domain.

F. Additional Results

Additional results cover qualitative reconstruction, iterative refinement, editing, comparisons with IDInvert, and image toonification across multiple image domains. Results are shown at domain-specific full resolutions.

  • Iterative refinement: Iterative outputs are shown for ReStyle applied over pSp on faces and churches, and over e4e on cars and horses.These visualizations cover Figures 30–33.
  • Evaluation settings: All additional results are displayed at full resolution: 1024 × 1024 for faces, 512 × 512 for cars and wild animals, and 256 × 256 for churches and horses.The reported resolutions vary by image domain.
  • Encoder comparisons: Figure 29 compares ReStyle with the IDInvert encoder on human facial and cars test sets.The figure reports results with ReStyle applied over both pSp and e4e encoders.
  • Editing comparisons: Editing comparisons with optimization-based inversion cover age and smile edits for faces, cube-shape, viewpoint, and color edits for cars, and pose and rider edits for horses.The edits use InterFaceGAN, GANSpace, and SeFa, respectively.
  • Image toonification: Additional image toonification results use the encoder bootstrapping method presented in Section 5.5.Figure 37 shows the single-step FFHQ inversion followed by iterative outputs from the Toonify encoder, with Figure 38 providing further results.
Loading 2104.02699v2…