Source-linked AI summary
Pivotal Tuning for Latent-based Editing of Real Images
Daniel Roich, Ron Mokady, Amit H. Bermano, Daniel Cohen-Or
TL;DR
Real-image editing with StyleGAN is limited by a distortion-editability trade-off during inversion. PTI lightly tunes the generator around an inverted pivot with local regularization, and the paper reports faithful reconstruction with preserved editing quality, including challenging identities. The approach adds approximately three minutes of computation per new identity and motivates faster, multi-image, and broader-architecture extensions.
Problem
Inverting out-of-domain real faces into StyleGAN requires balancing faithful reconstruction against the editability of the resulting latent code.
Method
PTI inverts an image to an editable pivot code, then lightly tunes the generator around it while regularization contains the adaptation locally.
Results
PTI combines accurate reconstructions with preserved editing capabilities and supports realistic edits for challenging real images and multiple identities.
Takeaways & Limitations
Personalizing StyleGAN extends latent-based facial editing to practical real-life images that are outside the pretrained generator’s distribution.
Takeaways & Limitations
PTI currently costs around three minutes to incorporate a new identity, motivating a trainable mapper for faster forward-pass approximation.
Abstract
from arXiv · showhide
Recently, a surge of advanced facial editing techniques have been proposed that leverage the generative power of a pre-trained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pre-trained generator's domain. As it turns out, however, StyleGAN's latent space induces an inherent tradeoff between distortion and editability, i.e. between maintaining the original appearance and convincingly altering some of its attributes. Practically, this means it is still challenging to apply ID-preserving facial latent-space editing to faces which are out of the generator's domain. In this paper, we present an approach to bridge this gap. Our technique slightly alters the generator, so that an out-of-domain image is faithfully mapped into an in-domain latent code. The key idea is pivotal tuning - a brief training process that preserves the editing quality of an in-domain latent region, while changing its portrayed identity and appearance. In Pivotal Tuning Inversion (PTI), an initial inverted latent code serves as a pivot, around which the generator is fined-tuned. At the same time, a regularization term keeps nearby identities intact, to locally contain the effect. This surgical training process ends up altering appearance features that represent mostly identity, without affecting editing capabilities. We validate our technique through inversion and editing metrics, and show preferable scores to state-of-the-art methods. We further qualitatively demonstrate our technique by applying advanced edits (such as pose, age, or expression) to numerous images of well-known and recognizable identities. Finally, we demonstrate resilience to harder cases, including heavy make-up, elaborate hairstyles and/or headwear, which otherwise could not have been successfully inverted and edited by state-of-the-art methods.
1. Introduction
StyleGAN enables high-quality facial editing but real-image inversion exposes a distortion-editability trade-off. PTI personalizes the generator around an inverted pivot to improve reconstruction while preserving local editing capabilities.
- StyleGAN’s disentangled latent space supports realistic edits such as facial orientation, expression, and age.
- W+ inversion reduces distortion for out-of-domain images but weakens editability because its codes lie outside StyleGAN’s trained domain.
- PTI first inverts an input to an editable latent code, then lightly tunes the pretrained generator around that pivot.
- Regularization keeps pivotal tuning local, preserving nearby latent identities and enabling multiple target identities without interference.
- PTI achieves state-of-the-art inversion and real-image editing results, combining better reconstruction with superior editability on challenging images.
2. Related Work
Related work develops latent editing controls and GAN inversion methods, but W+ inversion retains a distortion-editability trade-off. PTI is positioned as combining W-like editing quality with accurate reconstruction.
- Prior latent-editing methods use semantic labels, self-supervision, PCA, eigenvector decomposition, borrowed codes, style parameters, 3D face models, or CLIP.
- GAN inversion methods are typically optimization-based, directly adjusting a latent code, or encoder-based, training an encoder over many samples.
- W+ inversion achieves minimal distortion, but latent manipulations over W+ are inferior to equivalent manipulations over StyleGAN’s native W space.
- Earlier methods improve editability at the cost of distortion, whereas PTI combines W-inversion editing quality with highly accurate reconstructions.
- Generator-tuning approaches can suffer significant reconstruction distortion when tuning is too extensive.
3. Method
The method first inverts an input image into an editable latent code, then fine-tunes StyleGAN around that pivot to improve reconstruction while retaining latent-space editing. Locality regularization limits side effects, enabling edits for multiple target identities.
- Inversion and pivotal tuning: The method handles out-of-distribution images by finding an editable latent code and then adapting the generator to reconstruct the target appearance.The approach is designed for real identities, extreme lighting, heavy makeup, and extravagant hair or headwear.
- Inversion and pivotal tuning: Inversion optimizes a latent code and noise vector with LPIPS perceptual loss and noise regularization while keeping the pretrained generator frozen.The resulting pivot code provides the starting point for generator tuning.
- Inversion and pivotal tuning: Pivotal tuning unfreezes the generator and minimizes reconstruction loss for the inverted pivot code, whose use is crucial for successful convergence.The tuned generator produces xp from wp using the updated weights θ∗.
- Multi-image editing: The tuning process can be extended to multiple target images, after which standard latent-space editing techniques can be applied to the tuned generator.The paper reports using editing methods such as those of Shen et al. and Harkonen et al.
- 3.3. Locality Regularization: Locality regularization samples random latent codes, interpolates them with the pivot, and penalizes changes between images generated by the original and tuned generators.This regularization addresses the ripple effect that can compromise non-local image quality, particularly when tuning multiple identities.
- Qualitative evaluation: The method reconstructs challenging out-of-domain details such as face paintings and hands, while also improving reconstruction quality on less challenging CelebA-HQ examples.The cited comparisons report better visual quality than state-of-the-art methods.
4. Experiments
Across reconstruction and editing experiments, PTI improves reconstruction quality while preserving identity and retaining strong editability. It also supports meaningful edits on challenging and famous out-of-domain images, including sequential and StyleClip edits.
- 4.1. Reconstruction Quality: PTI achieves the best reconstruction score across MSE, LPIPS, MS-SSIM, and identity similarity, with a substantial margin over baselines.Evaluation uses CelebA-HQ images and a StyleGAN2 generator trained on FFHQ.
- 4.2. Editing Quality: PTI produces meaningful pose and smile-removal edits while preserving identity, whereas SG2 W+ fails to induce the desired edit and SG2 and e4e lose identity.The CelebA-HQ comparison demonstrates this tradeoff directly for pose and smile-removal edits.
- 4.2. Editing Quality: PTI substantially improves identity preservation for StyleClip’s bowl-cut and mohawk edits while retaining the editing capabilities of e4e codes.The method extends StyleClip to more challenging images by reducing e4e’s inherent distortion.
- 4.2. Editing Quality: PTI preserves identity best while providing editing quality comparable to W-based inversion, unlike W+ inversion’s weak edits and W inversion’s identity loss.The comparison evaluates GANSpace and InterfaceGAN edits on identity preservation and editing magnitude.
- 4.2. Editing Quality: PTI’s edit magnitude is almost identical to W’s, surpassing e4e and W+ inversions and indicating high editability.The result is reported for a common pose-editing operation.
- 4.2. Editing Quality: Regularization significantly reduces side effects while maintaining similar reconstruction quality for the target image.The ablation compares reconstruction of random latent codes and the target image with and without regularization.
5. Conclusions
Pivotal Tuning Inversion personalizes a pretrained StyleGAN generator to preserve reconstruction quality while retaining latent-based editability for real facial images. The method improves quality at the cost of additional computation and remains limited by the lack of a fast single-pass approximation.
- Pivotal Tuning Inversion breaks the reconstruction–editability trade-off through surgical personalization of the generator for the desired image.The approach leverages StyleGAN’s disentanglement between appearance and geometry.
- PTI increases quality at the cost of additional computation, requiring around three minutes to incorporate a new identity.The paper reports that this cost is similar to some optimization-based inversion methods.
- The method can be applied to several individuals, supporting editing sessions involving multiple identities.The paper gives a movie casting team as an example application.
- A trainable mapper approximating PTI in a short forward pass remains desirable to reduce the computational cost of real-image editing.The paper also proposes using multiple photographs per individual and extending PTI beyond StyleGAN as future directions.
- Ad-hoc fine-tuning of pretrained generators may apply to editing, manipulation, and other generation-based machine-learning tasks.
A. Locality Regularization
The locality regularization study evaluates how α balances target reconstruction against preservation of random in-domain latent codes. Visual interpolation shows that increasing α shifts outputs away from the pivot and can eventually harm realism.
- For small α values such as α = 8, interpolated outputs remain very similar to the pivot image.
- The evaluation measures both target-image reconstruction and reconstruction of sampled random latent codes using MSE and LPIPS.Random codes are compared with the original pretrained generator, while target reconstruction is measured after tuning.
B. Visual Results
The visual evaluations show PTI reconstructing and editing multiple identities, including uncurated CelebA-HQ examples and challenging real images. Results emphasize preservation of identity and fine out-of-distribution details alongside meaningful edits.
- Visual Results: PTI achieves higher reconstruction and editing quality across additional comparisons, including challenging images.These results preserve original identity while maintaining high editability.
- Locality Regularization: Increasing α makes interpolated images progressively resemble the random image rather than the pivot, with extreme values losing realism.
- Visual Results: Multiple Barcelona Football Club and Modern Family identities are inverted into a shared StyleGAN latent space with high reconstruction and editing quality.
- Visual Results: Uncurated CelebA-HQ identities are repeatedly evaluated with smile, age, and rotation edits across Figures 21–23.
- Visual Results: Additional comparisons cover real-image editing and StyleClip hair edits with and without pivotal tuning inversion.
- Visual Results: On a challenging image, PTI performs meaningful edits while surpassing other methods in preserving fine details.
- Visual Results: PTI preserves out-of-distribution details such as earrings and complicated makeup during reconstruction.