Source-linked AI summary
In-Domain GAN Inversion for Real Image Editing
Jiapeng Zhu, Yujun Shen, Deli Zhao, Bolei Zhou
TL;DR
GAN inversion must recover real images while keeping inverted codes within the generator’s learned semantic domain for editing. The proposed in-domain approach combines a domain-guided encoder with domain-regularized optimization, yielding faithful reconstruction and improved semantic editing support over existing approaches.
Problem
GANs learn latent-space semantics useful for image manipulation, but real-image inversion methods often prioritize pixel reconstruction without ensuring semantically meaningful codes.
Method
The method trains a domain-guided encoder to produce in-domain latent codes, then uses encoder-based domain-regularized optimization to refine pixel reconstruction.
Results
Qualitative and quantitative experiments show faithful recovery of target images at low-level pixels and high-level semantics, significantly surpassing existing approaches.
Takeaways & Limitations
In-domain codes better support image editing by reusing semantic knowledge learned by GAN models.
Takeaways & Limitations
All experiments use the StyleGAN W latent space, although the approach can also be performed in Z space.
Abstract
from arXiv · showhide
Recent work has shown that a variety of semantics emerge in the latent space of Generative Adversarial Networks (GANs) when being trained to synthesize images. However, it is difficult to use these learned semantics for real image editing. A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. However, existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. As a result, the reconstructed image cannot well support semantic editing through varying the inverted code. To solve this problem, we propose an in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing. We first learn a novel domain-guided encoder to project a given image to the native latent space of GANs. We then propose domain-regularized optimization by involving the encoder as a regularizer to fine-tune the code produced by the encoder and better recover the target image. Extensive experiments suggest that our inversion method achieves satisfying real image reconstruction and more importantly facilitates various image editing tasks, significantly outperforming start-of-the-arts.
1 Introduction
GANs learn semantic controls in latent space, but applying them to real images requires inversion methods that preserve both reconstruction fidelity and semantic meaning. The proposed in-domain approach addresses this gap with a domain-guided encoder and domain-regularized optimization, enabling diverse real-image editing tasks.
- GAN latent spaces encode semantics that can control generated attributes, but GANs cannot directly infer latent codes from particular real images.
- Existing GAN inversion methods primarily reconstruct input pixels, leaving whether inverted codes remain in the GAN’s semantic domain unresolved.
- The method first trains a domain-guided encoder that maps images to semantically meaningful latent codes, then uses the encoder as a regularizer during instance-level optimization.
- The approach targets both pixel-level recovery and semantic alignment, allowing a fixed GAN generator to support facial manipulation, code interpolation, and semantic diffusion.
- Qualitative and quantitative experiments report faithful recovery of low-level pixels and high-level semantics, significantly surpassing existing approaches across image-editing tasks.
2 In-Domain GAN Inversion
In-domain GAN inversion targets both faithful image reconstruction and semantic alignment with the generator’s learned latent domain. The approach combines a domain-guided encoder trained through the fixed generator with domain-regularized optimization that preserves semantic validity during refinement.
- Problem setting: GAN inversion seeks a latent code that reconstructs a real image while aligning with the semantic space learned by the pretrained generator.The paper performs experiments in StyleGAN’s W space, while noting that the approach can also operate in Z space.
- Domain-guided encoder: Encoder outputs are constrained to the generator’s semantic domain because the generator participates in training and the reconstructed image supplies the objective.This is intended to produce codes that align with the generator’s semantic knowledge rather than merely matching latent targets.
- Conventional encoder: Conventional encoders recover latent codes from synthesized images without ensuring that their outputs align with the generator’s learned semantic knowledge.Latent-code supervision alone omits generator gradients and is argued to provide insufficiently powerful supervision for accurate inversion.
- Domain-guided encoder: The domain-guided encoder feeds its output through the fixed generator and trains on real images, making image-space reconstruction the supervision source.The design incorporates generator semantics and uses perceptual and discriminator-related objectives to improve reconstruction quality.
- Domain-regularized optimization: Domain-regularized optimization uses the trained domain-guided encoder as a regularizer while refining the latent code to recover the target image.The regularizer addresses the limited representation capability of an encoder alone and prevents unconstrained optimization from producing out-of-domain inversions.
3 Experiments
The experiments evaluate semantic preservation, reconstruction quality, inference speed, and real-image editing on face and tower data. Across these evaluations, in-domain inversion outperforms existing approaches while balancing reconstruction fidelity against semantic meaningfulness.
- Experimental Setup: The evaluation uses FFHQ faces and LSUN scene images, with StyleGAN models and 7,000 real face images for semantic analysis.Facial attributes include age, gender, eyeglasses, and pose; the main paper reports tower results for LSUN.
- Semantic Analysis: Precision-recall curves show that in-domain inversion produces more semantically meaningful codes than Image2StyleGAN across facial attributes.The evaluation directly classifies facial attributes from inverted codes using a fixed StyleGAN model.
- Inversion Quality and Speed: In-domain inversion surpasses competing methods on FID, SWD, and MSE, while domain-guided encoding provides comparable inference time and optimization is approximately 35X faster than Image2StyleGAN.The full in-domain method also gives the best qualitative reconstruction quality in Fig. 4.
- Real Image Editing: During interpolation, in-domain codes produce smoother faces and higher-quality tower transitions than Image2StyleGAN, avoiding distorted attributes, artifacts, and blurriness.Interpolated towers remain high-quality even when transitioning between different spire configurations.
- Real Image Editing: In-domain inversion improves facial and tower attribute manipulation, preserving identity and limiting unintended changes to unrelated image regions.For tower editing, cloud manipulation barely affects the tower, and the method outperforms Image2StyleGAN on all reported manipulation metrics.
- Ablation Study: Increasing λ_dom makes inverted codes more semantically meaningful but reduces per-pixel reconstruction quality, so the method uses λ_dom = 2 in practice.The ablation compares λ_dom = 0, 2, and 40 and exposes a reconstruction–manipulation trade-off.
4 Discussion and Conclusion
The paper frames GAN inversion as requiring both pixel-level reconstruction and semantic alignment, and presents in-domain inversion as a way to obtain semantically meaningful codes for editing.
- The paper argues that inverted codes should capture semantic information beyond merely recovering the target image’s pixel values.
- In-domain inversion is introduced as a novel approach for explicitly considering the semantic knowledge encoded in a pretrained GAN.
- The method aims to produce codes that support semantic image manipulation by remaining within the GAN’s learned domain.
A Overview
The appendix extends evaluation beyond the main-paper experiments to reconstruction, interpolation, manipulation, style mixing, and semantic diffusion analyses.
- The appendix reports bedroom reconstruction experiments using an LSUN bedroom model.
- Additional interpolation and image-manipulation results evaluate whether in-domain GAN inversion recovers targets at both pixel and semantic levels.
- The appendix also includes style-mixing results and a detailed analysis of semantic diffusion.
B Image Reconstruction
On bedroom reconstruction, the proposed domain-guided encoder and subsequent domain-regularized optimization improve reconstruction quality over existing inversion approaches.
- The domain-guided encoder produces much better bedroom reconstructions than the compared methods.
- The further domain-regularized optimization surpasses Image2StyleGAN with higher reconstruction quality.
C Image Interpolation
Interpolation experiments compare Image2StyleGAN with in-domain inversion across faces, towers, and bedrooms, testing whether inverted codes retain meaningful semantics. The proposed method produces higher-quality, more interpretable interpolations across these image types.
- Evaluation scope: The evaluation compares Image2StyleGAN and in-domain inversion on face, tower, and bedroom interpolation.
- Comparison: Image2StyleGAN interpolations can contain artifacts and blurs, especially when source and target images differ substantially.
- Comparison: Some Image2StyleGAN interpolations lose semantic identity, with a sample ceasing to depict a tower.
- In-domain inversion: In-domain inversion keeps interpolated samples high quality and semantically explanatory across the evaluated image categories.
- In-domain inversion: The method interpolates diverse faces, tower types, and bedroom viewpoints while preserving realistic structures such as windows and wall paintings.
D Semantic Manipulation
The method is evaluated for semantic manipulation across face, tower, and bedroom images, with additional style-mixing results showing transfer of appearance while retaining content details.
- Semantic manipulation: Semantic manipulation is compared with Image2StyleGAN on face, tower, and bedroom images.The comparisons are reported in Figs. 18–20.
- Style mixing: Style mixing transfers style from artistic face images while preserving most details of real-person content images.The results use layer-wise latent codes and replace the content codes from the last four layers.
E Style Mixing
The style-mixing experiment transfers artistic appearance to real-person content while retaining most content details using layer-wise latent-code replacement.
- Style mixing: The method inverts style and content images into layer-wise latent codes before replacing the content image’s last four layers with style codes.This produces the style-mixing results shown in Fig. 21.
- Results: Each mixture inherits the painting style of the artistic face while maintaining most details from the real person.The figure presents style images in the first column and content images in the first row.
F Semantic Diffusion
Semantic diffusion combines encoder initialization with masked optimization to transfer a target foreground into a new context while preserving contextual appearance and target identity.
- Method: A target crop is pasted onto a context image, encoded into a latent code, and refined using masked optimization on the target foreground.The encoder supplies initialization, while the masked loss focuses reconstruction on the target region.
- Semantic diffusion: The procedure diffuses the target image into other contexts while retaining the original context image’s style.The reconstruction from the encoder output already provides a smooth target-context transition.
- Effect of crop size: Larger crop sizes provide larger references for semantic diffusion.Figure 23 compares direct copy-paste and semantic diffusion across different crop sizes.
- Comparison: Image2StyleGAN fails to produce semantically meaningful faces in the diffusion task because it focuses on pixel reconstruction rather than semantics.The reported failure includes nonsmooth stitch boundaries.