Source-linked AI summary
High-Fidelity GAN Inversion for Image Attribute Editing
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, Qifeng Chen
TL;DR
Existing GAN inversion methods face a trade-off: low-rate codes preserve editability but lose image-specific details, while high-rate codes improve reconstruction at the cost of editability. The paper introduces distortion consultation inversion and adaptive distortion alignment to combine fidelity with editing, and experiments show clear improvements in inversion and editing quality.
Problem
Low-rate GAN inversion codes often lose image-specific details, while increasing latent rate can improve reconstruction but harm editability.
Method
The framework projects an inversion distortion map into a high-rate latent map for consultation fusion and uses adaptive distortion alignment with self-supervised training for editing.
Results
Extensive experiments show improved preservation of image-specific details in both reconstructed and edited results across challenging images, domains, and videos.
Takeaways & Limitations
Combining low-rate latent editing with distortion consultation provides high-fidelity reconstruction and editing while retaining robustness to occlusion and extreme viewpoints.
Takeaways & Limitations
The method has difficulty handling large misalignment cases because ADA training does not cover extreme misalignment.
Abstract
from arXiv · showhide
We present a novel high-fidelity generative adversarial network (GAN) inversion framework that enables attribute editing with image-specific details well-preserved (e.g., background, appearance, and illumination). We first analyze the challenges of high-fidelity GAN inversion from the perspective of lossy data compression. With a low bit-rate latent code, previous works have difficulties in preserving high-fidelity details in reconstructed and edited images. Increasing the size of a latent code can improve the accuracy of GAN inversion but at the cost of inferior editability. To improve image fidelity without compromising editability, we propose a distortion consultation approach that employs a distortion map as a reference for high-fidelity reconstruction. In the distortion consultation inversion (DCI), the distortion map is first projected to a high-rate latent map, which then complements the basic low-rate latent code with more details via consultation fusion. To achieve high-fidelity editing, we propose an adaptive distortion alignment (ADA) module with a self-supervised training scheme, which bridges the gap between the edited and inversion images. Extensive experiments in the face and car domains show a clear improvement in both inversion and editing quality.
1. Introduction
GAN inversion enables attribute editing by projecting real images into a pretrained generator’s latent space, but existing methods struggle to preserve image-specific details while retaining editability. The paper addresses this rate–distortion–editability trade-off with distortion consultation and adaptive alignment.
- Image attribute editing modifies desired attributes while preserving other image details, using GAN inversion to project images into a pretrained generator’s latent space.
- Optimization-based inversion can improve reconstruction by overfitting, whereas encoder-based methods are faster and more editable but often lose high-frequency image-specific details.
- Low-rate latent codes lose information, while directly increasing latent rate can reduce distortion but substantially harm interpretability and editability.
- The proposed distortion consultation branch equips a low-rate encoder with image-specific information to improve reconstruction without compromising editability.
- Adaptive distortion alignment adjusts the distortion map for edited images, addressing misalignment between inversion and editing results.
- Experiments report significantly improved detail preservation in reconstructed and edited results, with robustness to viewpoint and illumination changes for temporally consistent video editing.
2. Related Work
Related work covers GAN inversion strategies, latent-space choices, and semantic editing directions. These approaches reflect recurring trade-offs among reconstruction quality, inference speed, detail preservation, and editability.
- GAN inversion methods include optimization-based, encoder-based, and hybrid approaches, with optimization methods generally offering high reconstruction quality but slower inference.
- StyleGAN inversion commonly uses Z, W, W+, S, or P spaces, while other work employs multi-scale latent codes for ProgressGAN.
- The rate–distortion–edit trade-off contrasts low-rate frameworks that lose details with high-rate frameworks that improve reconstruction but reduce interpretability and editability.
- Low-rate latent spaces inevitably lose reconstruction details, motivating a distortion consultation branch to convey high-frequency image-specific information.
- Semantic editing methods learn latent directions through supervised classifiers, annotated data, geometric transformations, or self-supervised learning.
3. Approach
The framework treats low-rate GAN inversion as a rate–distortion–editability trade-off and adds distortion consultation to preserve image-specific details. DCI supplements the low-rate latent code with a distortion-derived latent map, while ADA aligns those details for attribute editing.
- Motivation: Low-rate latent codes preserve coarse, common patterns but lose image-specific details, whereas higher-rate codes improve reconstruction at the expense of editability.The paper frames this tension as the Rate-Distortion-Edit trade-offs.
- Overview: The framework keeps a low-rate latent code for editable semantic manipulation and conveys lost image-specific details through a distortion consultation branch.This design aims to improve fidelity without the trivial overfitting associated with simply increasing latent rate.
- Distortion Consultation Inversion: The basic encoder produces W and an initial reconstruction, then computes the distortion map ˜∆ = X − ˆXo and encodes it as a high-rate latent map C.The generator uses both W and C to produce the high-fidelity reconstruction ˆX = G0(W, C).
- Distortion Consultation Inversion: Layer-wise consultation fusion converts C into gate and high-frequency detail maps, filtering undesired features while augmenting low-fidelity generator features.Fusion is restricted to early generator layers to avoid overfitting the inversion result.
- Adaptive Distortion Alignment: Editing moves W along a semantic direction, but the original distortion map becomes misaligned after attribute changes such as age, pose, or expression.Applying the unaligned map directly can introduce obvious artifacts.
- Adaptive Distortion Alignment: ADA adaptively aligns the distortion map with the edited image before consultation encoding, enabling the edited generator output to preserve more details.It is trained self-supervised using simulated misaligned distortion maps and an L1 alignment loss, while training uses only inversion images.
4.1. Settings
The evaluation uses separate training and evaluation datasets for faces, while cars use one dataset for both. Attribute editing applies InterfaceGAN to faces and GANSpace to cars.
- FFHQ trains the face model, while CelebA-HQ evaluates it; Stanford Cars serves both training and evaluation for cars.
- Figure 4 presents visual comparisons for face inversion and editing.
- Table 1 reports quantitative comparisons of inversion quality on faces.
- InterfaceGAN provides face editing directions, whereas GANSpace provides car editing directions.
4.2. Evaluation
The method is evaluated against encoder-based and optimization-based inversion approaches using reconstruction, visual, and user-study comparisons. It improves reconstruction quality over encoder baselines, is much faster than optimization methods, and receives higher editing preferences.
- The method substantially outperforms encoder-based baselines in reconstruction quality and is considerably faster than optimization-based methods at inference.The comparison includes pSp, e4e, Restyle, and two optimization-based approaches.
- Visual comparisons show robustness to occlusion and extreme viewpoints while preserving backgrounds, shadows, reflections, accessories, expressions, and appearance.
- Table 2 reports user-study preference rates of Ours against a baseline.
- In a study of 1,500 votes from 30 participants, the method outperforms baselines by a large margin for editing fidelity and proper editing.Participants compared source, proposed, and baseline editing results across extensive attributes.
4.3. Ablation Study
The ablation study isolates distortion consultation and adaptive distortion alignment. DCI improves reconstruction detail retention, while ADA prevents distortion maps from degrading edited-image quality.
- Effect of Distortion Consultation: Adding the DCI consultation branch improves robustness to occlusion and extreme poses and preserves more details in reconstructed images.
- Effect of Adaptive Distortion Alignment: Without ADA, the distortion map fails to generalize to edited images and degrades generated-image quality.
- Effect of Adaptive Distortion Alignment: Image-space warping and fusion using facial-landmark coordinates produce obvious artifacts.
4.4. Application on Video Editing
Video editing is evaluated through temporal detail consistency under pose and viewpoint changes. The proposed method preserves details more robustly across frames, while comparisons also document its speed and component effects.
- Video editing requires temporal consistency of details across frames, making reconstruction fidelity especially important.
- Figure 5 compares the method visually with optimization-based methods and reports much faster inference.
- Figure 6 examines the effects of distortion consultation inversion.
- Figure 7 examines ADA by integrating the distortion map in image space and feature space and showing editing results.
- The proposed method is more robust to cross-frame pose and viewpoint changes and achieves higher-fidelity detail preservation.
5. Conclusion
The proposed GAN inversion framework uses distortion consultation and adaptive distortion alignment to improve high-fidelity image attribute editing while preserving image-specific details. It remains robust to occlusion and extreme viewpoints, but large misalignment cases remain difficult.
- Distortion consultation uses a distortion map as a high-rate reference to enhance encoder-based reconstruction without compromising editability.The consultation branch supplies lost image information, including background, appearance, and illumination details.
- Adaptive distortion alignment and distortion consultation improve preservation of image-specific details in both reconstruction and editing.The reported improvements include preservation of background, appearance, and illumination.
- The framework is more robust to challenging cases such as occlusion and extreme viewpoints.
- Large misalignment cases remain a limitation because the ADA training data does not cover extreme misalignment.The authors note that ADA may be insufficient when editing images with large viewpoint changes.