PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing

Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, Youngjung Uh

arXiv:2104.14754v2cs.CVcs.LG

TL;DR

Real-image GAN editing requires either time-consuming latent optimization or encoder projections that lose image fidelity. StyleMapGAN addresses this gap with a spatially structured intermediate latent space and spatially variant modulation, enabling accurate real-time projection and high-quality manipulation across projection, interpolation, and local editing tasks.

  • Problem

    Real-image editing with GANs is limited by slow latent-code optimization or low-fidelity encoder-based projection.

  • Method

    StyleMapGAN uses an intermediate latent stylemap with explicit spatial dimensions and spatially variant modulation, allowing encoders to project images into spatial latent representations.

  • Results

    StyleMapGAN substantially improves projection quality across multiple datasets and outperforms state-of-the-art methods on projection, interpolation, and local editing.

  • Takeaways & Limitations

    The stylemap enables accurate real-time projection and high-quality image interpolation and local editing, including region transplantation with unaligned masks.

  • Takeaways & Limitations

    SEAN requires ground-truth segmentation masks for both training and testing, limiting its practical use relative to mask-free operation.

Abstract

from arXiv · show

Generative adversarial networks (GANs) synthesize realistic images from random latent vectors. Although manipulating the latent vectors controls the synthesized outputs, editing real images with GANs suffers from i) time-consuming optimization for projecting real images to the latent vectors, ii) or inaccurate embedding through an encoder. We propose StyleMapGAN: the intermediate latent space has spatial dimensions, and a spatially variant modulation replaces AdaIN. It makes the embedding through an encoder more accurate than existing optimization-based methods while maintaining the properties of GANs. Experimental results demonstrate that our method significantly outperforms state-of-the-art models in various image manipulation tasks such as local editing and image interpolation. Last but not least, conventional editing methods on GANs are still valid on our StyleMapGAN. Source code is available at https://github.com/naver-ai/StyleMapGAN.

1. Introduction

GAN latent spaces support semantic image manipulation, but applying those manipulations to real images remains difficult because inversion is either slow or low-fidelity. StyleMapGAN introduces spatially structured latent representations to enable accurate, real-time projection and local editing, with improvements across multiple manipulation tasks.

  • GANs encode rich semantics in latent spaces, but lack an inverse mapping from real images to corresponding latent codes.
  • Optimizing latent codes for individual real images can take minutes on high-end GPUs and does not guarantee a valid optimized code.
  • Encoder-based projection is real-time but loses image details because vector-based latent spaces entangle local semantics and lack spatial dimensions.
  • StyleMapGAN replaces vector-based latent representations with stylemaps containing explicit spatial dimensions for encoding local image semantics.
  • Stylemaps support high-fidelity, real-time projection and local image editing by manipulating matching spatial positions.
  • Across multiple datasets, StyleMapGAN substantially improves projection quality and outperforms state-of-the-art methods on projection, interpolation, and local editing.

2. Related work

Prior work projects images into GAN latent spaces through optimization or learned encoders, while local editing methods target specific regions. These approaches face substantial computation, supervision, real-image fidelity, or vector-representation limitations.

  • Optimization-based editing: Optimization-based methods iteratively update pretrained GAN latent codes to project real images, with some approaches combining pixel reconstruction and semantic-domain constraints.
  • Optimization-based editing: Optimization methods can be three orders of magnitude slower than faster alternatives, underscoring their computational cost for projection.
  • Learning-based editing: Learning-based methods use encoders to infer latent codes directly, including adversarial, variational, and intermediate-latent-space approaches.
  • Local editing: Local editing methods manipulate specific parts rather than global appearance, but several do not target real images or require paired images and segmentation masks.

3. StyleMapGAN

StyleMapGAN uses a spatially dimensional stylemap and spatially varying modulation to support accurate real-image projection and local editing. Its encoder, generator, and discriminator are jointly trained with reconstruction, perceptual, adversarial, latent, and domain-guided losses.

  • Stylemap-based generator: StyleMapGAN replaces vector-based latent representations with a spatially dimensional stylemap for encoding local image semantics.The stylemap is an intermediate latent representation designed for accurate, real-time projection and local manipulation.
  • Stylemap-based generator: The stylemap is resized through convolutions and upsampling to match synthesis features at different spatial resolutions.Learned affine transforms produce spatial modulation parameters γ and β, followed by element-wise multiplication and addition.
  • Stylemap-based generator: The generator removes per-pixel noise because the stylemap already supplies spatially varying inputs.This simplifies projection and editing by retaining a single spatially varying input source.
  • Training procedure and losses: The networks are jointly trained using pixel-level, perceptual, latent, adversarial, and domain-guided losses.The encoder and generator reconstruct real images, the encoder reconstructs stylemaps, and the discriminator distinguishes real images from generated images.
  • Local editing: Local editing blends encoded stylemaps from an original and reference image according to a mask, including masks of arbitrary shape.The mask is shrunk by max pooling, and blending is generally performed in the w+ space for detailed manipulation.
  • Local editing: Rough masks as coarse as 8 × 8 can produce plausible images, reducing the need for detailed user-provided masks.The operation can also be revised for unidentical masks between two images.

4. Experiments

Experiments evaluate StyleMapGAN across projection, generation, interpolation, and local editing tasks on multiple datasets. The results support 8×8 stylemaps as a practical resolution and show accurate, fast projection with strong editing quality.

  • Experimental setup: Experiments cover projection, generation, interpolation, and local editing on CelebA-HQ, AFHQ, and LSUN Car & Church.The evaluation uses MSE, LPIPS, FID, FIDlerp, and AP-related local-editing assessments.
  • Stylemap resolution: Higher stylemap resolution improves reconstruction, while 8×8 already provides accurate reconstruction and later gains become visually negligible.The encoder-generator inference time is less than 0.1s with almost perfectly reconstructed images.
  • Local editing: 8×8 produces the most plausible local edits, balancing seamless integration with preservation of the original and reference identities.Resolutions of 4×4 reconstruct poorly, while resolutions at least 16×16 make edited regions too heterogeneous.
  • Stylemap resolution: 9.97 FIDlerp is achieved by the 8×8 model on CelebA-HQ, compared with 10.72 for 4×4, 11.05 for 16×16, and 12.10 for 32×32.The authors select 8×8 for subsequent experiments based on editing quality and FIDlerp.
  • Real image projection: StyleMapGAN achieves better reconstruction quality than competitors except Image2StyleGAN, while providing the best FIDlerp and at least 100× faster projection than optimization-based baselines.Image2StyleGAN has the smallest reconstruction error but requires minutes and produces poor interpolation quality.
  • Local editing: StyleMapGAN seamlessly composes images in local editing, preserving details and better reflecting original and reference regions than the compared methods.On AFHQ, it combines species with preserved details, while other methods tend toward one species.

5. Discussion and Conclusion

The paper concludes that explicit spatial dimensions in StyleMapGAN’s stylemap improve real-image projection and enable high-quality interpolation and local editing in real time. It also identifies applying the representation to conditional GANs or variational autoencoders as future work.

  • Discussion and Conclusion: StyleMapGAN introduces explicit spatial dimensions into the latent space through a representation called a stylemap.The representation is proposed to address practical invertibility for editing real images with unconditional GANs.
  • Discussion and Conclusion: StyleMapGAN accurately projects real images into latent space in real time and synthesizes high-quality interpolated and locally edited outputs.The conclusion reports these advantages over prior approaches based on extensive evaluation.
  • Discussion and Conclusion: The local-editing pipeline projects original and reference images into stylemaps, resizes them across resolutions, blends them with a binary mask, and generates the output from the mixed stylemaps.The pipeline supports spatially mixed editing through a learned encoder, stylemap resizer, and generator.

A. Local editing in w+ space

The w+ editing method enables detailed local editing by applying high-resolution masks to resized stylemaps, overcoming the coarse resolution of the base stylemap. It blends the original and reference stylemaps using resized masks.

  • Local editing in w+ space: The w+ method enables detailed local editing with high-resolution masks regardless of the base stylemap resolution.This avoids the coarse mask resolution imposed by the 8×8 stylemap.
  • Local editing in w+ space: The edited resized stylemap is formed by alpha blending the original and reference w+ stylemaps.The blending uses a resized mask for each stylemap resolution.
  • Local editing in w+ space: The resized mask is obtained by max pooling, allowing precise blending even when its shape does not align with the 8×8 stylemap.This supports detailed spatial control in w+ space.

B. Experiments in the high-resolution dataset

On 1024 × 1024 FFHQ, StyleMapGAN is evaluated against StyleGAN2 and Image2StyleGAN, including a smaller variant and joint-learning analysis. StyleMapGAN outperforms the baselines across metrics, while its light version reduces model size but loses some reconstruction detail.

  • Comparison with baselines: StyleMapGAN outperforms the baselines in all reported metrics on 1024 × 1024 FFHQ.The comparison includes StyleGAN2 and Image2StyleGAN under the stated evaluation protocol.
  • Comparison with baselines: Image2StyleGAN reconstructs images well but has poor interpolation quality and lengthy runtime, limiting its suitability for image editing.The passage also describes its interpolation results as rugged.
  • Comparison with baselines: StyleMapGAN-Light is 2.5× smaller than the original version while still outperforming the baselines.Its reconstruction images lack some detail, but its FIDlerp is better than the original version.
  • Training strategy: Joint learning stabilizes training and improves network performance compared with sequential learning.Sequentially training the encoder after the generator fails to reconstruct images; joint learning lets the generator and encoder affect each other.

C. Implementation details

StyleMapGAN uses an MLP mapping network and a convolutional encoder, with joint training of the generator, encoder, and discriminator. The MLP produces more natural local-editing results than autoencoder- and convolution-based alternatives.

  • Architecture: The mapping network is an eight-layer MLP followed by a reshape layer, while the encoder follows the discriminator architecture through the 8 × 8 layer.The MLP channel sizes are 64 except for the final 4,096-channel layer; the encoder omits minibatch discrimination.
  • Training: The generator, encoder, and discriminator are jointly trained for more stable training and higher performance than separate training.The stated training setup otherwise mostly follows StyleGAN2 settings.
  • Mapping network design: MLP-based mapping produces more natural local-editing images than autoencoder-based and convolution-based mapping networks.The comparison is shown in Figure 10.
  • Mapping network design: Autoencoder-based mapping fails to produce realistic images from projected stylemaps, whereas Gaussian-latent training covers more latent values for editing.The passage contrasts discrete image-only inputs with the continuous Gaussian latent space used by StyleMapGAN.
  • Mapping network design: Convolution-based mapping struggles with reconstruction, while fully connected MLP weights and inputs provide a more flexible latent space.The passage attributes the convolutional limitation to mappings bounded to local areas.

D. Loss details

StyleMapGAN combines adversarial, domain-guided, latent-reconstruction, image-reconstruction, perceptual, and R1 losses. The ablations emphasize that different losses trade off reconstruction quality against interpolation and overall performance.

  • Loss overview: All six losses are needed for the best editing quality measured by FIDlerp, although MSE and LPIPS can degrade reconstruction quality.The ablation study removes one loss at a time, using CelebA-HQ at 256 × 256 resolution with an 8 × 8 stylemap.
  • Adversarial loss: Adversarial loss supports a smooth latent manifold for interpolation; removing it produces unnatural interpolation and checkerboard artifacts.The loss is applied to random Gaussian generations and reconstructions of input images.
  • Domain-guided loss: Domain-guided loss keeps projected latent codes in the GAN’s original latent space, facilitating smooth real-image editing.It uses adversarial training on images generated from real images through the encoder and generator.
  • Latent reconstruction loss: Latent reconstruction loss trains the encoder to recover latent codes for generated images and reduces bias against pixel-level reconstruction.It encourages projections to remain in the semantic domain of the original latent space.
  • Reconstruction losses: Image reconstruction loss enforces pixel-level similarity between input and output images, while perceptual loss improves perceptual-level reconstruction.Without image reconstruction loss, visual reconstruction fails; LPIPS is used for perceptual loss to avoid blurry outputs.
  • R1 regularization: R1 regularization improves training stability, and removing it degrades performance across all metrics.Lazy regularization is applied every 16 discriminator steps.

E. Additional results

Additional qualitative experiments examine generation, reconstruction, local editing, unaligned transplantation, compatibility with other latent-editing methods, and limitations. These results broaden the evaluation beyond the main comparisons.

  • Additional results: The additional-results section covers random generation, reconstruction, local editing, unaligned transplantation, compatibility with other latent-editing methods, and limitations.The subsections provide expanded qualitative results for each of these areas.
  • Additional results: The method is presented as applicable to other latent-based editing methods in the additional qualitative experiments.The section also includes expanded comparisons and a discussion of limitations.

E.1. Random generation

StyleMapGAN generates high-quality images from random Gaussian noise across CelebA-HQ, AFHQ, LSUN Car, and LSUN Church, using dataset-specific stylemap resolutions and truncation.

  • Random generation is evaluated on CelebA-HQ, AFHQ, LSUN Car, and LSUN Church.
  • StyleMapGAN uses 8 × 8 stylemaps except for AFHQ, where 16 × 16 improves generation quality.
  • The model applies the truncation trick with ψ = 0.7 to generate high-quality images.
  • Figure 12 presents uncurated samples and FID values for the datasets.

E.2. Image projection & Interpolation

StyleMapGAN is compared with encoder- and optimization-based methods for projection, interpolation, local editing, and transplantation. Its spatial stylemap supports local semantic manipulation and unaligned region transfer, while performance degrades for mismatched poses and target sizes.

  • Image projection & Interpolation: Encoder-based baselines provide real-time projection but have lower projection quality, whereas Image2StyleGAN reconstructs faithfully but struggles with latent interpolation.
  • Image projection & Interpolation: The method can transplant arbitrary numbers and locations of reference regions while adjusting their color tone and structure to the original image.
  • Image projection & Interpolation: StyleMapGAN supports local semantic manipulation by applying semantic directions only within specified stylemap locations.
  • Image projection & Interpolation: Local manipulation avoids changing undesired regions and enables partial attributes such as lip makeup or beard edits.
  • Image projection & Interpolation: Style mixing and unaligned transplantation transfer local structure from reference images into target regions.
  • Image projection & Interpolation: Transplantation fails more often when original and reference images differ in pose or target semantic size, especially as stylemap resolution increases.
Loading 2104.14754v2…