Source-linked AI summary
EditGAN: High-Precision Semantic Image Editing
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler
TL;DR
Existing GAN-based image editing methods often require extensive annotations or offer only coarse control. EditGAN edits detailed semantic part masks through latent-space optimization and reusable editing vectors, achieving high-precision, high-quality edits across varied images while preserving image quality.
Problem
Many GAN-based image editing methods require large annotated datasets or provide only high-level, coarse editing control.
Method
EditGAN jointly models images and semantic segmentations, optimizes latent codes to match edited masks, and amortizes edits into reusable latent-space editing vectors.
Results
EditGAN demonstrates high-precision editing across cars, cats, birds, and human faces, outperforming multiple baselines on identity preservation, quality preservation, and target attribute accuracy.
Takeaways & Limitations
The framework supports flexible, compositional, interactive editing with few labeled examples and can produce plausible edits beyond its training data.
Takeaways & Limitations
EditGAN is limited to images that its GAN can model, and some challenging edits still require iterative optimization for each example.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN based image editing methods often require large scale datasets with semantic segmentation annotations for training, only provide high level control, or merely interpolate between different images. Here, we propose EditGAN, a novel method for high quality, high precision semantic image editing, allowing users to edit images by modifying their highly detailed part segmentation masks, e.g., drawing a new mask for the headlight of a car. EditGAN builds on a GAN framework that jointly models images and their semantic segmentations, requiring only a handful of labeled examples, making it a scalable tool for editing. Specifically, we embed an image into the GAN latent space and perform conditional latent code optimization according to the segmentation edit, which effectively also modifies the image. To amortize optimization, we find editing vectors in latent space that realize the edits. The framework allows us to learn an arbitrary number of editing vectors, which can then be directly applied on other images at interactive rates. We experimentally show that EditGAN can manipulate images with an unprecedented level of detail and freedom, while preserving full image quality.We can also easily combine multiple edits and perform plausible edits beyond EditGAN training data. We demonstrate EditGAN on a wide variety of image types and quantitatively outperform several previous editing methods on standard editing benchmark tasks.
1 Introduction
EditGAN addresses the limited data and coarse control of many GAN-based image editing methods by enabling detailed semantic-part edits with few labeled examples. It combines latent-space optimization with reusable editing vectors and demonstrates broad, high-precision editing with preserved image quality.
- Motivation: Many existing GAN-based editing methods require large labeled datasets and provide only high-level or coarse control.These limitations restrict applicability to image types with large annotated datasets.
- Approach: EditGAN enables high-precision editing by letting users modify detailed object-part segmentation masks.The framework targets edits such as changing individual semantic parts rather than only global attributes.
- Approach: EditGAN jointly models RGB images and semantic segmentations and requires as few as 16 labeled examples.This design supports scaling across object classes and part-label choices.
- Approach: Latent-code optimization realizes a desired segmentation edit, while learned editing vectors amortize the process for direct application to other images.The vectors enable interactive use with no or few additional optimization steps.
- Results: EditGAN supports high-precision editing across cars, cats, birds, and human faces while preserving image quality and outperforming multiple baselines on several metrics.The reported metrics include identity preservation, quality preservation, and target attribute accuracy.
2 Related Work
Prior GAN-based editing work includes unsupervised latent-space manipulation, conditional generation, and embedding-based approaches, but generally lacks detailed, high-precision control. EditGAN builds on shared image-segmentation latent models and positions its approach as complementary to prior methods.
- GAN-based image editing: Unsupervised latent-space methods find interpretable or disentangled variables for editing but usually lack high-precision capabilities.EditGAN uses segmentation-based guidance to learn editing vectors that would otherwise be difficult to find independently.
- GAN-based image editing: Conditional GANs and auxiliary classifiers guide image synthesis and editing through labels, segmentations, or attributes.These approaches are part of the broader categories of GAN-based editing methods.
- GAN-based image editing: Most prior methods focus on relatively high-level edits rather than detailed semantic-part manipulation.The authors characterize EditGAN as complementary to this body of work.
- GANs and latent-space image embedding: EditGAN builds on DatasetGAN and SemanticGAN, which jointly model images and semantic segmentations using shared latent codes for semi-supervised learning.EditGAN repurposes this model design for image editing and combines it with image embedding through an encoder and optimization.
- GANs and latent-space image embedding: A concurrent approach shares similarities with DatasetGAN, but EditGAN differs methodologically by leveraging editing vectors and reporting more diverse and stronger experiments.Another related approach uses CLIP and targets text-driven editing.
3 High-Precision Semantic Image Editing with EditGAN
EditGAN jointly models images and semantic segmentations with shared latent codes, then uses segmentation edits to optimize precise image changes. Learned editing vectors amortize this optimization for interactive reuse, with optional refinement to reduce artifacts.
- 3.1 Background: StyleGAN2 maps latent codes through learned transformations into synthesis-block feature maps, while EditGAN adds a segmentation branch to model images and labels jointly.The segmentation branch follows DatasetGAN by classifying concatenated, upsampled feature maps per pixel.
- 3.2 Segmentation Training and Inference by Embedding Images into GAN’s Latent Space: An encoder initializes each image’s W+ embedding, which is iteratively refined using pixel-wise L2 and perceptual LPIPS reconstruction objectives.For GAN samples, the encoder is also regularized using their known latent codes.
- 3.3 Finding Semantics in Latent Space via Segmentation Editing: Users modify a detailed segmentation mask, and optimization finds an editing vector whose generated segmentation matches the target while preserving RGB appearance outside the editing region.The fixed generator synthesizes both image and segmentation outputs from the updated latent code.
- 3.3 Finding Semantics in Latent Space via Segmentation Editing: The editing region includes changed part labels and a 5-pixel buffer, allowing the generator freedom to model transitions between edited and unchanged areas.For wheel edits, the relevant labels can include tire, spoke, and wheelhub.
- 3.4 Different Ways of Editing during Inference: Previously learned editing vectors can be applied directly to embedded new images at interactive rates, with s_edit controlling edit magnitude and self-supervised refinement available at test time.For s_edit = 0 there is no edit, while s_edit > 1 exaggerates the operation.
- 3.4 Different Ways of Editing during Inference: Disentanglement is imperfect: editing vectors do not always transfer perfectly across images, so additional test-time optimization can remove artifacts outside the editing region.This refinement assumes the vector still induces a plausible segmentation change on the new image.
4 Experiments
Experiments evaluate EditGAN across four image categories, qualitative edit types, out-of-domain faces, and the smile benchmark. Results show detailed and composable edits, strong preservation and accuracy, interactive application, and refinement-dependent quality improvements.
- Qualitative Results: Qualitative experiments show that learned editing vectors preserve image quality while supporting multiple combined edits across people, cars, birds, and cats.Examples include opening eyes, adding hair, removing car parts, changing bird shape, and enlarging cat eyes or ears.
- Qualitative Results: EditGAN performs extreme high-precision edits, including rotating a car wheel spoke and changing pupil size, while also supporting large-scale structural modifications.The method edits semantic parts consisting of only a few pixels and can remove or reshape an entire car roof.
- Out-of-Domain Results: Editing vectors learned from in-domain faces transfer to out-of-domain MetFaces portraits when combined with 30-step self-supervised refinement.The reported operations include changing gaze, expression, eye openness, and eyebrow position.
- Quantitative Results: Using 1,875× less training labels, EditGAN outperforms MaskGAN on all three smile-edit metrics and surpasses LocalEditing, while refinement-time results outperform InterFaceGAN.The benchmark measures semantic correctness, distribution-level image quality, and identity preservation.
- Runtime and Ablations: Applying editing vectors takes 0.4 seconds, whereas conditional optimization with 30 steps takes 11.4 seconds and 30-step refinement adds 9.5 seconds.The learned vector therefore supports interactive application, while optimization and refinement trade runtime for quality.
- Runtime and Ablations: Larger edit scales degrade FID, while self-supervised refinement slightly improves image quality and cleans artifacts caused by imperfect latent disentanglement.Beyond 30 additional refinement steps, reported improvements become marginal.
5 Conclusions
EditGAN enables high-precision semantic editing through segmentation-guided latent optimization and transferable editing vectors, while remaining limited by the capabilities of its underlying GAN.
- EditGAN is limited to images that can be modeled by the GAN, making vivid city scenes challenging and some edits dependent on iterative optimization.
- Figure 11 compares InterFaceGAN and EditGAN on the smile benchmark across editing-vector scalings and optional additional optimization steps.
- EditGAN jointly models images and semantic segmentations, then edits images by optimizing latent codes conditioned on modified segmentation masks.
- Editing vectors amortize this optimization, enabling real-time interactive editing on other images with little or no further optimization.
6 Broader Impact
EditGAN may broaden creative image editing, but its increased precision also raises manipulation, authenticity, and dataset-bias concerns.
- The method offers high-precision semantic editing that can support artistic purposes and creative expression for designers, photographers, and content creators.
- More precise editing can also facilitate nefarious photo manipulation, intensifying concerns about image authenticity.
- Biases in training datasets remain present in synthesized images and are preserved during editing, motivating representative data collection and bias correction research.
Funding Statement
The work was funded by NVIDIA, with additional scholarship revenue acknowledged for two authors.
- This work was funded by NVIDIA.
- Huan Ling and Seung Wook Kim acknowledge additional student-scholarship revenue from the University of Toronto and Vector Institute.
A Model and Training Details
EditGAN uses StyleGAN2 for image generation, embeds images in W+ through encoder initialization and optimization, and adds a segmentation branch to jointly model images and pixel-wise labels.
- A.1 Image GAN: EditGAN uses StyleGAN2 as its backbone, mapping latent codes through intermediate and layer-specific codes into realistic images and deep feature maps.
- A.2 Image Encoder: The encoder embeds images into W+ space, where independent layer codes represent the generator input, using only 16 or 30 annotated image-segmentation pairs in the datasets described.
- A.2 Image Encoder: Encoder training uses LPIPS, L2, and distribution-regularization objectives, with specified hyperparameters and Adam optimization.
- A.2 Image Encoder: After encoder initialization, image embeddings are refined by 500 optimization steps in W+ without updating encoder parameters.
- A.3 Segmentation Branch: Annotated images are paired with their W+ embeddings to train a segmentation branch that predicts each pixel’s label from concatenated, upsampled feature maps.
- A.3 Segmentation Branch: The segmentation branch uses bilinear upsampling and pixel-wise cross-entropy, then defines a generator for the joint distribution of images and semantic segmentations.
- A.3 Segmentation Branch: The appendix distinguishes Iψ, the feature-map classifier, from the complete segmentation-generation module ˜Gy defined in the main paper.
- Additional Experiment Details: Experiments cover Face, Car, Bird, and Cat label schemes with 34, 32, 16, and 11 possible pixel labels, respectively.
B.1 Smile Edit Benchmark
The smile-edit benchmark evaluates semantic correctness, identity preservation, and comparison against several established editing baselines. EditGAN is assessed using consistent image embeddings, editing scales, and a trained smile classifier.
- Benchmark: The benchmark is introduced as the smile-edit evaluation used for comparison with prior methods.
- Evaluation protocol: The benchmark measures smile correctness with a CelebA-trained binary smile classifier achieving 92.2% test accuracy.Faces are resized to 256 × 256 before classification.
- Evaluation protocol: Identity preservation is evaluated using features extracted by the pretrained ArcFace network.
- Baselines: The comparison includes MaskGAN, LocalEditing, InterFaceGAN, and StyleGAN2 Distillation baselines.InterFaceGAN uses a StyleGAN2 generator and the same test image embeddings as EditGAN.
- Editing configuration: Each editing curve reports results for five editing-vector scales: s_edit ∈ [0.7, 1, 1.3, 1.5, 1.7].
B.2 Additional Results: Editing Vector Scale Experiment
The editing-vector scale experiment examines visual quality across edit strengths, with and without self-supervised refinement. On face images, direct vector application already produces nearly perfect edits, while refinement removes a subtle beard artifact.
- Editing-vector scale experiment: The experiment evaluates editing quality at different editing-vector scales both visually and quantitatively, with and without self-supervised refinement.
- Face-image results: Figure 14 extends the scale experiment to face images using eyebrow-raising as the example edit.
- Face-image results: Pure application of the learned editing vector yields virtually perfect eyebrow edits at interactive rates.
- Refinement: Self-supervised refinement fully removes the almost unnoticeable entanglement with the beard when that artifact correction is needed.
B.3 Additional Results: Smile Edit Benchmark with more Test Images
A larger smile-edit benchmark reevaluation uses 4,000 images instead of 400 and changes the numerical estimates while preserving method rankings and conclusions. EditGAN retains the best attribute accuracies and identity scores.
- Evaluation scale: The reevaluated benchmark computes FID, attribute accuracy, and ID score using 4,000 images, ten times the original evaluation size.The original FID comparison used 400 edited test images.
- Evaluation caveat: MaskGAN’s relatively low FID is attributed to an unfair comparison because MaskGAN alone uses the annotated training data underlying the benchmark.MaskGAN still performs significantly worse than InterFaceGAN and EditGAN in attribute accuracy and ID score.
- Results: The larger evaluation changes the reported numbers but leaves the rankings, method comparisons, and conclusions unchanged.
- Results: EditGAN achieves the best attribute accuracies and ID scores in the 4k benchmark.
- Compute: The project used approximately 14,000 GPU hours overall, including around 3,500 GPU hours for final experiments.
D Additional Qualitative Results
Additional qualitative experiments test precise disentanglement, edit composition, interpolation, and transfer across faces, cars, birds, and cats. The figures cover detailed part edits learned from image–mask pairs and applied to new images.
- D Additional Qualitative Results: EditGAN disentangles semantically related parts by lifting one eyebrow while leaving the other unchanged and enlarging one wheel while preserving the other.
- D Additional Qualitative Results: Multiple edits are combined in single results, including facial-expression, gaze, hair, wrinkle, car-body, wheel, and light modifications.
- D Additional Qualitative Results: Latent-code interpolations between editing operations are demonstrated in the accompanying video.
- Face edits: Face experiments apply learned edits for gaze position, eye closure, eyebrow raising, smiling, wrinkles, hairstyles, and pupil size to new images.
- Car and bird edits: Car experiments edit front-light size, license plates, side mirrors, wheel size and rotation, while bird experiments edit beak, belly, and head position.
- Cat edits: Cat experiments demonstrate eye size, ear size, and mouth opening edits, with learned edits subsequently applied to new images.