Source-linked AI summary
Semantic Photo Manipulation with a Generative Image Prior
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba
TL;DR
GANs struggle to precisely reproduce existing photographs and to make synthesized edits fit their original context. This paper adapts a GAN prior to each image, enabling semantic editing with an image-specific generator; the resulting system supports varied edits and outperforms compositing-based methods in human perception studies. The approach remains limited by per-edit optimization time and imperfectly disentangled latent spaces.
Problem
GANs have difficulty precisely reproducing an existing natural photograph and integrating newly synthesized pixels with its original content.
Method
The method adapts a GAN generator to the statistics of an individual image while retaining semantic representations for latent-space edits.
Results
The interactive GANPaint editor supports inserting, removing, and changing objects, and its method outperforms compositing-based methods in human perception studies.
Takeaways & Limitations
Image-specific adaptation produces new visual content consistent with the original photograph while preserving faithfulness in unedited regions.
Takeaways & Limitations
Each edit requires about 30 seconds of optimization on a modern GPU, while faster variants introduce more artifacts; latent-space entanglement can also cause unwanted interactions.
Abstract
from arXiv · showhide
Despite the recent success of GANs in synthesizing images conditioned on inputs such as a user sketch, text, or semantic labels, manipulating the high-level attributes of an existing natural photograph with GANs is challenging for two reasons. First, it is hard for GANs to precisely reproduce an input image. Second, after manipulation, the newly synthesized pixels often do not fit the original image. In this paper, we address these issues by adapting the image prior learned by GANs to image statistics of an individual image. Our method can accurately reconstruct the input image and synthesize new content, consistent with the appearance of the input image. We demonstrate our interactive system on several semantic image editing tasks, including synthesizing new objects consistent with background, removing unwanted objects, and changing the appearance of an object. Quantitative and qualitative comparisons against several existing methods demonstrate the effectiveness of our method.
1 INTRODUCTION
GAN-based semantic editing promises concept-level control over natural photographs, but precise reconstruction and seamless integration of synthesized pixels remain challenging. The paper addresses both issues with image-specific adaptation and demonstrates an interactive editor for varied semantic edits.
- Motivation: Generative models can represent editable visual concepts while preserving realism during high-level photo manipulation.Examples include adding a lit lamp, changing leaf color, and manipulating object types or attributes.
- Technical Challenges: Two challenges are precise reproduction of the input photograph and compatibility between synthesized pixels and existing image content.Rough reconstructions capture content but differ in visual details, making edited regions difficult to integrate.
- Image-Specific Adaptation: The method learns an image-specific generator G′ that preserves unedited regions while producing semantically edited content consistent with the original photograph.G′ shares semantic representations with the original generator while adapting to the statistics of the individual input image.
- Interactive System: GANPaint editor supports inserting objects and changing attributes through an interactive interface.The system is presented as a general-purpose editor for a range of semantic photo manipulation tasks.
- Evaluation: The method outperforms compositing-based methods in human perception studies and ablations show the importance of image-specific adaptation.The reported comparisons include fewer visible artifacts and more realistic outputs than the compared reconstruction variants.
2 RELATED WORK
Prior work covers GAN-based synthesis and low-level photo editing, but interactive semantic manipulation of entire natural photographs remains less developed. This method targets that gap by generating final results across natural scenes rather than restricting edits to single low-resolution objects.
- GAN-Based Synthesis: GANs support user-guided synthesis tasks including sketches, face animation, language-conditioned photos, inpainting, and generated-scene manipulation.Most prior work focuses on generating new images from scratch given user controls.
- GAN-Based Photo Manipulation: Earlier GAN-based photo-manipulation methods often handle a single object at 64x64 resolution and require post-processing.The cited approaches differ from this work’s stated goal of semantic manipulation across an entire natural scene.
- Positioning: The proposed method directly generates final results and enables semantic manipulation of an entire natural scene.This positions the approach against prior work focused on single objects or image synthesis from scratch.
- Interactive Photo Manipulation: Traditional photo-editing tools handle low-level features effectively but high-level semantic edits often require manual geometry, layout, object selection, RGBD data, or segmentation.These systems can use simple interactions such as scribbles for low-level changes, while semantic operations demand additional structure or annotation.
- Deep Image Manipulation: Deep learning systems have produced compelling results in inpainting, colorization, stylization, and enhancement, with learning-based computation avoiding handcrafted heuristics.Recent work also incorporates user interaction into end-to-end learning systems.
3 METHOD
The method adapts a GAN to each input photograph, combining latent editing with image-specific reconstruction so unedited regions remain faithful while new content stays consistent. It supports interactive object insertion, removal, and appearance changes through semantic feature manipulation.
- Semantic editing operations: The user edits the intermediate latent representation, and the modified representation is rendered to produce the final image.The system uses spatial feature maps between the fourth and fifth convolutional layers of a 15-layer Progressive GAN producing 256 × 256 images.
- Image reconstruction: The pipeline first reconstructs an input photograph by optimizing its latent representation and adapting the generator when ordinary GAN reconstruction is insufficient.The reconstruction loss combines color-pixel and perceptual terms, with an encoder providing initialization for optimization.
- Image-specific adaptation: The image-specific generator G′ matches the input outside the edited region while preserving the original generator’s semantic representations.Only the unedited pixels are required to match exactly for a successful edit, rather than the entire image.
- Image-specific adaptation: G′ preserves early high-level layers and perturbs only later fine-grained layers, separating object semantics and layout from details such as edges and colors.A small network R produces layer perturbations, while regularization discourages large perturbations.
- Interactive editing: The adaptation takes less than 30 seconds on one GPU, reaching an average unedited-reconstruction PSNR of 30.6 after 1000 optimization steps.Using 100 steps reduces the PSNR to 24.6, providing a quality–speed trade-off for previews or faster processing.
- Semantic editing operations: GANPaint supports adding, removing, and altering objects by blending class-specific activation vectors into selected spatial regions.Setting s = 0 removes a class, whereas s > 0 adds it; the approach focuses on object-centric representations such as trees and doors.
4 EXPERIMENTAL RESULTS
The experiments compare image-specific adaptation with compositing methods and simpler generator adaptations, then demonstrate editing on training and in-the-wild images, style variation, and latent-vector recovery.
- 4.1 Comparing Image-Specific Adaptation to Compositing: Human evaluations find the method more realistic than traditional image-blending approaches on average.The comparison used 206 Amazon MTurk workers and 1,200 pairwise comparisons per method across 20 edited images.
- 4.2 Ablation Studies: Compared with two generator ablations, the method produces fewer visible artifacts and is rated more realistic on average.The directly adapted generator introduces high-frequency differences and color artifacts near edited regions.
- 4.3 Qualitative Results: On LSUN images, image-specific adaptation renders edited results for the same user requests after the input is inverted and a segmentation mask is applied.The workflow compares the original image, its inversion, the user mask, the unadapted edit, and the adapted edit.
- 4.3 Qualitative Results: On newly collected church images, the method edits modeled building content even when cars and lampposts are not generated by the GAN.The church GAN models shapes, doors, windows, and surfaces, while the image-matching method applies edits in their presence.
- 4.4 Style Variants: Style variants of inserted objects come from changing the reference image or varying the strength of style adaptation while fixing the reference.Figure 8 demonstrates appearance changes for domes, grass, and trees.
- 4.5 Recovering the Latent Vector z: Latent recovery is nearly perfect for images generated by G, but far from perfect for real images outside the generator’s image distribution.The method first trains an encoder and then optimizes the latent vector; optimizing earlier layers regularizes the fourth-layer representation.
5 LIMITATIONS AND DISCUSSION
The method remains limited by optimization time, imperfect disentanglement, and the quality and resolution of its underlying generator, although the learned image prior can improve realism with less user input.
- Interactive speed: Each edit requires about 30 seconds of optimization on a modern GPU.Fewer steps, incremental optimization, or full-image adaptation can provide faster previews but introduce more artifacts.
- Disentanglement: Latent spaces are not fully disentangled, so edits such as object removal can affect unrelated content.Chair removal may leave distorted remnants, and appearance edits vary in brittleness across object classes.
- Disentanglement: Some object attributes remain difficult to manipulate selectively; dome shape could be varied, but dome color could not.
- Generator quality: Current result quality and resolution are limited by the deep generator, which may omit details such as cars and signage.The method cannot add objects that are not modeled by the generator.
- Discussion: The learned natural image prior can produce more realistic results with less user input in many computer graphics applications.The authors describe the work as a small step toward leveraging deep generative models for semantic photo manipulation.