Source-linked AI summary
Eye In-Painting with Exemplar Generative Adversarial Networks
Brian Dolhansky, Cristian Canton Ferrer
TL;DR
Facial editing must contend with sensitivity to small structural errors, especially in familiar faces. Exemplar GANs condition in-painting on reference images or perceptual codes, producing photo-realistic, identity-preserving eye in-paintings.
Problem
People are sensitive to small facial-structure errors, particularly in familiar faces, and the uncanny valley complicates facial-feature manipulation.
Source: §1 · p. 1Method
Exemplar GANs condition a GAN on identifying information available at inference, supplied as a reference image or perceptual code and usable at multiple points in the adversarial network.
Source: §1 · p. 2Results
Exemplar-guided GAN training for eye in-painting produces photo-realistic, identity-preserving results.
Source: §1 · p. 2Takeaways & Limitations
By incorporating identifying information from reference images or perceptual codes, Exemplar GANs provide superior perceptual results and may extend to other computer-vision tasks and domains.
Source: §6 · p. 8In-painting quality was sensitive to mask placement and size.
Source: §6 · p. 8
Abstract
from arXiv · showhide
This paper introduces a novel approach to in-painting where the identity of the object to remove or change is preserved and accounted for at inference time: Exemplar GANs (ExGANs). ExGANs are a type of conditional GAN that utilize exemplar information to produce high-quality, personalized in painting results. We propose using exemplar information in the form of a reference image of the region to in-paint, or a perceptual code describing that object. Unlike previous conditional GAN formulations, this extra information can be inserted at multiple points within the adversarial network, thus increasing its descriptive power. We show that ExGANs can produce photo-realistic personalized in-painting results that are both perceptually and semantically plausible by applying them to the task of closed to-open eye in-painting in natural pictures. A new benchmark dataset is also introduced for the task of eye in-painting for future comparisons.
1. Introduction
Facial in-painting can look semantically plausible yet fail to preserve the photographed person’s identity, a concern made sharper by sensitivity to small facial errors. Exemplar GANs address this by conditioning generation on identifying information available at inference time.
1. Introduction
- Facial retouching is a growing research topic, while people are sensitive to small facial-structure errors and the uncanny valley complicates facial manipulation.The introduction notes that this sensitivity is especially strong for one’s own or familiar faces.Source: §1 · p. 1
- Deep networks can produce realistic, semantically plausible facial in-paintings, but may replace a person’s eyes with training-set examples that do not preserve distinctive ocular structure.The risk is especially salient for uncommon eye shapes.Source: §1 · p. 1
- Exemplar GANs (ExGANs) condition a cGAN on identifying traits available at inference, using a reference image or perceptual code as the exemplar.The paper assumes exemplar information is available when generating the result.Source: §1 · p. 2
- Exemplar information can reduce from-scratch texture or structure hallucination while retaining the original image’s semantics and personalizing the output.For eye in-painting, another instance of the subject’s eyes can serve as the exemplar.Source: §1 · p. 2
- ExGANs allow exemplar information to enter at multiple points, including as perceptual loss or as a hint to the generator or discriminator, and yield photo-realistic, identity-preserving eye in-paintings.The paper applies the framework to eye in-painting.Source: §1 · p. 2
2. Related Work
Prior eye-opening and facial in-painting methods have used exemplar photos, deep networks, or identity-preserving objectives, but they face trade-offs in structural context or personalization. This section situates the paper’s inference-time reference-image approach among those lines of work.
2. Related Work
- Earlier closed-eye opening methods combined burst photographs with patch matching and blending, but did not fully use semantic or structural context such as global illumination or pose.Small lighting variations or incorrect gaze direction can produce uncanny results.Source: §2 · p. 2
- Figure 1 identifies panels (a,b) as exemplar and original images and (c,d) as Photoshop Elements and ExGAN eye-opening results.The caption frames the panels as a comparison between the commercial algorithm and the proposed technique.Source: Figure 1 · p. 2
- Face GANs can manipulate attributes such as beards or glasses, but the described technique does not preserve identity.The cited work allows parts to be changed by manipulating elements of a low-dimensional noise vector.Source: §2 · p. 2
- Exemplar in-painting combines structural reconstruction with texture synthesis, but has struggled with complex or structured in-painting.The paper contrasts this with cGAN in-painting that uses remaining image portions as extra information.Source: §2 · p. 2
- GAN in-painting can produce smooth transitions without pixel blending, but prior face methods cannot personalize results at inference unless the face was in training.The discriminator can use discontinuities between in-painted and original regions to judge results.Source: §2 · pp. 2–3
- Prior identity-preserving face synthesis uses multi-view training images or perceptual-distance objectives; this paper instead assumes a reference image is available at inference.The paper also proposes using the raw reference image, not only a perceptual code, to help preserve identity.Source: §2 · p. 3
3. Exemplar GANs for in-painting
ExGANs use a related reference image or perceptual code as a semantic guide for in-painting, conditioning a GAN objective on that exemplar.
3. Exemplar GANs for in-painting
- ExGANs use a second related source as a semantic guide rather than copying exemplar pixels, while retaining characteristics of the original photograph.Source: §3 · p. 3
3.1. Reference image in-painting
- Reference-based training pairs each input with a same-person image, potentially from another pose; multiple reference images can expand the training set.Source: §3.1 · p. 3
3.2. Code in-painting
- Code-based training derives a lower-dimensional code from the reference image and can optionally compare generated and reference images in perceptual space.Source: §3.2 · pp. 3–4
- Training marks the eyes, guides in-painting with an exemplar, updates generator and discriminator losses, and can optionally add a perceptual-loss update.Source: Figure 2 · p. 4
3.3. Model architecture
- The generator receives the masked RGB image and fill mask, optionally stacked with reference-image channels; global and local discriminator branches assess face consistency and eye detail.Source: §3.3 · p. 4
- For code-based ExGANs, a 256-dimensional eye code joins the generator bottleneck, while a separate autoencoder combines 128-dimensional left- and right-eye codes.Source: §3.3 · p. 4
4. Experiment setup
The experiments use paired identity-specific images, an internally assembled training set, and a separate celebrity benchmark; training uses image pairs and Adam optimization.
4. Experiment setup
- ExGANs require paired images for each object and many unique identities; varied, high-resolution images and non-extreme poses improved generated-eye quality.Source: §4 · pp. 4–5
- The internal training set contains roughly 2 million aligned images of around 200K people, with at least three images per identity and open eyes throughout.Source: §4 · p. 5
- The Celeb-ID1 benchmark contains around 100K images across 17K celebrity identities, and celebrity images were excluded from training.Source: §4 · p. 5
- Training used different images of each identity as input and exemplar, generally stopped after 2M image pairs, and optimized with Adam using L1 reconstruction and binary cross-entropy adversarial losses.Source: §4 · p. 5
5. Results
Exemplar GANs improved reconstruction losses and produced more convincing personalized eye in-paintings than the non-exemplar model. The results also show that pixel-based losses do not fully track perceptual quality, and that occlusions remain a challenge.
5. Results
- Adding eye codes decreased non-exemplar content loss by 8%, while reference images decreased L1 loss by 17%.Source: §5 · p. 5
- Figure 4 compares ground truth, non-exemplar, and exemplar outputs; both ExGAN variants were qualitatively superior, with code-based results most convincing and personalized.Source: §5 · pp. 5–6, Figure 4 · p. 6
- Code-based ExGAN outputs matched original eye shape while accounting for pose and lighting in global and local in-painted images.The examples use a code exemplar in both generator and discriminator, without perceptual loss.Source: §5 · p. 6, Figure 5 · p. 7
5.1. Content loss vs. perceptual loss
- At epoch 18, L1 loss was 0.01386 versus 0.01457 at epoch 10, but the output was blurrier and FID rose from 7.67 to 10.55.The authors stopped training early as regularization because lower content loss did not reliably indicate better perceptual quality.Source: §5.1 · p. 6, Figure 6 · p. 7
- Figure 6 illustrates that content loss and perceptual quality are not one-to-one proxies, motivating early stopping as a regularizer.Source: §5.1 · p. 6, Figure 6 · p. 7
- FID correlated strongly with perceived quality, whereas MS-SSIM and inception score did not; inception features lacked variation in fine-grained eye details.Source: §5.1 · pp. 6–7
- In an internal-dataset A/B test, participants chose the generated image or were unsure which pair was real 54% of the time; occlusions were the most common failure.Failure examples also include incomplete iris-color or eye-shape preservation, especially when one eye is occluded.Source: §5.1 · p. 7, Figure 7 · p. 8
6. Conclusions and Future Work
Exemplar GANs use identifying information from reference images or perceptual codes to support in-painting when a region has identifying features. The paper highlights eye in-painting and outlines future work on exemplar combinations, masks, and other tasks.
6. Conclusions and Future Work
- Eye in-painting demonstrates Exemplar GANs’ perceptual results through identifying information stored in reference images or perceptual codes.The paper presents eye in-painting as a clear example of the framework’s capabilities.Source: §6 · p. 8
- The authors describe Exemplar GANs as a general framework that could extend to other computer-vision tasks and domains.Source: §6 · p. 8
- Future work proposes combining reference-based and code-based exemplars, after keeping the approaches separate to show both were viable and their results differed.Source: §6 · p. 8
- Because in-painting quality was sensitive to mask placement and size, future work proposes non-square masks and higher-weighted iris-color loss.The paper also proposes applying Exemplar GANs to missing regions in uniquely identifiable natural scenes.Source: §6 · p. 8
- Figure 8 pairs reference images in column (a) with input images in (b) and Exemplar GAN in-paintings in (c).Source: Figure 8 · p. 8