Source-linked AI summary
Generative Face Completion
Yijun Li, Sifei Liu, Jimei Yang, Ming-Hsuan Yang
TL;DR
Face completion requires generating semantically meaningful pixels for unique facial regions that patch-based methods cannot reliably recover. The paper uses an autoencoder-based generative model with reconstruction, local and global adversarial, and semantic parsing losses, producing realistic completions across varied masks while remaining limited on some unaligned faces and pixel-level color details.
Problem
Patch-based completion cannot reliably fill unique facial parts because nearby low- and midlevel patterns may not provide semantically valid content.
Method
An autoencoder-based generator synthesizes missing regions, regularized by reconstruction, local and global adversarial, and semantic parsing losses.
Results
The model generates semantically valid and visually plausible face completions for large missing regions with varied positions, sizes, shapes, and occlusions.
Takeaways & Limitations
Semantic parsing helps produce completion results that are more plausible and consistent with surrounding image context.
Takeaways & Limitations
The model remains unreliable on some unaligned faces and does not fully exploit adjacent-pixel correlations, including incorrect lip color recovery.
Abstract
from arXiv · showhide
In this paper, we propose an effective face completion algorithm using a deep generative model. Different from well-studied background completion, the face completion task is more challenging as it often requires to generate semantically new pixels for the missing key components (e.g., eyes and mouths) that contain large appearance variations. Unlike existing nonparametric algorithms that search for patches to synthesize, our algorithm directly generates contents for missing regions based on a neural network. The model is trained with a combination of a reconstruction loss, two adversarial losses and a semantic parsing loss, which ensures pixel faithfulness and local-global contents consistency. With extensive experimental results, we demonstrate qualitatively and quantitatively that our model is able to deal with a large area of missing pixels in arbitrary shapes and generate realistic face completion results.
1. Introduction
Face completion must generate semantically valid content for unique facial regions that cannot be recovered by copying nearby patches. The paper proposes a deep generative model combining reconstruction, adversarial, and semantic parsing objectives.
- Problem: Existing patch-based completion methods are constrained when missing object parts contain unique patterns unavailable elsewhere in the input image.Low- and midlevel visual cues are insufficient to infer semantically valid content for such regions.
- Task: Figure 1 contrasts original faces, masked inputs, and the method’s completions for square masking and microphone-occluded mouths.The images are 128 × 128 pixels.
- Method: The proposed model masks inputs with noise, encodes them, and decodes filled images through an autoencoder-based generator.The model generates missing content from learned hidden representations rather than retrieving patches.
- Method: Local and global adversarial losses regularize missing-region coherence and whole-image realism, while semantic parsing enforces more reasonable generation.The local loss targets the missing region; the global loss targets the entire image.
- Contribution: The model synthesizes semantically valid patterns for face completion and uses semantic parsing to improve plausibility and consistency with surrounding context.These are stated as the paper’s main contributions.
2. Related Work
Prior completion methods propagate or match low-level image patterns, but they struggle with large or semantically unique missing regions. This work combines generative, adversarial, reconstruction, and semantic parsing approaches for face completion.
- Image completion: Traditional inpainting propagates low-level features, while patch matching fails when the source image lacks sufficient data for the unknown region.Object parts are often unique, so plausible source patches may not exist.
- Image completion: Sparse-signal recovery methods require highly structured, well-aligned images, whereas the proposed algorithm is designed to avoid strict alignment constraints.The cited comparison specifically uses well-aligned face images as an example.
- Image generation: Denoising autoencoders and decoder-based reconstruction learn to recover clean or object images from corrupted inputs.These methods are presented as related image-generation approaches.
- Image generation: Variational autoencoders enable latent-space sampling but commonly produce blurry images because of pixel-wise Gaussian-likelihood training.Adversarial training is described as an improvement to this limitation.
- Positioning: The paper differs from prior representation-learning work by targeting severe corruption with combined reconstruction, adversarial, and parsing losses.It also uses semantic labels or segmentations to regularize face completion.
3. Proposed Algorithm
The proposed completion model uses an autoencoder generator with local and global discriminators plus semantic parsing to synthesize realistic, context-consistent missing contents. A reconstruction-to-adversarial curriculum and parsing regularization progressively improve completion quality, while Poisson blending can reduce boundary color differences.
- Network architecture: The model combines an autoencoder generator, two discriminators, and a semantic parsing network to synthesize missing contents consistent with the whole object.The generator encodes masked inputs and decodes filled images; the discriminators assess local and global realism, while parsing encourages harmony with existing pixels.
- Generator: The encoder captures relationships between known and unknown regions before the decoder generates the completed image.Unlike a noise-only GAN, the generator conditions its hidden representation on the corrupted input and surrounding context.
- Adversarial losses: Local adversarial training sharpens missing-region details and promotes semantically valid parts, whereas global adversarial training improves whole-image realism and contextual consistency.The local discriminator supplies signals within the mask; the global discriminator evaluates the entire image and addresses inconsistencies across mask boundaries.
- Semantic regularization: Semantic parsing regularization makes generated facial features more harmonious with existing pixels by encouraging natural feature shapes and sizes.The parsing result of the generated image is compared with that of the original image, and the parsing loss uses pixel-wise softmax.
- Objective and training: Training proceeds in three stages: reconstruction loss, local adversarial fine-tuning, then global adversarial loss and semantic regularization.Reconstruction-only outputs are blurry, while the full objective produces the most realistic and plausible completion; Poisson blending can further reduce subtle boundary color differences.
4. Experimental Results
Experiments on CelebA and Helen evaluate semantic parsing, realistic completion, robustness to mask variation, identity preservation, and model limitations. The model produces plausible completions across challenging masks, generally outperforms ContextEncoder on large masks, but remains sensitive to alignment and some pixel-level color details.
- Experimental setup: CelebA experiments use 19,962 test images, with 64 × 64 training masks designed to remove at least one essential facial component.The dataset contains 202,599 images, split into training, validation, and testing subsets.
- Semantic parsing: The model uses Helen’s 11-part face parsing labels to regularize CelebA completion through a fixed parsing network and generator back-propagation.Helen provides segment labels because CelebA lacks them; parsing predictions on generated faces are compared with original-face parsing results.
- Qualitative results: The method generates visually realistic completions for side-view faces and masks with different shapes and sizes, including masks covering multiple key components.Testing permits masks beyond the training square shape, although the total masked area is suggested to remain no more than 64 × 64 pixels.
- Quantitative results: The proposed model generally performs better than ContextEncoder, especially for large masks such as O1-O3 and O6.Poisson blending further improves performance in the reported comparison.
- Quantitative results: Reconstruction loss alone yields relatively higher PSNR and SSIM but blurry results, while the final model’s face-recognition performance is closest to the original-image upper bound.A substantial recognition gap remains for large masks, and the authors identify identity-preserving completion as future work.
- Latent-space variation: Different input noise produces multiple semantically consistent appearances, while surrounding unmasked context constrains the variation of generated facial parts.For example, a generated eyebrow remains similar in shape, size, and position to the visible eyebrow.
5. Conclusion
The proposed deep generative network synthesizes semantically valid, visually plausible facial content and handles varied maskings or occlusions. The work is supported by qualitative and quantitative experiments.
- The model uses an autoencoder generator, local and global adversarial losses, and semantic regularization to complete missing facial key parts.
- The generated contents are semantically valid and visually plausible across different positions, sizes, and shapes of masks or occlusions.