Source-linked AI summary
Semantic Image Inpainting with Deep Generative Models
Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark Hasegawa-Johnson, Minh N. Do
TL;DR
Semantic image inpainting must infer detailed content in large missing regions, where single-image methods lack high-level context and Context Encoder struggles with arbitrary hole shapes. The paper searches a trained generative model’s latent manifold using context and prior losses, then generates the missing content; across three datasets, it reports more realistic results than state-of-the-art methods. Its performance depends strongly on the generative model and training procedure, with the presented GAN unable to represent complex scenes adequately.
Problem
Semantic inpainting seeks to infer arbitrary large missing regions from surrounding image semantics, while existing single-image methods and Context Encoder have difficulty providing realistic content for complex or arbitrarily structured holes.
Method
The method searches for a latent encoding closest to the corrupted image under weighted context and prior losses, then reconstructs the image through a trained generative model.
Results
The method produces more realistic semantic inpainting results than state-of-the-art techniques across CelebA, SVHN, and Stanford Cars.
Takeaways & Limitations
The approach can infer meaningful content for large corrupted regions with arbitrarily structured missing areas without requiring masks during training.
Takeaways & Limitations
Performance strongly relies on the generative model and training procedure, and the paper’s GAN is too small to represent complex real-world scenes.
Abstract
from arXiv · showhide
Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.
1. Introduction
Semantic inpainting infers arbitrary large missing regions from image semantics, a harder problem than classical inpainting because it requires high-level context. The paper proposes latent-space image generation conditioned by visible pixels and reports more realistic results across challenging settings.
- Semantic inpainting infers arbitrary large missing regions from image semantics, unlike classical inpainting focused on small corruption or object removal.Large missing regions and complex scenes make the task increasingly difficult.
- Most single-image methods rely on local or non-local image information and hand-crafted priors such as smoothness, texture similarity, patch offsets, planarity, or low rank.These methods exploit information available in the input image to address ill-posedness.
- External-data methods can fail when the test scene differs substantially from database or Internet images because they require exact semantic matching.Learning-based methods have subsequently shown promising results.
- Context Encoder uses hole structure during training but not inference, producing blurry or unrealistic images for arbitrarily shaped missing regions.Its performance is strongest when the hole structure is fixed during training and testing.
- The proposed method searches a trained adversarial generator’s latent manifold for an encoding minimizing weighted context and prior losses, without requiring masks during training.It is evaluated on CelebA, SVHN, and Stanford Cars, where it produces more realistic images than state-of-the-art techniques.
2. Related Work
Related work spans generative models, adversarial training, autoencoding, and input optimization. The paper builds on these ideas by updating a latent encoding so a trained generator produces an image consistent with the corrupted input.
- Generative Adversarial Networks train a generator to produce realistic images and a discriminator to distinguish generated images from real samples.The generator maps a random latent vector z to image space, while the discriminator outputs an image likelihood.
- GAN training optimizes a loss involving real samples h from pdata and random latent encodings z.
- VAEs learn complex distributions without supervision but tend to generate overly smooth images compared with GANs.The related work also notes VAE variants for attribute-based image editing.
- Context Encoder can be viewed as an autoencoder conditioned on corrupted images, working best when hole structure is fixed across training and inference.It is less effective for arbitrarily structured regions.
- The proposed framework uses back-propagation to update a latent vector toward a solution on the learned image manifold.The figure describes random initialization z(0), iterative states z(k), and the final solution ˆz.
3. Semantic Inpainting by Constrained Image Generation
The method treats semantic inpainting as constrained generation: it searches latent space for a code whose generated image matches known pixels while remaining realistic. It then reconstructs the missing region and uses blending to reduce visible seams.
- The method optimizes a latent code z so the generator produces an image close to the corrupted input while staying on the learned image manifold.The code is randomly initialized and updated by back-propagation using context and prior losses.
- 3.1. Importance Weighted Context Loss: The context loss compares generated and observed pixels, weighting uncorrupted pixels near holes more heavily than distant pixels.The implementation uses a weighted ℓ1 difference, with importance weights based on corrupted neighbors in a local window.
- 3.2. Prior Loss: The prior loss uses the discriminator to penalize unrealistic generated images and balance realism against contextual fidelity.Without the prior loss, optimization can converge to perceptually implausible results.
- 3.3. Inpainting: After generation, Poisson blending preserves generated gradients while shifting colors to match the observed image.The paper reports visible seams in examples without blending.
- The approach is compatible with different generative architectures; experiments use DCGAN with 64×64×3 outputs and a 100-dimensional latent vector.The experiments set λ = 0.003 and optimize the latent code with Adam while restricting z to [−1, 1].
4. Experiments
Experiments across three datasets and four mask types compare the method with local, retrieval-based, and learning-based baselines. The method generally produces sharper, more realistic results and handles arbitrary masks without retraining, though its performance depends on generative-model capacity.
- Comparisons with local methods: TV and LR produce blurry or noisy results with large missing regions, while PatchMatch cannot operate when too few patches remain.The comparison uses examples with random 80% missing pixels.
- Comparisons with retrieval: Nearest-patch retrieval visibly misaligns skin texture, eyebrows, eyes, and hair, whereas the proposed method requires no registration.The comparison concerns nearest-neighbor filling from the training dataset.
- Comparisons with Context Encoder: On CelebA, Context Encoder performs best for fixed central masks but produces more artifacts and blur for randomly positioned holes; the proposed results are often as realistic as real images.Context Encoder does not use the mask during inference and pre-fills holes with mean color.
- Quantitative and qualitative results: The proposed method generally yields sharper, less artifact-prone SVHN and car results than Context Encoder, while PSNR can favor Context Encoder when the ground truth is only one plausible completion.For random holes, the proposed method outperforms Context Encoder, and PSNR is more meaningful when uncorrupted pixels are distributed throughout the image.
- Limitations: Performance strongly relies on the trained generative model and procedure; the GAN used here handles relatively simple faces but is too small for complex real-world scenes.Failure examples arise when the method cannot find the correct latent-manifold encoding.
5. Conclusion
The paper presents generative-model-based semantic inpainting as an alternative to local priors, patches, and Context Encoder. Across challenging examples, it reports meaningful corrupted-image content and sharper, more realistic edges.
- The proposed method learns representations of training data to predict meaningful content for corrupted images, unlike methods based on local image priors or patches.
- Compared to Context Encoder, the method often produces sharper edges that look more realistic.
- Experimental results demonstrate superior performance on challenging image-inpainting examples.
6. Supplementary Material
The supplementary material provides additional visual results across CelebA, SVHN, and car datasets. These figures extend the paper’s qualitative comparisons across the three evaluated domains.
- Additional results are shown for the CelebA dataset.
- Additional results are shown for the SVHN dataset.
- Additional results are shown for the car dataset.