Source-linked AI summary
Contextual-based Image Inpainting: Infer, Match, and Translate
Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang, Hao Li, C. -C. Jay Kuo
TL;DR
High-resolution image inpainting must fill missing regions with plausible content despite the difficulty of learning high-dimensional image distributions. The paper separates coarse inference from feature-to-image translation and uses boundary patch-swap texture propagation. It reports sharper, visually coherent results than prior methods in several settings, with feed-forward inference and 512x512 capability.
Problem
High-resolution inpainting requires filling missing regions with semantically and visually plausible content, while direct high-dimensional image modeling can introduce blur and artifacts.
Method
The method trains separate Image2Feature and Feature2Image networks and uses patch-swap to propagate similar boundary neural patches into the hole.
Results
The authors report better visual quality than previous state-of-the-art methods, including sharper results than GL inpainting, 512x512 scaling, and faster inference than iterative optimization.
Takeaways & Limitations
Breaking inpainting into lower-dimensional stages makes high-resolution completion easier to train while preserving sharp contents and textures in a feed-forward system.
Takeaways & Limitations
The method can fail on overly complicated structures or when a major object portion is missing and Image2Feature cannot provide a good inference.
Abstract
from arXiv · showhide
We study the task of image inpainting, which is to fill in the missing region of an incomplete image with plausible contents. To this end, we propose a learning-based approach to generate visually coherent completion given a high-resolution image with missing components. In order to overcome the difficulty to directly learn the distribution of high-dimensional image data, we divide the task into inference and translation as two separate steps and model each step with a deep neural network. We also use simple heuristics to guide the propagation of local textures from the boundary to the hole. We show that, by using such techniques, inpainting reduces to the problem of learning two image-feature translation functions in much smaller space and hence easier to train. We evaluate our method on several public datasets and show that we generate results of better visual quality than previous state-of-the-art methods.
1 Introduction
The paper frames high-resolution image inpainting as filling missing regions with semantically and visually plausible content, then addresses the challenge through staged inference, feature translation, and boundary texture propagation.
- High-resolution image generation remains difficult because directly modeling pixel distributions can produce blur and artifacts as dimensionality increases.
- Image inpainting fills missing image regions with semantically and visually plausible content for object removal and photograph restoration.
- The method decouples inpainting into Image2Feature inference and Feature2Image translation, using coarse structure first and sharp texture refinement second.The two networks can be trained end-to-end, and inference runs in a single forward pass.
- Patch-swap propagates high-frequency details by replacing hole patches with similar neural patches from the boundary before image reconstruction.Reusing boundary neural patches supplies sufficient detail for high-resolution reconstruction.
- The staged approach reduces mapping dimensionality and, compared with GL inpainting, produces sharper results at 256x256 while scaling to 512x512.The authors also report comparable or better visual quality than neural inpainting, more seamless boundary blending, and faster execution.
- The system is designed to synthesize missing parts in high-resolution images with high-quality contents and textures, with reported generalization to style transfer.
2 Related Work
Related work spans GAN-based image generation, neural style-transfer inpainting, the paper’s staged architecture, and traditional patch-based methods, each addressing different reconstruction or efficiency challenges.
- GAN research has produced sharp images, but training instability makes scaling generation to higher resolutions difficult.Methods such as DCGAN, WGAN, WGAN-GP, BEGAN, LSGAN, and Progressive GANs were proposed to stabilize training.
- Neural style-transfer methods match neural patches to propagate texture, but optimization-based approaches can be computationally costly for large images.
- The proposed architecture combines Image2Feature coarse inference, VGG feature extraction, patch-swap boundary matching, and Feature2Image reconstruction.
- Traditional non-neural inpainting uses image-space patch matching and propagation but is generally agnostic to high-level semantic and structural information.
3 Methodology
The method decomposes inpainting into coarse inference, neural-patch matching, and feature-to-image translation. It uses perceptual and adversarial training, multi-scale processing, and boundary texture reuse to produce complete images from incomplete inputs.
- Inference: The pipeline first uses Image2Feature to fill the hole coarsely and extract a 64x64x256 feature map, then embeds the prediction for subsequent processing.G1 uses down-sampling, residual blocks, up-sampling, VGG19 features, and an adversarial discriminator.
- Training: The networks use perceptual and adversarial losses, with perceptual loss restricted to the hole and weighted more heavily at hole-boundary overlaps.The VGG19 weights are fixed during training, and the Image2Feature discriminator receives incomplete-complete image pairs.
- Matching: Patch-swap replaces each neural patch inside the hole with its most similar boundary patch, then averages overlapping replacements into a new feature map.Similarity is computed using cross-correlation, and parallel convolution accelerates matching across the boundary patches.
- Translation: Feature2Image maps the swapped 64x64x256 feature map to a complete sharp 256x256x3 image using a U-Net-style generator and Patch-GAN discriminator.The generator uses skip connections, while the discriminator receives an image-feature-map pair.
- Training: Feature2Image training uses ground-truth-derived swapped features because coarse-prediction features contain noise and ambiguity, while the trained model generalizes to noisy test inputs.At test time, ground truth is unavailable, so inference uses swapped features derived from Image2Feature output despite the train-test input mismatch.
4 Experiments
Experiments compare the approach with prior inpainting methods, analyze design choices, and demonstrate applications to arbitrary holes, style transfer, and difficult cases. The method achieves strong perceptual quality and broader resolution and shape coverage, while remaining vulnerable to highly complex structures or poor initial inference.
- 4 Experiments: The evaluation uses COCO and ImageNet, comparing against CAF, CE, NPS, and GLI on fixed-hole and random-hole settings.The datasets contain 118,287 COCO images and 1,281,167 ImageNet images; Table 1 reports results on 200 ImageNet test images.
- 4.2 Results: The method achieves the best SSIM and inception score among compared methods, despite a higher mean ℓ1 error.The authors argue that mean ℓ1 error favors averaged colors and blurry results, whereas SSIM and inception score better reflect perceptual quality.
- 4.2 Results: User rankings favor the method in 75.9% of rankings, while comparisons report fewer artifacts, stronger coherence, and faster execution than competing approaches.The user study collected 399 valid votes from 20 users evaluating 400 COCO images against NPS and GLI.
- 4.3 Analysis: Resize-convolution reduces checkerboard patterns, perceptual loss produces sharper results than ℓ2 loss, and ground-truth or mixed training inputs outperform prediction-only training.Prediction-only training yields very noisy results; fine-tuning with ground-truth and prediction mixtures produces satisfying inpainting. VGG19 relu3_1 performs better than relu2_1 and relu4_1.
- 4.3 Analysis: Iterative Feature2Image application sharpens textures but can aggregate artifacts near boundaries, while complex structures or major missing object regions can cause failure.Such failures occur when Image2Feature cannot provide a good inference.
- 4.3 Analysis: The fully convolutional framework handles arbitrary image sizes and hole shapes, including larger holes, and the Feature2Image network also supports real-time arbitrary style transfer.Style-transfer examples use a swapped feature map constructed from content and style images; the network was trained only on COCO.
5 Conclusion
The paper presents a learning-based approach for synthesizing realistic, sharp missing contents in high-resolution images. It uses feed-forward, multi-stage mappings and reports visual comparisons on ImageNet.
- The approach synthesizes realistic and sharp missing contents in high-resolution images.
- ImageNet experiments compare the input, GLI, and the proposed result on 256 × 256 images with random holes.
- Breaking inpainting into multiple stages reduces each mapping’s dimensionality and simplifies training while retaining feed-forward completion.