Source-linked AI summary
High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, Hao Li
TL;DR
General image hole-filling remains challenging because natural images are complex and ambiguous, while existing learning-based methods struggle with larger resolutions. This paper combines global content and local texture constraints through multi-scale neural patch synthesis, producing sharper and more coherent high-resolution inpainting results.
Problem
General image hole-filling remains challenging because natural images are inherently ambiguous and natural scenes are complex.
Method
The approach jointly optimizes global content and local texture constraints using encoder-decoder predictions and neural patches from middle-layer network features.
Results
The method produces more realistic and coherent inpainting that preserves structure and texture details across quantitative and qualitative evaluations on two public datasets.
Takeaways & Limitations
Middle-layer neural features can synthesize realistic image contents and textures, supporting sharper high-resolution inpainting results.
Takeaways & Limitations
The current implementation is significantly slower than Content-Aware Fill, taking roughly one minute to fill a 256 × 256 hole in a 512 × 512 image.
Abstract
from arXiv · showhide
Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal. While these learning-based methods are significantly more effective in capturing high-level features than prior techniques, they can only handle very low-resolution inputs due to memory limitations and difficulty in training. Even for slightly larger images, the inpainted regions would appear blurry and unpleasant boundaries become visible. We propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints, which not only preserves contextual structures but also produces high-frequency details by matching and adapting patches with the most similar mid-layer feature correlations of a deep classification network. We evaluate our method on the ImageNet and Paris Streetview datasets and achieved state-of-the-art inpainting accuracy. We show our approach produces sharper and more coherent results than prior methods, especially for high-resolution images.
1. Introduction
Image hole-filling is challenging because natural-image content is ambiguous, while existing texture-based and data-driven methods trade off high-frequency detail against semantic and global-structure fidelity. The paper proposes joint global-content and local-texture optimization with multiscale neural patch synthesis for realistic, coherent high-resolution inpainting.
- Automated hole-filling supports photo editing tasks such as removing distracting elements, repositioning objects, and recovering occluded content, but remains challenging because natural-image completion is inherently ambiguous.
- Texture-synthesis methods propagate high-frequency details from surrounding patches but fail to capture image semantics or global structure, whereas data-driven methods depend on finding examples with sufficiently similar context.
- Middle-layer neural features are used to synthesize realistic image content and textures, extending their prior use in artistic style transfer.
- The proposed hybrid framework combines an encoder-decoder CNN’s structured prediction with neural patches, using global content as one constraint and local patch similarity as the texture constraint.
- The constraints are jointly optimized by backpropagation with limited-memory BFGS, while a three-level, step-size-two image pyramid enables high-resolution inpainting with a 256 × 256 central hole in a 512×512 image.
- Experiments on two public datasets show that the multiscale approach produces more realistic and coherent results while preserving structure and texture details across comparisons with baselines and existing techniques.
2. Related Work
Prior work established deep networks for structure prediction, segmentation, and image generation, motivating their use in image inpainting. The approach adapts Context Encoder for global content prediction and draws on neural style transfer to synthesize realistic textures through multi-scale neural patch synthesis.
- Structure Prediction using Deep Networks: Deep neural networks have advanced image classification and been applied to structure prediction, semantic segmentation, and image generation, motivating their use as the hole-filling backbone.The method is motivated by the generative power of deep neural networks.
- Context Encoder: Context Encoder combines ℓ2 and adversarial losses for inpainting, and this work uses it as the global content predictor to initialize synthesis at the smallest scale.The Context Encoder output initializes the multi-scale neural patch synthesis algorithm.
- Style Transfer: Neural style transfer motivates the method’s realistic texture synthesis, while local texture loss compares neural patches from the hole and its boundary.The framework jointly uses holistic content loss and local texture loss, with content conditioned on a pre-trained network at the smallest scale.
3. The Approach
The approach reconstructs missing regions by jointly optimizing holistic content, local texture, and total-variation constraints. A multiscale scheme progressively refines low-resolution content predictions with neural-patch texture matching at finer scales.
- 3. The Approach: The objective combines holistic content, local texture, and TV-loss terms to preserve global semantics, match local texture statistics, and encourage smoothness.The content constraint is initialized using a trained content network.
- 3. The Approach: Local texture matching compares neural patches inside the hole with their most similar patches outside it in an intermediate feature map.For each query patch, the texture loss averages distances to its nearest non-hole neighbor, which can be computed as a convolutional layer.
- 3. The Approach: The multiscale procedure downsizes the image, optimizes content and texture constraints at low resolution, then upsamples the result to initialize joint optimization at finer scales.This scheme is designed to apply naturally to high-resolution images with large holes.
- 3. The Approach: Setting α and β to 5e−6 empirically balanced the loss magnitudes and gave the best experimental results.These weights control the relative importance of the objective terms.
- 3. The Approach: The texture network is ImageNet-pretrained VGG-19, using relu3_1 and relu4_1 together because their combination produced more accurate results than either single layer.Using the content network as the texture network produced lower-quality results than using pre-trained VGG-19.
4. Experiments
Experiments on Paris StreetView and ImageNet evaluate the method visually and quantitatively, including low- and high-resolution comparisons, ablations, and distractor removal. The approach achieves the highest numerical performance on 128 × 128 Paris StreetView images and produces coherent structure with high-frequency details at higher resolution, but remains slower than Content-Aware Fill.
- Datasets and evaluation: The method is evaluated on Paris StreetView and ImageNet using visual and quantitative comparisons, including a real-world distractor-removal application.The experiments compare the approach with other methods and assess high-resolution image inpainting.
- High-resolution comparisons: At 512 × 512 resolution, the multi-scale iterative approach produces coherent global structure and high-frequency details compared with Content-Aware Fill and Context Encoder.Context Encoder outputs are directly upsampled from 128 × 128 to 512 × 512 using bilinear interpolation.
- Quantitative comparisons: On 128 × 128 Paris StreetView images, the method achieves the highest numerical performance among the baseline methods.The paper attributes this to recovering correct image structure and synthesizing better details than the compared methods.
- Ablation study: Removing the content constraint makes the inpainting structure completely incorrect, while increasing content weight makes results more consistent with the initial prediction.The quality of content-network initialization also matters: blurry ℓ2-only predictions lead to blurrier final results than ℓ2 plus adversarial training.
- Limitations: Filling a 256 × 256 hole in a 512 × 512 image takes roughly 1 min on a Titan X GPU, making the method significantly slower than Content-Aware Fill.The implementation generates new textures rather than directly propagating existing patches.
5. Conclusion
The paper advances semantic inpainting through neural patch synthesis by combining texture-driven high-frequency detail generation with content-driven semantic and global-structure priors. It also identifies potential applications beyond inpainting and acknowledges discontinuities and artifacts in some cases.
- Neural patch synthesis advances the state of the art in semantic inpainting.
- The texture network generates high-frequency details, while the content network provides semantic and global-structure priors.
- The approach may also support denoising, superresolution, retargeting, and view/time interpolation.
- Some cases introduce discontinuities and artifacts.