Source-linked AI summary
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling
Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, Huchuan Lu
TL;DR
Large holes in real image editing remain difficult because existing inpainting methods can produce artifacts. The paper combines confidence-driven iterative filling, guided high-resolution upsampling, and realistic object-shaped training masks, and reports better visual quality and performance than existing methods.
Problem
Existing image inpainting methods often produce artifacts when filling large holes in real applications.
Method
The method predicts an inpainting result and confidence map for iterative revision, guided upsampling from high-resolution feature patches, and realistic object-mask training data.
Results
The method outperforms existing methods and achieves better visual quality in experiments.
Takeaways & Limitations
The approach supports high-resolution large-object-removal inpainting by combining iterative correction, high-resolution reconstruction, and realistic training samples.
Takeaways & Limitations
The confidence-prediction loss analysis assumes the overall loss decomposes into nonnegative local pixel-wise losses and ground-truth pixels have zero loss.
Abstract
from arXiv · showhide
Existing image inpainting methods often produce artifacts when dealing with large holes in real applications. To address this challenge, we propose an iterative inpainting method with a feedback mechanism. Specifically, we introduce a deep generative model which not only outputs an inpainting result but also a corresponding confidence map. Using this map as feedback, it progressively fills the hole by trusting only high-confidence pixels inside the hole at each iteration and focuses on the remaining pixels in the next iteration. As it reuses partial predictions from the previous iterations as known pixels, this process gradually improves the result. In addition, we propose a guided upsampling network to enable generation of high-resolution inpainting results. We achieve this by extending the Contextual Attention module to borrow high-resolution feature patches in the input image. Furthermore, to mimic real object removal scenarios, we collect a large object mask dataset and synthesize more realistic training data that better simulates user inputs. Experiments show that our method significantly outperforms existing methods in both quantitative and qualitative evaluations. More results and Web APP are available at https://zengxianyu.github.io/iic.
1 Introduction
The paper targets large-hole inpainting for real image editing by iteratively revising low-confidence regions, generating high-resolution details, and training on realistic object-shaped masks.
- Image inpainting reconstructs missing regions for applications including object removal, restoration, manipulation, re-targeting, compositing, and image-based rendering.
- Existing deep models improve complex-hole filling but often produce artifacts because reconstruction becomes ambiguous when holes are large.
- The proposed model predicts an inpainting result and confidence map, then iteratively reuses high-confidence pixels as known content while revising remaining regions.
- A guided upsampling network extends Contextual Attention to borrow high-resolution feature patches, separating global structure generation from fine-grained texture reconstruction.
- The training procedure synthesizes realistic object-shaped holes, including holes overlapping foreground objects and holes appearing only in backgrounds.
- The paper’s contributions comprise iterative confidence feedback, guided high-resolution upsampling, and realistic training-data synthesis for object removal.
2 Related work
Prior inpainting methods use patches, deep generative models, and progressive filling, while this paper distinguishes its approach through learned confidence feedback that automatically selects regions for revision.
- Patch-based methods borrow similar content from valid regions but often fail on large holes because they lack high-level structural understanding and novel-content generation.
- Deep learning methods model semantic structure and commonly use adversarial training to produce sharper, locally and globally consistent results.
- The upsampling approach uses coarse-scale results to guide fine details while filling those details from high-resolution input features.
- Related progressive methods fill holes through predefined sequences, such as shrinking central masks, residual-block updates, or boundary-to-interior video filling.
- Unlike those approaches, this method jointly predicts confidence and uses previous-iteration feedback to detect poorly filled regions for subsequent revision.
3 Approach
The approach combines an iterative inpainting model with confidence feedback and a guided upsampling network. It also uses object-shaped training holes and confidence-aware generation to improve realistic large-hole and high-resolution inpainting.
- Data generation: The training data uses realistic object-shaped holes, combining 82,020 object masks with random strokes to better mimic image-editing scenarios.The data includes masks from multiple segmentation datasets and mixes object masks with random strokes to avoid a bias toward object-shaped holes.
- Inpainting model: The generative model uses coarse-to-fine completion, producing both an inpainted image and a confidence map with a PatchGAN discriminator for adversarial training.The fine network contains an image decoder and confidence decoder, while the discriminator classifies local image patches as real or fake.
- Confidence prediction loss: The confidence loss encourages maps to select low-loss generated regions while penalizing an all-zero confidence solution.Its hyperparameter λ controls the size of the confident area and is set to 0.1 in evaluations.
- Iterative inpainting: Confidence feedback progressively updates the hole by retaining high-confidence predictions and treating the remaining regions as missing for later iterations.The method initializes the first completion, then updates pixels when their confidence improves over the previous iteration.
- Guided upsampling: Guided upsampling converts a low-resolution result into a high-resolution image by matching and borrowing valid high-resolution feature patches from surrounding input regions.The network extends contextual attention and uses shallow similarity and reconstruction networks to enhance high-resolution details.
- Guided upsampling: The guided upsampling network supports user control by allowing valid source regions to be excluded or specified for filling the hole.Because hole content is constructed from valid patches, changing the valid-region set changes the available sources for reconstruction.
4 Experiments
Experiments evaluate two variants of the method against existing approaches on object-shaped holes, standard validation data, high-resolution inputs, and user preferences. Results examine confidence feedback, iteration count, hole size, realistic training data, and guided upsampling.
- Experimental setup: The evaluation compares Ours* without guided upsampling and Ours with downsampled inpainting followed by guided upsampling.The two variants are evaluated quantitatively and visually alongside state-of-the-art methods.
- Experimental setup: The test set contains 1,000 images with object-shaped holes spanning image sizes from short side 256 to long side 1024.Half exclude salient objects from holes, while the other half place holes randomly.
- Comparisons: Visual comparisons cover synthetic samples and real object-removal tasks, while quantitative comparisons include Global&Local, PatchMatch, GConv, and EdgeConnect.The user study uses 25 real object-removal cases shown to 11 users.
- Ablation and analysis: High-confidence regions produce significantly better predictions than low-confidence regions, supporting the confidence feedback mechanism.The comparison evaluates regions above and at or below a confidence threshold of 0.5.
- Ablation and analysis: More test iterations generally improve scores, especially during the first four iterations, so testing uses four iterations.Performance is not very sensitive to λ in a small tested range; for λ ∈[0.7, 0.13], PSNR is in [28.1, 28.5].
- High-resolution results: Guided upsampling produces more visually pleasant 1024 × 1024 outputs and is preferred more frequently by users.Because it reconstructs details from existing high-resolution patches, Ours has lower quantitative scores than Ours*.
5 Conclusion
The paper presents a high-resolution inpainting method for large object removal that combines confidence-guided iteration, guided upsampling, and realistic object-mask training data. Experiments report better performance and visual quality than existing methods.
- Method: The method predicts an inpainting result and confidence map, then revises unsatisfactory regions iteratively.It fills high-confidence regions first and treats remaining regions as new holes.
- Method: Guided upsampling reconstructs low-resolution results using high-resolution neural patches to improve visual quality for high-resolution inputs.The method first obtains a low-resolution result and then reconstructs it with high-resolution features.
- Training data: The training procedure uses a large object-mask dataset and realistic synthetic samples that simulate user inputs.The data targets realistic object-removal scenarios rather than only square or irregular holes.
- Results: Experiments show that the method outperforms existing methods and achieves better visual quality.The conclusion reports both quantitative and visual improvements at the paper level.