Source-linked AI summary
Foreground-aware Image Inpainting
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, Jiebo Luo
TL;DR
Image inpainting remains difficult when missing regions overlap or touch foreground objects, where existing methods lack information about foreground and background extent. The paper proposes contour-first foreground-aware completion, using predicted contours to guide image inpainting, and reports significantly better results than existing methods. Artifacts can nevertheless still occur in some cases.
Problem
Inpainting methods struggle with holes overlapping or touching foreground objects because the missing foreground and background extents are unknown.
Method
The system detects and completes foreground contours, then uses the completed contours with the corrupted image to guide image completion.
Results
The model produces reasonable object contours and significantly outperforms various state-of-the-art models quantitatively and qualitatively.
Takeaways & Limitations
Explicit foreground structure is a promising direction for guiding image completion in challenging compositions.
Takeaways & Limitations
The model can still produce artifacts while completing reasonable object shapes.
Abstract
from arXiv · showhide
Existing image inpainting methods typically fill holes by borrowing information from surrounding pixels. They often produce unsatisfactory results when the holes overlap with or touch foreground objects due to lack of information about the actual extent of foreground and background regions within the holes. These scenarios, however, are very important in practice, especially for applications such as the removal of distracting objects. To address the problem, we propose a foreground-aware image inpainting system that explicitly disentangles structure inference and content completion. Specifically, our model learns to predict the foreground contour first, and then inpaints the missing region using the predicted contour as guidance. We show that by such disentanglement, the contour completion model predicts reasonable contours of objects, and further substantially improves the performance of image inpainting. Experiments show that our method significantly outperforms existing methods and achieves superior inpainting results on challenging cases with complex compositions.
1. Introduction
Image inpainting is especially difficult when holes overlap or approach foreground objects, because existing methods lack sufficient structural information. The proposed system separates contour inference from image completion and uses completed contours to guide cleaner inpainting.
- Motivation: Holes overlapping or near foreground objects are challenging because existing methods rely on patches or implicitly learned structure.Conventional methods use low-level patch matching, while learning-based methods assume image structure can be inferred implicitly.
- Proposed Approach: The model detects foreground contours, completes missing contour regions, and feeds the completed contour with the input image into the image-completion module.This cascade uses predicted foreground structure as guidance for content prediction in holes.
- Results: The contour module infers reasonable missing structure, while contour-guided completion produces cleaner content around object borders.Figure 1 compares the guided result with PartialConv and GatedConv outputs.
- Proposed Approach: The system explicitly disentangles structure inference and image completion for challenging foreground-aware inpainting.Its contributions include explicit contour guidance and curriculum training for the contour and image-completion modules.
- Results: The experiments report higher-quality inpainting results than existing methods.The contribution summary describes improvements on challenging cases where holes overlap with or touch foreground objects.
2. Related Work
Image inpainting research includes traditional propagation and patch-matching methods alongside deep neural network approaches. Learning-based methods improve semantic understanding, but earlier approaches retain limitations in guidance and post-processing.
- Traditional Methods: Traditional methods propagate neighborhood appearance or retrieve similar patches to fill missing regions.Propagation works best for small or narrow holes, while patch-based methods can be computationally expensive and memory-intensive.
- Traditional Methods: PatchMatch accelerates patch-based search but searches the whole image without high-level information to guide retrieval.This limits its ability to exploit semantic or structural context during inpainting.
- Learning-Based Methods: Learning-based methods map corrupted images to completed outputs while learning image semantics from large-scale datasets.Their semantic capability is useful for complex scenes, faces, and objects.
- Learning-Based Methods: Context Encoders use convolutional networks with pixel-space L2 loss and adversarial loss to fill holes and sharpen results.Other approaches use separate global and local discriminators to enforce plausible whole-image and hole appearance.
3. Approach
The approach cascades contour detection, contour completion, and image completion to infer missing foreground structure and guide inpainting. It combines contour-aware adversarial training with curriculum learning to obtain sharper, more stable contour predictions.
- Framework: The system detects incomplete foreground contours, completes missing contour regions, and feeds the completed contour with the incomplete image into image completion.The framework is a three-module cascade: incomplete contour detection, contour completion, and image completion.
- Contour Completion: The contour completion module predicts a binary foreground boundary map, using coarse prediction followed by refinement with contextual attention.The refined contour values represent the probability that each pixel lies on the actual contour.
- Contour Completion: A PatchGAN contour discriminator is conditioned on the image and contour pair because sparse contour masks alone make adversarial training difficult to optimize.The additional image condition requires generated contours to align with image contours and stabilizes training.
- Loss Functions: Focal loss addresses sparse contour-mask imbalance, while spectral-normalization GAN training encourages sharper contours in corrupted regions.The method interprets contour pixels probabilistically rather than relying only on raw-pixel L1 or L2 distances.
- Curriculum Training: Curriculum training first learns rough contours with content loss, then adds adversarial loss at weight ratio 0.01:1, and finally uses equal loss weights.This staged procedure avoids training failure observed when content and adversarial losses are applied simultaneously.
3.4. Image Completion Module
The image completion module uses the incomplete image, completed contour, and hole mask to generate the final image. It is pretrained without contour guidance, then fine-tuned with the additional contour input.
- Inputs and Architecture: The image completion module takes incomplete image Iin, completed contour, and hole mask H as inputs and outputs completed image Ic.Its generator contains coarse and refinement networks.
- Adversarial Training: The discriminator receives the generated or ground-truth image together with the hole mask and uses a PatchGAN hinge adversarial loss.The adversarial loss is applied only to the refinement-network output, not the coarse output.
- Content Loss: The image content loss uses L1 distance between generated and ground-truth images, with pixel locations indexed by [p] and image pixel count denoted by N.The coarse and refinement outputs and the ground truth are identified in the accompanying formulation.
- Training: The module is pretrained on Places2 without a contour channel and then fine-tuned on a saliency dataset using contour guidance.Fine-tuning accommodates the additional contour input while retaining pretrained parameters in the network.
- Training: Jointly fine-tuning contour and image completion modules produces only minor differences from fixing the contour module, so the method uses joint fine-tuning.The comparison concerns the two stated training variations.
4. Experiments
Experiments compare the proposed foreground-aware model with state-of-the-art methods using quantitative metrics, qualitative examples, and user preferences. Results consistently favor contour guidance, especially for producing natural object boundaries and reducing artifacts.
- Quantitative Evaluation: The evaluation uses a saliency dataset with overlap and non-overlap holes, comparing the full model against several state-of-the-art methods and a no-guidance baseline.The quantitative evaluation uses 500 test images with both hole types; metrics include L1, L2, PSNR, and SSIM.
- Quantitative Evaluation: Our models outperform all other deep learning-based methods in L1, L2, PSNR, and SSIM, while contour guidance consistently improves over the no-guidance baseline.The metrics are computed between completed images and ground-truth images in pixel space.
- Qualitative Evaluation: Qualitatively, the full contour-guided model produces fewer border artifacts and more natural object boundaries than PatchMatch, Global&Local, ContextAttention, PartialConv, GatedConv, and Ours No Guide.The comparison includes samples with overlapped holes in Rows 1–4 and non-overlapped holes in Rows 5–8.
- User Study: The user study collected 1,099 valid votes from 22 users across 50 corrupted images, and the full model was preferred by the largest margin.Users selected a single best result for each image.
- Ablation Study: Ablation results show that contour guidance correctly preserves object boundaries in both overlapping and non-overlapping hole cases, whereas removing it leaves obvious artifacts.Figure 4 compares the input, the model without contour guidance, the full model, and the ground truth.
5. Conclusion
The paper proposes foreground-aware inpainting for cases requiring prediction of both foreground and background pixels. It detects and completes foreground contours before using them to guide image completion, and reports quantitative and qualitative advantages over state-of-the-art methods.
- 5. Conclusion: The foreground-aware model targets challenging inpainting scenarios involving prediction of both foreground and background pixels.The model is trained on a specifically collected saliency image dataset.
- 5. Conclusion: The model detects and completes foreground-object contours, then uses the completed contours as guidance for image inpainting.The conclusion describes explicit structure-based guidance for completion.
- 5. Conclusion: Experiments show that the model generates natural object contours and significantly outperforms various state-of-the-art models quantitatively and qualitatively.The paper presents explicit foreground and background structure as a promising direction for inpainting.
1. Network Architecture
The system uses three main stages: foreground contour detection, contour completion, and image completion. Its contour completion module combines coarse and refinement networks, while image completion uses completed contours as guidance.
- Network Architecture: The contour detection network combines high-level, low-level, and boundary-refinement processing.High-level features provide compact semantic representations, low-level features preserve input-resolution detail, and the refinement module fuses them.
- Network Architecture: Contour completion uses a coarse encoder-decoder followed by a refinement network that receives the coarse contour and mask.The coarse network predicts an initial completed contour, which is concatenated with the mask before refinement.
- Network Architecture: The contour completion architecture is specified through convolutional branches, dilated layers, resizing operations, concatenation, and sigmoid activation.The listed branches and decoder define the coarse and refinement network configurations.
- Network Architecture: The image completion module mirrors contour completion but takes the completed contour as guidance and uses tanh rather than sigmoid activation.Its refinement network receives the coarse completed image, completed contour, and mask.
2. Comparison with State-of-the-arts
The method is evaluated against state-of-the-art inpainting systems using additional perceptual and quantitative comparisons. The reported qualitative results show consistent superiority over existing models.
- Comparison with State-of-the-arts: The model consistently outperforms state-of-the-art models in the reported qualitative comparisons.The section presents additional qualitative results corresponding to Figure 1.
- Comparison with State-of-the-arts: Additional quantitative comparisons use Perceptual Similarity LPIPS on VGG or AlexNet feature spaces.These results are reported in Table 1.
- Comparison with State-of-the-arts: The additional quantitative table reports metrics where smaller values are better.The supplied table caption does not specify the individual metric values.
3. Contour Completion Results
The contour completion module produces clean, sharp, and reasonable object contours that benefit subsequent image completion.
- Contour Completion Results: The contour completion module infers clean, sharp, and reasonable contours.These contour predictions are presented as beneficial for completing the image.
4. Failure Cases
The method can complete reasonable contours in cases where existing models fail, but artifacts may still occur in the completed images. Reducing these artifacts remains future work.
- Failure Cases: The model completes a reasonable contour for incomplete objects in cases that existing models fail to inpaint.The comparison includes failure cases of existing models and the corresponding completed-contour output.
- Failure Cases: Artifacts can still occur even when the model predicts a reasonable object shape.The authors identify artifact reduction as future work.
- Failure Cases: Figure 3 presents the sequence from the image with a hole through contour-related intermediate results to the completed image.The displayed sequence includes the saliency map, incomplete contour, completed contour, and completed image.