Source-linked AI summary
SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting
Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, C. -C. Jay Kuo
TL;DR
Image inpainting must reconstruct missing regions while preserving realistic context and object boundaries, but many generative methods do not exploit segmentation information. SPG-Net predicts missing-region segmentation with SP-Net, then uses SG-Net for segmentation-guided image generation; experiments report improved inpainting quality and interactive multi-modal outputs.
Problem
Many generative inpainting methods do not exploit segmentation information to constrain object shapes, leading to blurry boundaries between different objects.
Method
SPG-Net first predicts missing-region segmentation labels with SP-Net, then combines the completed mask with the input image for SG-Net to generate the final inpainting.
Results
The framework outperforms existing methods in image inpainting quality across multiple public datasets and produces sharper boundaries and realistic details.
Takeaways & Limitations
Interactive segmentation guidance enables users to manipulate segmentation maps and generate multi-modal inpainting outputs.
Abstract
from arXiv · showhide
In this paper, we focus on image inpainting task, aiming at recovering the missing area of an incomplete image given the context information. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, but existing methods based on generative models don't exploit the segmentation information to constrain the object shapes, which usually lead to blurry results on the boundary. To tackle this problem, we propose to introduce the semantic segmentation information, which disentangles the inter-class difference and intra-class variation for image inpainting. This leads to much clearer recovered boundary between semantically different regions and better texture within semantically consistent segments. Our model factorizes the image inpainting process into segmentation prediction (SP-Net) and segmentation guidance (SG-Net) as two steps, which predict the segmentation labels in the missing area first, and then generate segmentation guided inpainting results. Experiments on multiple public datasets show that our approach outperforms existing methods in optimizing the image inpainting quality, and the interactive segmentation guidance provides possibilities for multi-modal predictions of image inpainting.
1 Introduction
Image inpainting reconstructs missing regions while preserving realistic content, contextual coherence, and sharp boundaries. SPG-Net addresses blurry object boundaries by predicting segmentation labels before generating the final inpainting.
- Image inpainting reconstructs missing regions from surrounding context while seeking realistic details and coherence with the image boundaries.
- Traditional patch-based methods produce realistic textures but rely on low-level context and cannot predict high-level features in missing regions.
- Deep generative models frame inpainting as conditional image generation, enabling end-to-end prediction from incomplete images.
- Many generative approaches omit segmentation information, which the authors associate with blurry boundaries between different objects.
- SPG-Net uses Deeplabv3+ initialization, SP-Net segmentation prediction, and SG-Net to generate the complete image from the segmentation mask and input.
- Experiments and ablations report that segmentation maps provide useful information for texture details and improve image inpainting quality.
2 Related Work
Prior inpainting methods range from texture propagation and patch matching to deep generative models, but they remain limited in semantic structure and stable high-resolution synthesis. SPG-Net treats segmentation labels as latent guidance for sharper boundaries and interactive multi-modal outputs.
- Traditional methods propagate surrounding texture or search matching patches, but large holes can produce artifacts and semantic structure remains unmodeled.
- Deep generative inpainting methods offer realistic images but face challenges in stable GAN training and high-resolution generation.
- Context-encoder and related approaches apply adversarial or feature-space methods to reconstruct missing image regions.
- SPG-Net treats semantic segmentation labels as latent variables that guide image inpainting and produce sharper textures between object boundaries.
- Interactive segmentation guidance allows users to generate multi-modal inpainting results.
3 Approach
The framework models segmentation labels as an intermediate variable, first predicting labels in the missing region and then using them to guide image inpainting. SP-Net and SG-Net use convolutional generators with perceptual and adversarial objectives tailored to segmentation and image reconstruction.
- Framework: The model estimates the observed segmentation map S0, predicts missing-region labels SR with SP-Net, and generates the final image using SG-Net.The complete segmentation map guides the final image prediction.
- SP-Net: SP-Net takes incomplete labels S0 and image I0 as input and predicts a complete segmentation map S with C label channels.Its generator uses four down-sampling layers, nine residual blocks, and four up-sampling layers with progressive dilation.
- SP-Net: The segmentation loss uses feature matching across generator and discriminator layers, with a mask weighting that emphasizes the missing region.Starting at layer 0 also incorporates an input-level reconstruction component.
- Objectives: The full objective combines adversarial and perceptual losses, weighted by λadv and λperceptual, set to 1 and 10 respectively.These weights are reported for the experiment.
- SG-Net: SG-Net takes I0 together with SP-Net’s predicted segmentation map S and predicts the 256x256x3 inpainting result.Its architecture parallels SP-Net, while an additional local-hole AlexNet perceptual loss supports training.
- Objectives: SG-Net uses adversarial, perceptual, and AlexNet-based perceptual terms, with λadv = 1, λperceptual = 10, and λAlex = 10.The AlexNet term measures perceptual differences on local hole patches.
4 Experiments
Experiments on Cityscapes and Helen Face evaluate SPG-Net against established inpainting methods through visual, quantitative, user, ablation, and interactive-editing analyses.
- Experimental Setup: Cityscapes experiments compare SPG-Net with PatchMatch and GL on randomly masked 256×256 images, while Helen Face experiments compare it with GFC.The datasets provide segmentation annotations and use holes ranging from 1/8 to 1/2 of the image size.
- Qualitative Comparisons: On Cityscapes, SP-Net predicts more reasonable layouts and SG-Net recovers object boundaries more clearly than GL in scenes with multiple object boundaries.The method can infer part of a car and complete its wheel from a small segmentation shape in the input.
- Qualitative Comparisons: On Helen Face, the proposed method generates more realistic face inpainting results than GFC, despite GFC being specifically designed and trained for face completion.GFC outputs were up-sampled from 128×128 to 256×256 for comparison.
- Quantitative Comparisons: SPG-Net outperforms PatchMatch and GL on three of four image-quality metrics, while GL has slightly smaller ℓ2 errors.The paper notes that ℓ2 is less capable of assessing perceptual quality than SSIM and PSNR because it tends to reward blurry outputs.
- User Study: Users ranked SPG-Net highest in 70.8% of 600 Cityscapes comparisons, including 96.2% superiority over PatchMatch and 71.3% perceptual superiority over GL.Against GL, the results were ranked the same 16.3% of the time.
- Ablation Study and Interactive Editing: The ablation shows that SP-Net segmentation guidance enables SG-Net to produce sharper, more realistic details than a single SG-Net baseline, especially around vegetation and car boundaries.Interactive editing uses different label maps to produce multi-modal outputs for one incomplete image.
5 Conclusion
The paper concludes that its two-network framework uses predicted segmentation maps to guide realistic, sharp inpainting and supports interactive manipulation for multi-modal outputs.
- 5 Conclusion: SPG-Net combines two distinct networks that provide segmentation information and generate realistic, sharp details for image inpainting.The framework predicts segmentation label maps from incomplete inputs and uses them to guide texture generation.
- 5 Conclusion: Interactive editing manipulates segmentation maps to produce multi-modal inpainting outputs from a single input image.The paper suggests applications to broader image editing and manipulation tasks.