PaperlayerView exact revision on arXiv

Source-linked AI summary

Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes

Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, Shin'ichi Satoh

arXiv:2003.06877v3cs.CV

TL;DR

Inpainting mixed scenes is difficult because different semantic regions affect missing pixels differently, and structural predictions can be unreliable. SGE-Net couples semantic segmentation with image inpainting in a coarse-to-fine process, using segmentation guidance and confidence-based refinement. The authors report improved mixed-scene completion, with semantically realistic contexts and visually pleasing textures.

  • Problem

    Mixed-scene inpainting remains difficult because different semantic regions have different priors and contribute differently to missing pixels, while structural prediction can be unreliable.

    Source: §1 · p. 2

  • Method

    SGE-Net couples segmentation and inpainting coarse-to-fine, using semantic maps for guidance and segmentation confidence to locate ambiguous pixels for progressive refinement.

    Source: §1 · p. 3

  • Results

    SGE-Net outperforms state-of-the-art methods, especially on mixed scenes, in semantically realistic contexts and visually pleasing textures.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The progressive multi-scale refinement mechanism refines poorly inferred regions to produce semantic structures and texture details in a coarse-to-fine manner.

    Source: §5 · p. 14

    Confidence-based reliability evaluation assumes that an inpainted pixel is unreliable when its scores are low for all semantic classes.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Completing a corrupted image with correct structures and reasonable textures for a mixed scene remains an elusive challenge. Since the missing hole in a mixed scene of a corrupted image often contains various semantic information, conventional two-stage approaches utilizing structural information often lead to the problem of unreliable structural prediction and ambiguous image texture generation. In this paper, we propose a Semantic Guidance and Evaluation Network (SGE-Net) to iteratively update the structural priors and the inpainted image in an interplay framework of semantics extraction and image inpainting. It utilizes semantic segmentation map as guidance in each scale of inpainting, under which location-dependent inferences are re-evaluated, and, accordingly, poorly-inferred regions are refined in subsequent scales. Extensive experiments on real-world images of mixed scenes demonstrated the superiority of our proposed method over state-of-the-art approaches, in terms of clear boundaries and photo-realistic textures.

1 Introduction

Mixed scenes challenge inpainting because different semantic regions have distinct pixel contributions, while structural predictions can be unreliable and leave ambiguous textures. SGE-Net couples segmentation and inpainting in a coarse-to-fine process, using semantic guidance and confidence evaluation; the authors report stronger results, especially for mixed scenes.

1 Introduction

  • Context-based methods can produce unrealistic semantic content because mixed-scene regions have different prior distributions and contribute differently to missing pixels.The paper attributes this problem to mapping different semantics uniformly onto a single manifold.Source: §1 · p. 2
  • Two-stage structural methods can reduce blurry boundaries but may leave ambiguous textures at semantic boundaries and depend on unreliable reconstructed structures.The paper notes that edge or contour connection unreliability increases in mixed scenes.Source: §1 · p. 2
  • SPG benefits from semantic guidance, but unreliable predicted labels can degrade completion; a reliable segmentation map supports high-quality inpainting.The introduction identifies reliable segmentation prediction for corrupted regions as the key challenge.Source: §1 · p. 3
  • The proposed coarse-to-fine interplay between segmentation and inpainting aims to improve segmentation reliability and inpainting performance from a corrupted image.The authors describe this as simultaneously generating accurate semantic guidance from the corrupted input.Source: §1 · p. 3
  • Segmentation confidence scoring localizes pixels with ambiguous semantic meanings, allowing contexts and textures to be updated progressively.The method uses segmentation confidence as a self-evaluation mechanism during inpainting.Source: §1 · p. 3
  • SGE-Net outperforms state-of-the-art methods, especially on mixed scenes, in semantic realism and visually pleasing textures.The reported comparison emphasizes semantically realistic contexts and visually pleasing textures.Source: §1 · p. 3

2 Related Work

Learning-based inpainting has improved texture recovery and irregular-hole handling, but methods designed for single categories or backgrounds can struggle with mixed scenes. Structural-information methods guide texture generation with reconstructed structures, though recovering correct structures remains challenging for complex holes.

2.1 Deep Learning-Based Inpainting

  • Deep learning methods improve texture recovery through varied losses and feature strategies, while partial and gated convolutions address irregular holes; these methods can fail when holes span multiple scene categories.The passage contrasts their performance on one category or background with mixed-category missing regions.Source: §2.1 · p. 4

2.2 Structural Information-Guided Inpainting

  • Structural-information methods reconstruct edges, contours, smoothed images, or segmentation maps before using them to guide inpainting, but correct structures remain difficult to recover for complex holes.The cited methods include edge prediction, separate foreground/background reconstruction, and semantic segmentation guidance.Source: §2.2 · p. 4

3 Approach

The approach progressively predicts segmentation maps and inpainted images from a corrupted image, using semantic guidance and, in SGE-Net, confidence-based evaluation to refine contextual features across scales.

3 Approach

  • The approach progressively predicts a reliable segmentation map through the interplay of segmentation and inpainting, with SG-Net as a guidance baseline and SGE-Net adding semantic evaluation.Both networks operate in a coarse-to-fine manner.Source: §3 · p. 4, Figure 2 · p. 5, Figure 3 · p. 6

3.1 Semantic Guidance Network (SG-Net)

  • SG-Net uses decoder features to predict multi-scale images and segmentation maps, then uses each segmentation map with current context and encoder skip features to update context at the next scale.Its semantic guidance is applied through decoder supervision and the inference modules.Source: §3.1 · pp. 5–6

3.2 Semantic Guidance and Evaluation Network (SGE-Net)

  • SGE-Net evaluates segmentation confidence at each decoding layer, passes a reliability mask forward, and uses SGIM+ to update contextual features in regions marked unreliable.The method assumes pixels with low confidence across all semantic classes are unreliable.Source: §3.2 · p. 7

3.3 Training Loss Function

  • The final training loss weights multi-scale reconstruction, adversarial, and multi-scale cross-entropy losses for image inpainting and semantic segmentation.Reconstruction uses L1 and perceptual losses; the adversarial loss targets realistic textures, while cross-entropy penalizes segmentation deviations at each scale.Source: §3.3 · p. 8

4 Experiments

Experiments evaluate SGE-Net on Outdoor Scenes and Cityscapes, with additional tests on Places2 and ablations of its guidance and confidence components. Across these tests, the reported results favor SGE-Net in objective scores, visual quality, and user preference.

4.2 Image Inpainting Results

  • Visual comparisons show SGE-Net producing better semantic-region boundaries and more consistent textures than SG-Net and the three baselines.SG-Net produces more realistic textures than the baselines but retains boundary flaws tied to its initial inpainting result.Source: §4.2 · p. 9, Figure 4 · p. 10
  • SGE-Net achieves significantly better objective scores than the baselines, especially in PSNR and SSIM, on Outdoor Scenes and Cityscapes.The comparison also reports FID, but the supplied passages do not provide metric values.Source: §4.2 · p. 9, Table 1 · p. 10
  • In the user study, 67.4% of comparisons preferred SGE-Net over GC, 70.7% over EC, and 73.2% over SPG.The study involved 24 subjects ranking results on 80 images from both datasets.Source: §4.2 · pp. 9–10, Table 2 · p. 11
  • For high-complexity scenes containing more than four semantics, preference for SGE-Net over the baselines ranges from 77.6% to 83.7%.The 80-image analysis defines low, moderate, and high complexity using the number of semantics in each scene.Source: §4.2 · pp. 10–11, Table 2 · p. 11
  • On Places2, SGE-Net generates proper semantic structures when applied to images with scenes similar to Outdoor Scenes, despite lacking segmentation annotations for Places2.The reported Places2 test uses the model trained on Outdoor Scenes.Source: §4.2 · p. 11, Figure 5 · p. 11

4.3 Ablation Study

  • Ablations show improvements as modules are added: SGIM supports realistic textures and scene layout, while SCEM further improves completed structures and textures.The comparison includes Basic-Net, SG-Net with SGIM, and SGE-Net with both SGIM and SCEM.Source: §4.3 · pp. 11–12, Table 3 · p. 12
  • Across decoding scales, unreliable regions shrink and confidence scores rise; across 9,000 images, higher confidence is associated with lower L1 loss.The confidence analysis excludes the qualification that scores at region boundaries may not follow the reported pattern.Source: §4.3 · pp. 12–13
  • SGE-Net predicts the inpainted image and segmentation map simultaneously without completion-time annotations, and its segmentation outperforms segmentation-after-inpainting methods.The reported comparison attributes more accurate semantic assignments and object boundaries to joint optimization of semantics and image contents.Source: Figure 10 · p. 13, §4.3 · p. 14

5 Conclusion

SGE-Net uses progressive multi-scale refinement to couple semantic segmentation and image inpainting, refining poorly inferred regions into promising structures and texture details. Table 4 compares objective quality for models trained with automatic versus human-labeled segmentation.

5 Conclusion

  • Table 4 compares objective quality for models trained with automatic segmentation and human-labeled semantics.Source: Table 4 · p. 14
  • The progressive multi-scale refinement mechanism refines poorly inferred regions to generate promising semantic structures and texture details.The refinement proceeds in a coarse-to-fine manner.Source: §5 · p. 14
  • SGE-Net completes corrupted mixed-semantic-region images using a semantic-segmentation-guided scheme.Its refinement mechanism conducts interplay between semantic segmentation and image inpainting.Source: §5 · p. 14

Related papers

Loading 2003.06877v3…