PaperlayerView exact revision on arXiv

Source-linked AI summary

MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting

Xiaoguang Li, Qing Guo, Di Lin, Ping Li, Wei Feng, Song Wang

arXiv:2203.06304v1cs.CVcs.MM

TL;DR

Generative inpainting can neglect neighboring-pixel and feature smoothness, leaving fidelity dependent on data and training. MISF combines image-level and deep-feature filtering through linked kernel-prediction and semantic-and-image-filtering branches. The authors report that it outperforms state-of-the-art methods on three public datasets.

  • Problem

    Generative inpainting encodes and decodes through a latent space, neglecting explicit smoothness across neighboring pixels or features and leaving fidelity dependent on data and training strategy.

    Source: §1 · p. 2

  • Method

    MISF uses a predictive network to filter at the image and deep-feature levels, with linked branches that exchange multi-level features and dynamic kernels.

    Source: §4.2 · p. 5, §6 · p. 9

  • Results

    MISF outperforms state-of-the-art methods on three public datasets and achieves better PSNR, SSIM, and L1 scores across datasets and mask ratios than competing methods.

    Source: §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that dynamically predicted kernels give MISF high generalization capability.

    Source: §6 · p. 9

    Whether the growth continues as the network becomes deeper remains for future study.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Although achieving significant progress, existing deep generative inpainting methods are far from real-world applications due to the low generalization across different scenes. As a result, the generated images usually contain artifacts or the filled pixels differ greatly from the ground truth. Image-level predictive filtering is a widely used image restoration technique, predicting suitable kernels adaptively according to different input scenes. Inspired by this inherent advantage, we explore the possibility of addressing image inpainting as a filtering task. To this end, we first study the advantages and challenges of image-level predictive filtering for image inpainting: the method can preserve local structures and avoid artifacts but fails to fill large missing areas. Then, we propose semantic filtering by conducting filtering on the deep feature level, which fills the missing semantic information but fails to recover the details. To address the issues while adopting the respective advantages, we propose a novel filtering technique, i.e., Multilevel Interactive Siamese Filtering (MISF), which contains two branches: kernel prediction branch (KPB) and semantic & image filtering branch (SIFB). These two branches are interactively linked: SIFB provides multi-level features for KPB while KPB predicts dynamic kernels for SIFB. As a result, the final method takes the advantage of effective semantic & image-level filling for high-fidelity inpainting. We validate our method on three challenging datasets, i.e., Dunhuang, Places2, and CelebA. Our method outperforms state-of-the-art baselines on four metrics, i.e., L1, PSNR, SSIM, and LPIPS. Please try the released code and model at https://github.com/tsingqguo/misf.

1. Introduction

The introduction identifies a gap between natural-looking generated inpaints and high-fidelity results that generalize across scenes and missing areas. It motivates combining adaptive image-level filtering with semantic filtering to address that gap.

1. Introduction

  • Latent-space encoding and decoding neglect explicit smoothness across neighboring pixels or features, leaving fidelity dependent on data and training strategy.Source: §1 · p. 2
  • Deep generative inpainting can produce artifacts or distort local structures, and may fail to generate natural results across different scenes and missing areas.The introduction contrasts naturalness with fidelity to the clean image and calls for generalization across scenes and mask shapes.Source: §1 · p. 2
  • MISF achieves better scores than competitive methods on three public datasets across four image-quality metrics.The datasets are Places2, CelebA, and Dunhuang; the metrics are L1, PSNR, SSIM, and LPIPS.Source: Figure 2 · p. 2
  • The proposed MISF links kernel-prediction and semantic-and-image-filtering branches, using multi-level features and dynamic kernels for semantic and pixel-level filling.The introduction presents this combination as a way to retain the respective advantages of image-level and semantic filtering.Source: Figure 2 · p. 2, §1 · p. 2

2. Related Work

Related work moves from patch-based restoration to generative methods that model semantics, then to predictive filtering. The paper positions deep-feature filtering as a way to combine semantic information with image details.

2. Related Work

  • Conventional patch-based methods recover damaged regions from useful patches but omit semantic information, producing unsatisfactory results in complex scenes [8].Source: §2 · p. 2
  • Deep generative adversarial networks learn semantic information from data for better inpainting, while Pathak et al. and Iizuka et al. use conditional GANs [22] with a powerful information encoder to better polish image details.Source: §2 · p. 2
  • Other methods model recurrent feature reconstruction, long-range context, irregular damage, or image geometry, yet generated images can differ from the ground truth.Source: §2 · p. 3
  • Predictive filtering adapts to surrounding information but image-level filtering struggles with semantic understanding; the paper extends filtering to deep features and combines both levels to recover semantics and details.Source: §2 · p. 3

3. Discussion and Motivation

The discussion presents pixel-wise predictive filtering as a locally adaptive way to reconstruct image regions, then examines where its local context is insufficient. Larger missing areas break that local structure, and recurrent filtering accumulates reconstruction errors.

3.1. Predictive Filtering for Image Inpainting

  • Pixel-wise predictive filtering uses per-pixel kernels to linearly combine neighboring pixels, adapting the reconstruction to local context.The predictive network estimates kernels from the input image; the formulation includes kernels for filtering all pixels.Source: §3.1 · p. 3
  • The predictive network is an encoder-decoder trained with L1, GAN, style, and perceptual losses.Source: §3.1 · p. 3

3.2. Challenges and Motivations

  • Filtering can complete thin, small missing areas, but pixels far from boundaries remain unrecovered when larger holes break local structure; local context also cannot supply whole-scene understanding.Source: §3.2 · p. 3
  • A new technique is motivated by image-level filtering's limits on large missing areas and its inability to understand whole scenes.Source: §3.2 · pp. 3–4
  • Recurrent filtering makes pixels near the center of large missing areas vague as reconstruction errors accumulate; a generative-network hybrid can introduce artifacts.Source: §3.2 · p. 3
  • Figure 3 contrasts image-level filtering, semantic filtering, and MISF; Table 1 presents the MISF architecture.The architecture description identifies the encoder-decoder and predictive networks and defines the filtering kernel size and convolution notation.Source: Figure 3 · p. 4, Table 1 · p. 4, §3.2 · p. 4

4. Methodology

The method progresses from semantic filtering of deep features to MISF, which links kernel prediction with joint image- and feature-level filtering. Its design aims to recover missing semantic structure while retaining detail, with implementation choices balancing fidelity against memory and time costs.

4.1. Semantic Filtering for Image Inpainting

  • Semantic filtering applies adaptive kernels to deep encoder features, filling missing semantic structure but leaving detail artifacts.The encoder extracts features and the decoder maps them to a completed image.Source: §4.1 · pp. 4–5

4.2. Multi-level Interactive Siamese Filtering

  • MISF's KPB uses SIFB multi-level features to predict kernels, enabling joint image- and feature-level filtering instead of costly independent predictors.The predicted kernels dynamically filter SIFB features and the image.Source: §4.2 · p. 5

4.3. Relationship to Encoder-Decoder Network

  • The naive encoder-decoder produces artifacts and structure mismatches, while semantic filtering adapts to scenes and full MISF improves semantic and detail recovery.Semantic filtering adds dynamic convolution whose element-wise parameters are tuned according to the image and its semantic meaning.Source: §4.3 · pp. 5–6

4.4. Implementation Details

  • For memory and time, semantic filtering is applied only at F3; training uses L1, GAN, Style, and perceptual losses with Adam.Loss weights are λ1 = 1, λ2 = λ3 = 0.1, and λ4 = 250; training runs about 350,000 iterations with batch size 16 and learning rate 0.0001.Source: §4.4 · p. 6

5. Experiments

Experiments report consistent MISF advantages across three datasets and evaluate its filtering components, feature behavior, and dynamic-convolution design.

5.1. Setups

  • Evaluation covers Places2, CelebA, and Dunhuang, using four image-quality metrics and irregular-mask size categories on Places2 and CelebA.Places2 testing uses 30,000 random images; CelebA and Dunhuang use their standard test sets, and Dunhuang follows its official mask setup.Source: §5.1 · p. 7

5.2. Comparison Results

  • MISF achieves better PSNR, SSIM, and L1 scores across datasets and mask ratios, with reported LPIPS gains and consistent generalization.Reported examples include 7.01% relative higher PSNR than RFRNet on Places2 at 0%–20% mask ratio and 47.12% relative lower LPIPS than JPGNet at that ratio.Source: §5.2 · p. 7
  • Visual comparisons show more natural, higher-fidelity results on large holes and finer recovered structures on small holes than competing methods.The cited comparisons note distortions and blur in other methods on large missing areas and detail failures around local structures.Source: §5.2 · pp. 7–8

5.3. Ablation Study

  • MISF’s combined image-level and semantic filtering performs better across metrics, datasets, and missing sizes than the individual filtering variants.Image-level filtering performs well on Dunhuang but poorly on Places2 and CelebA, consistent with its strength on small holes and difficulty with large missing areas; semantic filtering loses details that MISF recovers.Source: §5.3 · p. 8, Table 3 · p. 8

5.4. Discussion

  • Filtered corrupted-image features become more similar to ground-truth features, with a larger similarity margin at larger mask ratios.The analysis randomly samples 1,000 examples from different mask ratios and the whole dataset.Source: §5.4 · p. 8, Figure 7 · p. 9
  • Semantic-filtering performance generally improves with deeper features across four metrics and three missing sizes, but whether the trend continues at greater depths remains open.The discussion attributes the observed trend to deeper features’ better semantic representations.Source: §5.4 · pp. 8–9, Table 4 · p. 8
  • Encoder-decoder variants with more dynamic convolutional operations yield better inpainting results across four metrics, datasets, and missing areas.MISF has two such operations, while the described single-operation variants have one and the naive encoder-decoder has none.Source: §5.4 · p. 9

6. Conclusions

The conclusion presents MISF as a two-level filtering approach for high-fidelity inpainting, reporting state-of-the-art comparisons on three public datasets while noting a real-world coverage limitation.

6. Conclusions

  • MISF uses image-level and deep-feature filtering to recover details and semantic information, respectively, with dynamically predicted kernels intended to support generalization.A single predictive network predicts kernels at both levels from the raw image and deep features.Source: §6 · p. 9
  • MISF outperforms state-of-the-art methods on three public datasets, and experiments support the effectiveness of its components.Source: §6 · p. 9
  • Training and validation on widely used public datasets may cover only part of real-world scenarios; broader scenarios are identified as future work.Source: §6 · p. 9

Supplementary Material

Supplementary visualizations present additional Places2 and CelebA results and feature-level examples of semantic filtering.

Supplementary Material

A. More Visualization Results

Additional CelebA and Places2 examples illustrate MISF’s reported fidelity and structural-detail comparisons across faces, masks, and natural scenes.

A. More Visualization Results

  • Across four CelebA cases, MISF is reported to produce natural, high-fidelity results, including on a large missing face region.Other methods are described as introducing structure distortions and blur.Source: A. More Visualization Results · p. 11
  • The supplemental examples are presented as evidence of MISF’s generalization across different faces, masks, and natural scenes.Source: A. More Visualization Results · p. 11
  • On Places2 examples, MISF is reported to recover detailed window and wall structures across different scenes.The cited case contrasts this with distortions from Str.Flow, EdgeCon., and RFRNet, and slight blur from JPGNet.Source: A. More Visualization Results · p. 11

B. Visualization of Semantic Filtering

Feature visualizations compare semantic filtering with a naive encoder–decoder on CelebA and Places2. The examples show less-corrupted features and more natural, detailed inpainting, including recovered structures in natural scenes.

B. Visualization of Semantic Filtering

  • The feature comparison visualizes three channels of F3 and ˆF3 against features from a naive encoder–decoder method.Source: B. Visualization of Semantic Filtering · p. 11
  • On CelebA, encoder–decoder features are corrupted around cropped eyes and mouths, while semantic filtering fills missing feature-level information.The feature-level differences are marked with red boxes, and image-level differences with green arrows.Source: B. Visualization of Semantic Filtering · p. 11, Figure C · p. 13, Figure D · p. 13
  • The feature visualizations show the method producing clear, natural eyes with more detail and high-fidelity structures, unlike distorted results from the encoder–decoder baseline.The comparisons include CelebA eye and mouth crops and Places2 natural-scene cases.Source: B. Visualization of Semantic Filtering · p. 11
  • On Places2, semantic filtering fills more missing areas and recovers structures where the encoder–decoder produces a distorted line.The Places2 visualizations mark feature-level differences with red boxes and image-level differences with green arrows.Source: B. Visualization of Semantic Filtering · pp. 11, 14

Related papers

Loading 2203.06304v1…