PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Feature Reasoning for Image Inpainting

Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, Dacheng Tao

arXiv:2008.03737v1cs.CV

TL;DR

Large continuous holes remain difficult to inpaint because constraints weaken toward the hole center. RFR-Net progressively reasons over feature boundaries with knowledge-consistent attention and outperforms several state-of-the-art methods in performance and efficiency.

  • Problem

    Large continuous holes weaken correlations between known and unknown pixels, loosening constraints at the hole center and producing semantically ambiguous inpainting results.

  • Method

    RFR-Net recurrently infers and gathers hole-boundary information in encoded features, using Knowledge Consistent Attention to combine information across recurrences.

  • Results

    RFR-Net demonstrates superior performance and efficiency to several state-of-the-art methods on benchmark datasets, with especially fewer inconsistencies for large holes.

  • Takeaways & Limitations

    The proposed network provides semantically explicit inpainting results while progressively enriching information for masked regions.

Abstract

from arXiv · show

Existing inpainting methods have achieved promising performance for recovering regular or small image defects. However, filling in large continuous holes remains difficult due to the lack of constraints for the hole center. In this paper, we devise a Recurrent Feature Reasoning (RFR) network which is mainly constructed by a plug-and-play Recurrent Feature Reasoning module and a Knowledge Consistent Attention (KCA) module. Analogous to how humans solve puzzles (i.e., first solve the easier parts and then use the results as additional information to solve difficult parts), the RFR module recurrently infers the hole boundaries of the convolutional feature maps and then uses them as clues for further inference. The module progressively strengthens the constraints for the hole center and the results become explicit. To capture information from distant places in the feature map for RFR, we further develop KCA and incorporate it in RFR. Empirically, we first compare the proposed RFR-Net with existing backbones, demonstrating that RFR-Net is more efficient (e.g., a 4\% SSIM improvement for the same model size). We then place the network in the context of the current state-of-the-art, where it exhibits improved performance. The corresponding source code is available at: https://github.com/jingyuanli001/RFR-Inpainting

1. Introduction

Large holes challenge one-shot inpainting because weakened correlations leave hole centers insufficiently constrained. RFR-Net addresses this with recurrent feature reasoning and knowledge-consistent attention, while improving efficiency and performance against existing methods.

  • Problem: Large damaged areas weaken correlations between known and unknown pixels, loosening constraints on the hole center and making one-shot reconstruction difficult.One-shot inference remains reasonable for small or narrow defects, where local pixels have stronger correlations.
  • RFR-Net: RFR-Net recurrently infers and gathers hole boundaries in encoded feature maps, progressively strengthening constraints on internal contents.Its feature-space progression avoids requiring network inputs and outputs to share the same representation space.
  • RFR module: The RFR module exploits adjacent-pixel correlations to estimate deeper pixels while reusing parameters for a lighter model.The recurrent design also bypasses several limitations of progressive methods.
  • KCA module: KCA shares and adaptively combines attention scores across recurrences to maintain consistency between patch-swapping processes and preserve exquisite details.Existing attention designs can produce blurred recovered textures because they neglect cross-recurrence consistency.
  • Evaluation: RFR-Net is evaluated for efficiency and performance, demonstrating superiority to several state-of-the-art methods on benchmark datasets.The paper assembles the RFR and KCA modules into the proposed network and analyzes both efficiency and performance.

2. Related Works

Earlier image inpainting methods progressed from patch-based algorithms to semantic deep networks and attention mechanisms. Traditional methods struggle with complex scenes, while deep and attention-based approaches address semantic content, coherence, detail, and texture borrowing but retain limitations associated with progressive inpainting.

  • Image Inpainting: Traditional inpainting methods search background regions for patches to fill holes but cannot handle complex scenes because they lack semantic understanding.These algorithms perform well on simple cases.
  • Image Inpainting: Deep convolutional networks, particularly GANs, were introduced to semantically inpaint damaged images.Context-Encoder used a conditional GAN, while Iizuka et al. added a discriminator for local coherence and Poisson blending for sharper detail.
  • Image Inpainting: Despite semantic improvements, deep inpainting methods generally suffer from the progressive-inpainting limitations described in the introduction.
  • Attentive Inpainting: Attention-based models borrow background features using textural similarity, multi-scale patch swapping, or coherent semantic features.These mechanisms aim to produce realistic textures and improve patch-swapping precision.

3. Method

RFR-Net centers on a plug-and-play recurrent feature reasoning module that alternates area identification, feature inference, and feature merging to progressively fill holes. Its Knowledge Consistent Attention mechanism maintains consistency across recurrences, while perceptual and style losses provide structural and textural supervision.

  • RFR module: The plug-and-play RFR module alternates area identification, feature reasoning, and feature merging recurrently to infer missing feature regions.Area identification uses updated masks, feature reasoning estimates missing values, and feature merging combines intermediate feature maps.
  • Area identification: Cascaded partial convolutions update masks and feature maps, defining the difference between updated and input masks as the area inferred in each recurrence.The updated-mask holes become smaller after each partial convolution, while holes in the updated mask are preserved through subsequent processing.
  • Feature reasoning: The feature reasoning module fills identified areas with feature values that support both the final result and subsequent inferences.The implementation stacks encoding and decoding layers connected by skip connections, then forwards updated masks and partially inferred features directly to the next recurrence.
  • Feature merging: Intermediate feature maps are merged because using only the last map can cause gradient vanishing and damage earlier-iteration signals.Merging feature maps of arbitrary number gives RFR the potential to fill larger holes.
  • Knowledge Consistent Attention: KCA adaptively accumulates attention scores from previous recurrences to control inconsistencies between synthesized feature maps.For valid pixels, current and previous scores are combined using a weighted sum with learnable parameter λ; invalid pixels use the current score without the extra operation.
  • Training objectives: Perceptual and style losses from a fixed pre-trained VGG-16 teach structural and textural information by comparing generated-image and ground-truth deep feature maps.The losses are used for image generation learning.

4. Experiments

The experiments establish reproducible training and evaluation settings for RFR-Net, including iterative reconstruction, public inpainting datasets, and comparisons with several state-of-the-art methods.

  • Experimental procedure: RFR-Net iteratively identifies feature areas, reasons over updated features, merges the resulting feature groups, and decodes the reconstructed image.The procedure repeats this process until the specified iteration count is reached.
  • Implementation settings: Training uses batch size 6 with Adam, learning rates of 1e−4 initially and 1e−5 for fine-tuning, and frozen batch-normalization layers during fine-tuning.The loss weights are λhole=6, λvalid=1, λperceptual=0.1, and λstyle=180.
  • Datasets: The model is validated on three public image datasets commonly used for inpainting and an additional mask dataset.The Places2 Challenge Dataset contains over 8,000,000 images spanning over 365 scenes, while CelebA contains over 180,000 training images.
  • Baselines: RFR-Net is compared with PIC, PConv, GatedConv, EdgeConnect, and PRVS using the same experimental settings and training each model until convergence.The compared methods are identified as PIC [32], PConv [13], GatedConv [30], EdgeConnect, and PRVS.

5. Results

Experiments across Places2, CelebA, and Paris StreetView show that RFR-Net improves qualitative and quantitative inpainting results, including for large holes. Efficiency comparisons and ablations further support its compact architecture, KCA module, and recurrence design.

  • State-of-the-art comparisons: RFR-Net produces fewer noticeable inconsistencies than five state-of-the-art approaches across Places2, CelebA, and Paris StreetView, especially for large holes.Qualitative comparisons are presented in Figs. 5, 6, and 7.
  • State-of-the-art comparisons: RFR-Net reports the highest SSIM, PSNR, and mean l1 loss across the three datasets and different irregular-mask ratios.Missing table entries are attributed to computational-resource limitations.
  • Efficiency comparison: The bare RFR-Net has fewer parameters than the Coarse-To-Fine and PConv-UNet backbones.The comparison excludes the attention module and uses Table 2’s model-size analysis.
  • Efficiency comparison: 85–95 ms: the bare RFR-Net’s usual per-image inference time is faster than several state-of-the-art methods, while achieving the best SSIM and PSNR among tested networks.The Paris StreetView comparison uses a 40%–50% mask ratio; Coarse-To-Fine results come from the CSA paper because its official code is unavailable.
  • Ablation studies: The KCA ablation compares attention modules quantitatively using SSIM, PSNR, and Mean l1 on Paris StreetView with a 40%–50% mask ratio.The study motivates KCA by noting that direct feature-map attention can produce shadow-like artifacts after feature merging.
  • Ablation studies: RFR-Net is robust to changes in IterNum, and increasing recurrence does not improve performance, indicating gains arise from architecture efficiency rather than deeper layers.The paper also notes that the model is smaller than state-of-the-art methods.

6. Conclusion

The paper proposes RFR-Net, which progressively enriches information in masked regions to produce semantically explicit inpainted results, supported by a Knowledge Consistent Attention module and extensive evaluations.

  • RFR-Net progressively enriches information for masked regions and produces semantically explicit inpainted results.
  • The Knowledge Consistent Attention module assists the inference process of the Recurrent Feature Reasoning module.
  • Extensive quantitative and qualitative comparisons, efficiency analysis, and ablation studies demonstrate RFR-Net’s superiority in performance and efficiency.

Supplementary Material · A. Network Architecture

The supplementary material specifies the RFR-Net and RFR Module architecture, defines the notation used in Table 4, and explains the design goals of feature reasoning and KCA. It also identifies global feature search as KCA’s distinction from Partial Convolution U-Net.

  • A. Network Architecture: Table 4 defines Input Feat as the feature source and In Size and Out Size as processed feature-map sizes.Ori denotes the original input size.
  • A. Network Architecture: Table 4 uses K Size for operator kernel size and Stride for operator stride.
  • A. Network Architecture: Table 4 presents the architectures of the whole RFR-Net and the RFR Module.The table is divided into an upper half for RFR-Net and a bottom half for the RFR Module.
  • Supplementary Material: Table 5 compares different places to put the module using higher-is-better and lower-is-better evaluation directions.The table marks these directions with ⋆ and †, respectively.
  • Supplementary Material: Figure 9 compares four feature-merging methods in the RFR module: Input, No Merging, Average Merging, and Adaptive Merging.
  • A. Network Architecture: The feature reasoning module partly recovers masked regions in deep feature maps by extracting known features and estimating new contents.The new contents are produced in a feature reconstruction style with additional components described in the architecture.
  • A. Network Architecture: KCA searches for features globally, whereas Partial Convolution U-Net captures information locally.

B. More Ablation Studies … B.3. RFR Module For Structure Estimation

The supplementary ablations examine where the RFR module can be inserted, how feature maps should be merged, and whether it benefits networks with different input and output spaces. They report flexible placement, adaptive feature merging, and use of RFR for structure estimation in a multi-stage method.

  • B. More Ablation Studies: The ablations test whether RFR can be installed at different locations in an existing network while controlling computational cost.The section describes placement flexibility and computational-cost control as primary goals.
  • B.1. Moving the RFR Module: RFR is evaluated in four models with modified encoding or decoding layers to move the module up and down the network.The tested variants include using RFR without downsampling layers by modifying PartialConv0 and DeConv4.
  • B.1. Moving the RFR Module: The placement study is designed to show that RFR can be flexibly installed into any part of a network.
  • B.2. Effect of Feature Merging: Using only the last feature map for RFR feature merging produces blurred and inadequate texture because earlier-recurrence features may be damaged during later hole recovery.
  • B.2. Effect of Feature Merging: Replacing adaptive merging with average merging reduces the quality of restored details.The passage contrasts the proposed adaptive merging with average merging in Paris StreetView.
  • B.3. RFR Module For Structure Estimation: RFR is tested as a way to boost a subnetwork in Edge-Connect, a multi-stage method that reconstructs boundaries before guiding image inpainting.Because the two networks have different input and output spaces, existing progressive methods are not feasible in this setting.
  • B.3. RFR Module For Structure Estimation: The structure-estimation ablation specifically investigates an RFR module within the first network of the Edge-Connect pipeline.Figure 10 is identified as showing the RFR module for structure estimation.

C. More Results · C.1. More Comparisons

Additional comparisons on CelebA and Paris StreetView show that RFR-Net produces more explicit, well-structured inpainting results with fewer boundary artifacts, including for large and challenging holes. Further visual results across three datasets with ground-truth images support the effectiveness of the proposed methods.

  • C. More Results: RFR-Net comparisons are presented on the CelebA and Paris StreetView datasets.These visual comparisons were omitted from the paper’s main text because of space limitations.
  • C. More Results: Additional visual results are shown on three datasets with ground-truth images.The results are presented as further evidence for the proposed methods.
  • C. More Results: Across the additional results, the proposed methods demonstrate effectiveness.The passage states this conclusion for the visual comparisons and ground-truth examples collectively.
  • C.1. More Comparisons: RFR-Net exhibits fewer boundary artifacts than the comparison methods.This comparison is reported for results on Paris StreetView and CelebA.
  • C.1. More Comparisons: RFR-Net generates substantially more explicit content than the comparison methods.The passage describes this as a characteristic of the model’s results on Paris StreetView and CelebA.
  • C.1. More Comparisons: RFR-Net stably produces well-structured results even when holes are large and challenging.Figure 11 compares the input, ground truth, GatedConv, PConv, EdgeConnect, and RFR-Net outputs.

C.2. More Visual Results

Figure 12 presents visual comparisons in which RFR-Net produces coherent structures and plausible details relative to the inputs and ground truths.

  • C.2. More Visual Results: Figure 12 shows RFR-Net results with coherent structures and plausible details alongside corresponding inputs and ground truths.The figure presents two comparisons, arranged left to right as input, RFR-Net output, and ground truth.
Loading 2008.03737v1…