Source-linked AI summary
Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding
Qiaole Dong, Chenjie Cao, Yanwei Fu
TL;DR
Image inpainting must recover vivid textures and reasonable structures despite CNN receptive-field limits and the computational cost of attention at large resolutions. ZITS addresses this with a low-resolution transformer structure restorer, learned sketch upsampling, ZeroRA integration with pretrained inpainting, and masking positional encoding. Experiments across datasets and resolutions report significant improvements over state-of-the-art baselines, while the method remains limited for distant views represented at only 256×256.
Problem
Image inpainting still struggles to recover vivid textures and holistic structures because CNNs have limited receptive fields while attention models are computationally heavy for large images.
Method
ZITS incrementally restores holistic structures in a low-resolution grayscale sketch space, upsamples them, and injects their features into a pretrained inpainting model using ZeroRA and masking positional encoding.
Results
ZITS achieves significant improvements over state-of-the-art inpainting models across experiments with various datasets and resolutions.
Takeaways & Limitations
The framework provides an incremental structure-enhanced inpainting approach that can be integrated with pretrained models using a few finetuning steps.
Takeaways & Limitations
Because edges and lines are recovered only at 256×256, some distant views and complex urban distant scenes are not described correctly.
Abstract
from arXiv · showhide
Image inpainting has made significant advances in recent years. However, it is still challenging to recover corrupted images with both vivid textures and reasonable structures. Some specific methods only tackle regular textures while losing holistic structures due to the limited receptive fields of convolutional neural networks (CNNs). On the other hand, attention-based models can learn better long-range dependency for the structure recovery, but they are limited by the heavy computation for inference with large image sizes. To address these issues, we propose to leverage an additional structure restorer to facilitate the image inpainting incrementally. The proposed model restores holistic image structures with a powerful attention-based transformer model in a fixed low-resolution sketch space. Such a grayscale space is easy to be upsampled to larger scales to convey correct structural information. Our structure restorer can be integrated with other pretrained inpainting models efficiently with the zero-initialized residual addition. Furthermore, a masking positional encoding strategy is utilized to improve the performance with large irregular masks. Extensive experiments on various datasets validate the efficacy of our model compared with other competitors. Our codes are released in https://github.com/DQiaole/ZITS_inpainting.
1. Introduction
Image inpainting must recover both semantically coherent textures and visually reasonable structures, but CNNs struggle with limited receptive fields while attention models incur heavy computation at large resolutions. ZITS incrementally restores holistic structures in a low-resolution sketch space and integrates them into pretrained inpainting models with positional encoding for masked regions.
- Image inpainting supports applications including object removal, photo restoration, and image editing while requiring coherent textures and reasonable structures.
- CNN-based methods struggle with large corrupted regions and high-resolution images because local operations provide limited receptive fields and weak holistic understanding.
- Attention-based methods improve long-range dependency but require substantial memory for large images, motivating more efficient structural inference.
- ZITS uses a transformer to learn holistic structures in a normalized grayscale sketch space that can be upsampled to higher resolutions.
- Zero-initialized residual addition incorporates auxiliary structural information into pretrained inpainting models without retraining, while masking positional encoding improves restoration for different masks.
- Experiments on Places2, ShanghaiTech, NYUDepthV2, and MatterPort3D report that the proposed model outperforms other state-of-the-art competitors.
2. Related Work
Prior inpainting work uses auxiliary structural information such as edges, segmentation maps, and gradients to constrain reconstruction. These approaches include edge-guided, coupled structure-texture, iterative segmentation, and sketch-space methods.
- Auxiliary edges, segmentation maps, and gradients have been used to improve image inpainting.
- EdgeConnect uses edges to help inpaint images with certain structures, while a two-stream network couples structure-constrained texture synthesis with texture-guided structure reconstruction.
- SGE-Net iteratively updates semantic segmentation maps and corrupted images, whereas sketch tensor methods learn a sketch space to guide image inpainting.
3. Method
ZITS restores edges and lines with a low-resolution transformer, upsamples the resulting sketch maps, encodes them into multiscale features, and injects them into an FFC-based texture-restoration backbone through ZeroRA. Masking positional encoding supplies distance and direction information for masked regions.
- Transformer Structure Restoration: TSR receives masked images, Canny edges, lines, and a binary mask, then predicts recovered edge and line maps in a sketch space.
- Transformer Structure Restoration: The transformer operates on 32×32 features with 256 channels after convolutional downsampling, then uses transpose convolutions to output 256×256 structures.
- Transformer Structure Restoration: Alternating axial and standard attention reduces the cost of global modeling while retaining standard attention modules for global correlations.
- Transformer Structure Restoration: TSR is trained with binary crossentropy losses for predicted Canny edges and antialiased wireframe lines.
- Simple Structure Upsampler: The learning-based SSU upsamples grayscale edges and lines to arbitrary scales, avoiding zigzag artifacts associated with interpolation-based resizing.
- ZeroRA Structure Enhanced Inpainting: FTR uses Fast Fourier Convolution with local and global branches, while SFE converts upsampled structural maps into multiscale features for texture restoration.
- Masking Positional Encoding: MPE represents masked-region relations through distance and direction encodings, and ZeroRA stabilizes incremental finetuning by preserving the pretrained output initially.
4. Experiments
Experiments evaluate ZITS on Indoor and Places2 at multiple resolutions, comparing inpainting quality, structure recovery, and component ablations. ZITS generally outperforms competing methods, while ZeroRA and MPE improve incremental finetuning and high-resolution results.
- Datasets and Settings: ZITS is evaluated on Places2 and a 20,055-image Indoor dataset at 256×256 and higher resolutions.The Indoor dataset combines ShanghaiTech and NYUDepthV2 images; Places2 uses about 1,800k training images and 36,500 validation images.
- Quantitative Comparisons: At 256×256, ZITS achieves the best results on all reported Indoor metrics and improves over LaMa through integrated structural information and positional encoding.Evaluation uses PSNR, SSIM, FID, and LPIPS with mixed segmentation and irregular masks.
- Quantitative Comparisons: ZITS achieves slightly better Places2 results than Co-Mod despite Co-Mod using more training data and a sophisticated StyleGAN.The comparison reports FID and LPIPS on Places2 and notes superior overall performance to LaMa with only 50k finetuning steps.
- Structure Recovery: The transformer-based TSR obtains much better edge and line recovery than MST on both Indoor and Places2.The structure metrics are reported as precision, recall, and F1-score; TSR results use iterative Mask-Predict sampling without improving quantitative metrics.
- Ablation Studies: Without ZeroRA, adding TSR structure information yields limited improvement, whereas ZeroRA provides more stable convergence during finetuning.The ablation attributes instability without ZeroRA to dramatic output changes that harm vulnerable GAN training.
- High-Resolution Inpainting: MPE improves 512×512 Places2 finetuning, while the full model achieves prominent gains over LaMa on Places2(512), Indoor(512), and MatterPort3D(1k).The high-resolution comparisons use dynamic-resolution finetuning from 256 to 512 with 50k steps; qualitative results with MPE show natural and smooth colors.
5. Conclusions
The paper concludes that ZITS combines transformer-based structure restoration, ZeroRA-based incremental integration, and masking positional encoding. Across multiple resolutions, experiments report significant improvements over a state-of-the-art model.
- ZITS uses a transformer-based structure restorer to obtain better holistic structures than previous methods.
- ZeroRA incorporates auxiliary structural information into a pretrained inpainting model with a few finetuning steps.
- Masking positional encoding further improves inpainting performance across experiments at various resolutions.
A. Broader Impacts
The broader-impacts statement emphasizes that generated results reflect learned training-data statistics rather than subjective opinion. It also notes that societal impacts should not be ignored.
- Generated results reflect biases in the learned statistics of the training data.
- The work is presented for algorithmic discussion, while users are cautioned not to ignore related societal impacts.
B. Detailed Network Settings
The detailed network-settings section points readers to Tab. 6 for additional configuration details and identifies the origins of several blocks.
- Tab. 6 provides detailed network settings, including transformer and FFC blocks introduced in the main paper.
- The dilated ResNet block follows the middle layer of EdgeConnect with dilation set to 2.
C. More Training Details
The model uses regular-resolution resizing during training to avoid GPU-memory swaps and preserve training smoothness. Dynamic-resolution training reduces training cost, while ZeroRA fine-tuning remains brief.
- C. More Training Details: Regular-resolution resizing from 512 to 256 and back to 512 avoids frequent GPU memory swaps during training.Indoor uses one cycle per epoch, while Places2 uses 64 cycles per epoch.
- C. More Training Details: Dynamic-resolution training saves training cost compared with directly training across changing resolutions.
- C. More Training Details: ZeroRA fine-tuning takes about half a day at 256×256 and one day at 256∼512 resolutions.
D. Upsampling Iteratively with SSU
The Simple Structure Upsampler (SSU) can be applied iteratively to enlarge edge and line outputs, then resize them to a target resolution. The resulting upsampled structures are reported as good and robust at large sizes.
- SSU processes edge and line outputs through shifted sigmoid before iterative upsampling.The evaluation uses γ = 2 and β = 2, while finetuning randomly selects them from [1.5, 3].
- Repeating h×w edge and line inputs q times enables SSU to produce outputs of size 2^qh×2^qw.The outputs can then be resized to the target size with bilinear interpolation.
- SSU produces good and robust upsampled results for large image sizes.Figure 10 shows iterative outputs ranging from 512 to 2048 pixels, with consistent and robust results.
E. Supplementary Experiments
Supplementary experiments examine qualitative results, masking conditions, transformer efficiency, iterative Mask-Predict, and user evaluation. The reported findings indicate competitive scene completion, efficient axial attention, and richer structures from iterative prediction.
- Qualitative results: The method performs better in many man-made scenes and remains competitive in natural scenes, benefiting from MPE and edges.
- Quantitative evaluation: Quantitative experiments evaluate Indoor and Places2 across masking rates from 10% to 50% using mixed segmentation and irregular masks.
- TSR ablations: Replacing one standard self-attention module with axial attention greatly reduces GPU memory usage and speeds inference while keeping metrics basically unchanged.
- Mask-Predict: Mask-Predict first predicts all target pixels, then iteratively re-masks and re-predicts low-confidence pixels to enrich generated results without heavy costs.
- Mask-Predict: Direct TSR probability maps have insufficient recall, but larger Mask-Predict iterations gradually restore the masked interior structures.
- User study: A user study asked 10 volunteers unfamiliar with image inpainting to select the best result among shuffled outputs from four methods.
E.6. Results of Rectangular Masks
Experiments on rectangular and other challenging image sets compare the method with established inpainting systems across qualitative, quantitative, texture, and high-resolution structure-recovery settings. The method is reported to handle rectangular masks, periodic textures, and structural images competitively or with better structures.
- Rectangular-mask results: Results on 40% center rectangular masks use 1,000 Places(512) images without retraining, while only Co-Mod was trained with some rectangular masks.
- Visual comparisons: Figures compare qualitative outputs across Indoor and Places2, including predicted edges and lines, Mask-Predict ablations, and user-study scores.
- Rectangular-mask results: On qualitative comparisons, exemplar-based inpainting fails and is time-consuming, Co-Mod produces hallucinated artifacts, and LaMa outputs blur despite high PSNR.
- Texture images: For strong periodic textures in 1,880 texture images, the method maintains competitive performance despite the dataset favoring LaMa.
- High-resolution structure recovery: On MatterPort3D, comprising 1,965 indoor images resized from 1280×1024 to 1024×1024, the method produces better structures than LaMa on structural images.
F. More High Resolution Results
The method is evaluated on high-resolution images from 1k to 2k resolutions, including 2048-pixel cases, and generally produces clearer, more consistent results than LaMa, especially in weak-texture scenes. Its structural enhancement remains limited for distant, complex urban views because edges and lines are recovered only at 256×256.
- High-resolution comparisons: Figures 21–23 show object-removal instances from 1k to 2k resolutions, including cases selected from LaMa’s open-source test set.
- High-resolution comparisons: Figure 18 compares LaMa and the proposed method on 512 texture images, while Figure 20 compares them on 1024×1024 MatterPort3D images.
- High-resolution comparisons: Our method outperforms LaMa in weak-texture scenes and produces clearer borderlines for sparse regular textures and lines.For dense regular textures, LaMa remains competitive, although the proposed method can produce less blurred results through precise structural constraints.
- High-resolution comparisons: At 2048-pixel image size, our method achieves more consistent results than LaMa.
- Limitations: The method may fail to describe distant views correctly because it recovers edges and lines only in 256×256.Consequently, complex urban distant scenes cannot be enhanced by Canny edges and wireframe lines.