Source-linked AI summary
Resolution-robust Large Mask Inpainting with Fourier Convolutions
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, Victor Lempitsky
TL;DR
Large-mask image inpainting requires global context, which conventional network architectures and perceptual losses may not provide effectively. LaMa combines fast Fourier convolutions, a high-receptive-field perceptual loss, and large training masks; it generalizes to higher resolutions, handles periodic structures and large masks, and costs less in parameters and inference time than competitive baselines.
Problem
Large missing areas, complex geometric structures, and high-resolution images remain challenging because the inpainting network and loss may lack an effective receptive field.
Method
LaMa combines an FFC-based inpainting network, a high-receptive-field perceptual loss, and aggressive generation of large training masks.
Results
LaMa generalizes from low-resolution training to high-resolution images, captures complex periodic structures, remains robust to large masks, and uses fewer parameters and less inference time than competitive baselines.
Takeaways & Limitations
A single-stage inpainting system can be highly competitive when its architecture, loss function, and mask-generation strategy provide high receptive fields.
Takeaways & Limitations
LaMa usually struggles with strong perspective distortion, especially in complex images from the Internet, and it remains unclear whether FFCs can account for such deformations of periodic signals.
Abstract
from arXiv · showhide
Modern image inpainting systems, despite the significant progress, often struggle with large missing areas, complex geometric structures, and high-resolution images. We find that one of the main reasons for that is the lack of an effective receptive field in both the inpainting network and the loss function. To alleviate this issue, we propose a new method called large mask inpainting (LaMa). LaMa is based on i) a new inpainting network architecture that uses fast Fourier convolutions (FFCs), which have the image-wide receptive field; ii) a high receptive field perceptual loss; iii) large training masks, which unlocks the potential of the first two components. Our inpainting network improves the state-of-the-art across a range of datasets and achieves excellent performance even in challenging scenarios, e.g. completion of periodic structures. Our model generalizes surprisingly well to resolutions that are higher than those seen at train time, and achieves this at lower parameter&time costs than the competitive baselines. The code is available at \url{https://github.com/saic-mdal/lama}.
1. Introduction
LaMa addresses large-mask inpainting by combining image-wide receptive fields in its network and loss with aggressively generated large training masks. The resulting single-stage system generalizes to higher resolutions, handles complex periodic structures and large masks, and uses fewer parameters and less inference time than competitive baselines.
- Large effective receptive fields are essential for accessing global image structure, particularly when missing regions are large.
- LaMa uses fast Fourier convolutions whose receptive field covers the entire image, improving perceptual quality, parameter efficiency, and high-resolution generalization.
- LaMa introduces a high-receptive-field perceptual loss to promote consistency of global structures and shapes.
- Aggressive mask generation produces wide and large masks that force the network and loss function to exploit their high receptive fields.
- LaMa is a single-stage system combining high-receptive-field architecture, high-receptive-field loss, and aggressive mask generation.
- Evaluation finds that LaMa generalizes from low-resolution training to high-resolution images, captures complex periodic structures, remains robust to large masks, and has lower parameter and inference-time costs than competitive baselines.
2. Method
LaMa combines image-wide receptive fields in the inpainting network and perceptual loss with adversarial supervision and aggressive large-mask training. These components target global structure, local realism, and robust performance across mask scales.
- Overall pipeline: LaMa processes a masked image and binary mask as a four-channel input, producing an inpainted three-channel color image with a fully convolutional generator.Training uses real images paired with synthetically generated masks.
- Global context: Fast Fourier convolutions split local and global processing, use frequency-domain convolution, recover spatial features, and fuse both branches.The global branch provides an image-wide receptive field from early layers.
- Global context: FFCs provide global context early, improving efficiency by letting parameters reason and generate rather than wait for information propagation.They are presented as differentiable drop-in replacements for conventional convolutions.
- Global context: FFCs are well suited to capturing periodic structures such as bricks, ladders, and windows, while shared frequency convolutions shift the model toward scale equivariance.This supports their use for geometric patterns common in human-made environments.
- Loss design: The high receptive field perceptual loss compares predicted and target features from a high-receptive-field base network, using sequential interlayer and intralayer means.The base network can use Fourier or dilated convolutions, and segmentation pretraining emphasizes high-level information over texture.
- Loss design: LaMa combines adversarial, high-receptive-field perceptual, discriminator-feature-matching, and gradient-penalty terms, assigning local realism to adversarial components and global consistency to HRFPL.The adversarial discriminator operates on patches intersecting the masked area, while HRFPL supplies supervised global-structure consistency.
- Training masks: The aggressive mask policy samples wide polygonal-chain masks and arbitrary-aspect-ratio rectangles, and generally improves performance on both narrow and wide masks.The policy is intended to unlock the high receptive field of the network and loss function.
3. Experiments
Experiments show that LaMa-Fourier consistently outperforms most baselines, especially on wide masks, while using fewer parameters and transferring better to higher resolutions. Ablations attribute these gains to FFCs, high-receptive-field perceptual loss, and wider training masks.
- Comparisons to the baselines: LaMa-Fourier consistently outperforms most baselines across Places and CelebA-HQ, with especially noticeable gains on wide masks.CoModGAN and MADF are the closest competitors but use approximately 4× and 3× more parameters, respectively.
- Ablation Study: Adding FFCs substantially improves FID for wide-mask inpainting and recovers complex visual structures better than larger regular-convolution models.FFC-based models may sacrifice a little performance on narrow masks while outperforming bigger regular-convolution models on wide masks.
- Ablation Study: LaMa-Fourier is 20% slower but 40% smaller than LaMa-Regular, while better generating repetitive structures such as windows and chain-link fences.The comparison uses 512×512 images and reports feasible inference time for LaMa-Fourier.
- Ablation Study: A high-receptive-field perceptual loss improves inpainting quality, with both dilated convolutions and the pretext problem improving ablation scores.The loss was evaluated after a weight-coefficient search for each variant.
- Ablation Study: Wider training masks improve LaMa’s inpainting of both wide and narrow holes, whereas they worsen narrow-mask results for DeepFill v2 and EdgeConnect.The authors link this difference to design choices affecting suitability for both narrow and wide masks.
- Generalization to higher resolution: FFC-based models preserve more quality and consistency at 1536×1536 than regular-convolution models trained on lower-resolution crops.All LaMa models were trained using 256×256 crops from 512×512 images, whereas MADF was trained directly on 512×512 images.
4. Related Work
Prior image-inpainting work explores architectures and losses for combining local and global context, irregular masks, and perceptual realism. LaMa extends this line with distant-information propagation through frequency-domain convolutions and emphasizes mask width in training.
- Deep inpainting commonly uses encoder-decoder or U-Net completion networks, often trained adversarially.
- Researchers expanded contextual access with dilated convolutions, multi-receptive-field branches, contextual attention, and other attention mechanisms.LaMa instead proposes frequency-domain transformations through fast Fourier convolutions.
- Two-stage methods often complete coarse global structure before refining local details, following structure-texture decomposition.
- Prior work adapts convolutions and mask generation to handle irregular missing regions, using partial, gated, region-wise, random, free-form, and object-shaped variants.
- Mask-contour diversity matters, but the width of training masks is more important than the exact generation method.
- Inpainting systems combine pixel-wise and adversarial losses, including spatial discounting, PatchGAN, Wasserstein, and gradient-penalized discriminator variants.
5. Discussion
The study finds that a simple single-stage large-mask inpainting approach can be competitive, especially for repetitive structures and unseen high resolutions. Its main boundary is difficulty with strong perspective distortion.
- LaMa is competitive with state-of-the-art baselines when architecture, loss function, and mask-generation strategy are appropriately chosen.
- LaMa is particularly good at generating repetitive visual structures, which many inpainting methods find difficult.
- FFCs let LaMa generalize to unseen high resolutions and use fewer parameters than state-of-the-art baselines.
- LaMa usually struggles with strong perspective distortion, especially in complex images from the Internet outside dataset conditions.The authors leave open whether FFCs can account for deformations of periodic signals.