Source-linked AI summary
RePaint: Inpainting using Denoising Diffusion Probabilistic Models
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, Luc Van Gool
TL;DR
Existing inpainting methods are limited by mask-specific training and can produce textural rather than semantically meaningful completions. RePaint conditions an unconditional pretrained DDPM by modifying reverse diffusion, and it reports better generalization and semantic quality across evaluated masks and datasets.
Problem
Existing approaches train for particular mask distributions and may extend textures into missing regions instead of generating semantically meaningful content.
Method
RePaint uses an unconditional pretrained DDPM and conditions generation by sampling known pixels during reverse diffusion, without training the model for inpainting.
Results
RePaint generalizes better and produces more semantically meaningful inpainted regions on CelebA-HQ and ImageNet than compared state-of-the-art approaches.
Takeaways & Limitations
The mask-agnostic conditioning approach supports free-form inpainting across mask types while producing photo-realistic images.
Takeaways & Limitations
Per-image DDPM optimization is significantly slower than GAN-based and autoregressive counterparts, limiting current real-time applicability.
Abstract
from arXiv · showhide
Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. Most existing approaches train for a certain distribution of masks, which limits their generalization capabilities to unseen mask types. Furthermore, training with pixel-wise and perceptual losses often leads to simple textural extensions towards the missing areas instead of semantically meaningful generation. In this work, we propose RePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pretrained unconditional DDPM as the generative prior. To condition the generation process, we only alter the reverse diffusion iterations by sampling the unmasked regions using the given image information. Since this technique does not modify or condition the original DDPM network itself, the model produces high-quality and diverse output images for any inpainting form. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks. RePaint outperforms state-of-the-art Autoregressive, and GAN approaches for at least five out of six mask distributions. Github Repository: git.io/RePaint
1. Introduction
RePaint addresses free-form inpainting by using an unconditional DDPM prior and conditioning only the reverse diffusion process, avoiding mask-specific training. It aims to improve generalization to arbitrary masks and produce semantically meaningful completions, with stronger results than prior approaches across evaluated settings.
- Free-form inpainting must fill missing regions with content that harmonizes with the image and is semantically reasonable.
- Existing methods train on particular mask distributions, which can lead to poor generalization to novel mask types.
- RePaint uses an off-the-shelf, unconditionally trained DDPM and samples known pixels during reverse diffusion instead of training a mask-conditional model.
- Its resampling strategy addresses cases where standard DDPM sampling matches neighboring textures but produces semantically incorrect content.
- Experiments on CelebA-HQ and ImageNet report better generalization and more semantically meaningful inpainted regions than other state-of-the-art approaches.
2. Related Work
Prior inpainting work spans low-level completion, GANs, autoregressive models, architectural guidance, and learned image priors. RePaint instead leverages an unconditional DDPM and conditions it during reverse diffusion to support arbitrary mask shapes.
- Early inpainting methods filled missing regions using low-level image cues or neighbors from large image datasets.
- GAN-based approaches commonly use encoder-decoder generators, adversarial training, and tailored losses designed for photorealism.
- Other methods add semantic context through dilated, partial, gated, contextual-attention, edge, segmentation, or Fourier-based architectural designs.
- VAE-based and autoregressive methods address diversity or irregular masks, while GAN-based methods can remain prone to deterministic synthesis.
- Existing image-prior approaches include StyleGAN and untrained generators, but StyleGAN-based methods are limited to specific scenarios such as faces.
- Unlike concurrent conditional diffusion methods, RePaint uses an unconditional DDPM, conditions only through reverse diffusion, and supports arbitrary mask shapes.
3. Preliminaries: Denoising Diffusion Probabilistic Models
DDPMs learn to reverse a Gaussian diffusion process: noisy images are progressively transformed into clean samples through neural-network reverse transitions. Training uses noise-prediction objectives and efficient sampling formulations.
- DDPM inference starts from random noise x_T and gradually denoises it to produce a high-quality image x_0.
- The forward process adds independent Gaussian noise with variance β_t while scaling x_{t-1} by √(1−β_t) according to a variance schedule.
- The reverse process models p_θ(x_{t−1}|x_t) as a Gaussian whose mean and variance are predicted by a neural network.
- The variational lower-bound objective decomposes into terms including reverse-step matching, and the L_{t−1} term trains one reverse diffusion step.
- The model parameterization predicts cumulative noise ε_θ(x_t,t) to determine the reverse-process mean μ_θ(x_t,t).
- A direct jump from x_0 to x_t samples noisy training inputs efficiently, allowing reverse-transition training pairs to be generated without simulating every intermediate step.
4. Method
RePaint conditions an unconditional DDPM by combining model-generated unknown regions with samples of known pixels during reverse diffusion. Resampling improves semantic harmonization by allowing generated content to incorporate conditional information across the denoising process.
- 4.1. Conditioning: RePaint conditions reverse diffusion by sampling known regions from the given image while the DDPM generates unknown regions.The masked components are combined into each new reverse-diffusion sample.
- 4.2. Resampling: Direct conditioning can match neighboring content types while producing semantically incorrect inpainting, such as furry texture extending a dog’s hair.The limitation arises because known-region sampling does not account for generated image parts at each step.
- 4.2. Resampling: Resampling diffuses the output backward in time so generated regions become more harmonized with known regions and incorporate conditional information.The DDPM’s consistency prior is used to harmonize the model input during inference.
- 4.2. Resampling: More resampling steps produce more harmonized images, with benefits saturating at about n = 10 resamplings.The figure compares n = 1, the DDPM baseline, with increasing numbers of resampling steps.
- 4.2. Resampling: Resampling addresses the limitation of one-step harmonization by incorporating semantic information over the entire denoising process.The approach increases reverse-diffusion runtime, but differs from slowing diffusion because it targets image harmonization.
5. Experiments
Experiments compare RePaint with state-of-the-art inpainting methods across datasets, mask types, sampling strategies, and computational-budget settings. RePaint shows strong mask robustness, semantic quality, diversity, and benefits from resampling.
- Comparison with State-of-the-Art: RePaint is evaluated against autoregressive and GAN-based methods on CelebA-HQ and ImageNet using standard and challenging mask settings.The evaluation includes 100 256×256 test images per dataset and reports LPIPS and user-study votes.
- Comparison with State-of-the-Art: RePaint outperforms all compared methods with 95% significance on both CelebA-HQ and ImageNet for Wide and Narrow masks.These are the standard image-inpainting settings used for benchmarking.
- Comparison with State-of-the-Art: 73.1% to 99.3% of user votes favor RePaint on thin-mask settings, where baselines produce failures, blur, or visible artifacts.The evaluated thin masks are Super-Resolution 2× and Alternating Lines.
- Comparison with State-of-the-Art: For large-area masks, RePaint generates semantically meaningful fillings while other methods often produce artifacts or copy textures.RePaint is preferred by users with 95% confidence except for the inconclusive ICT result on Half masks.
- Ablation Study: Resampling uses additional computation to harmonize images, whereas slowing diffusion shows no visible improvement under the same computational budget.The comparison uses 32 ImageNet validation images with the LaMa Wide mask setting.
- Ablation Study: RePaint’s resampling schedule significantly outperforms SDEdit across mask types except one Expand case, reducing LPIPS by over 53% for super-resolution masks.The Expand exception has LPIPS > 0.6, which the authors describe as outside a meaningful comparison range.
6. Limitations
RePaint produces sharp, detailed, semantically meaningful completions, but its per-image DDPM optimization is slow and extreme-mask outputs can differ substantially from ground truth.
- Per-image DDPM optimization is significantly slower than GAN-based and Autoregressive-based counterparts, limiting real-time application.
- For extreme masks, RePaint may generate realistic completions that differ substantially from the ground-truth image, complicating quantitative evaluation.The authors note that reliable FID evaluation typically requires more than 1,000 images, making the runtime infeasible for most research institutes.
7. Potential Negative Societal Impact
Because RePaint relies on an unconditional pretrained DDPM, its generations may reflect biases from the training dataset, including gender, age, and ethnicity.
- RePaint may inherit biases from the dataset used to train its unconditional DDPM, including biases related to gender, age, and ethnicity.
- The paper also identifies face anonymization as a possible use, replacing identity information with hallucinated artificial faces for data protection.
8. Conclusions
The paper presents RePaint as a DDPM-based, mask-agnostic solution for free-form image inpainting. Its appendix documents user-study procedures and diffusion-time scheduling details supporting the method’s evaluation and implementation.
- RePaint is presented as a novel denoising diffusion probabilistic model solution for image inpainting.
- The approach is mask-agnostic, increasing the allowable freedom of masks for free-form inpainting.
- Its conditioning approach complies with DDPM assumptions and is intended to produce photorealistic images regardless of mask type.
- Diffusion-time schedule: The appendix gives diffusion-time scheduling details for jumps, including forward and reverse transitions that change diffusion time by one.For jump length j = 10, the procedure applies ten forward transitions followed by ten reverse transitions.
- User study: The user study presents two candidate images in random order, shows each example twice, and uses agreement filtering to reduce bias and low-effort responses.
C. Ablation
The ablation examines how jump length and resampling count affect visual quality and consistency.
- Smaller jump lengths j tend to produce blurrier images, while increasing resamplings r improves overall image consistency.
D. Evaluation on Places2
Places2 experiments evaluate RePaint across five mask settings, finding robust performance and semantically meaningful fillings, with one inconclusive comparison against LaMa on Wide masks.
- The Places2 benchmark evaluates RePaint using an unconditional model trained for 300k iterations on four V100 GPUs over about six days.
- RePaint outperforms all competing methods for every mask setting at 95% significance except the Wide-mask comparison with LaMa.
- On Wide masks, users preferred RePaint to LaMa by 52.4%, but the significance interval overlapped the 50% threshold.
- RePaint better harmonizes sparse-mask results, while competing methods retain visible mask patterns and large-mask outputs often lack semantic content.
- For thin structured masks, RePaint achieves large-margin advantages in both LPIPS and Diversity Score over ICT and DSI.
- ImageNet failure cases show that RePaint can mix non-matching objects and overproduce dogs because of ImageNet’s class distribution.
H. Experiment on larger resolution
RePaint also works with a pretrained 512 × 512 model, although the authors could not perform the full analysis at that resolution because of limited computational resources.
- RePaint works on a pretrained 512 × 512 model, but the full analysis was not conducted because computational resources were limited.
I. Additional Visual Results
Additional visual results compare RePaint with state-of-the-art methods across CelebA-HQ, ImageNet, and Places2 using wide, narrow, sparse, half, and expand masks.
- Additional CelebA-HQ and ImageNet examples compare RePaint with the same state-of-the-art methods used in the main paper.
- The visual comparisons cover Wide and Narrow masks, sparse Super-Resolution 2× and Alternating Lines masks, and Half and Expand masks.
- CelebA-HQ figures present qualitative comparisons against state-of-the-art face-inpainting methods.
- ImageNet figures present qualitative comparisons for diverse inpainting across multiple mask settings.
- Places2 figures present qualitative comparisons for diverse inpainting across multiple mask settings.