PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models

Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi Jaakkola, Shiyu Chang

arXiv:2304.03322v1cs.CVcs.LG

TL;DR

Fixed diffusion inpainting methods can leave revealed and unrevealed regions incoherent, while Bayesian methods can incur constraint mismatches from posterior approximation errors. COPAINT jointly modifies both regions through a Bayesian approach whose approximation errors decrease to zero by the final denoising step. Experiments report improved inpainting quality and coherence, including lower LPIPS than REPAINT and reduced computation on ImageNet.

  • Problem

    Replacement-based fixed diffusion methods may leave unrevealed pixels unmodified to match context, while posterior approximation errors can make strict constraint satisfaction difficult.

    Source: §1 · p. 1

  • Method

    COPAINT uses a Bayesian framework to jointly modify revealed and unrevealed regions, with posterior approximation errors designed to decrease to zero during denoising.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    19% average relative LPIPS reduction versus REPAINT and 31% less computation on ImageNet accompany better quality and coherence across CelebA-HQ and ImageNet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the evaluated datasets and revealed-region shapes, COPAINT outperforms existing diffusion-based approaches on objective and subjective quality and coherence metrics.

    Source: §1 · p. 2

    With H = 5, multi-step approximation takes about six times longer per image than H = 1 despite a minor LPIPS decrease.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Image inpainting refers to the task of generating a complete, natural image based on a partially revealed reference image. Recently, many research interests have been focused on addressing this problem using fixed diffusion models. These approaches typically directly replace the revealed region of the intermediate or final generated images with that of the reference image or its variants. However, since the unrevealed regions are not directly modified to match the context, it results in incoherence between revealed and unrevealed regions. To address the incoherence problem, a small number of methods introduce a rigorous Bayesian framework, but they tend to introduce mismatches between the generated and the reference images due to the approximation errors in computing the posterior distributions. In this paper, we propose COPAINT, which can coherently inpaint the whole image without introducing mismatches. COPAINT also uses the Bayesian framework to jointly modify both revealed and unrevealed regions, but approximates the posterior distribution in a way that allows the errors to gradually drop to zero throughout the denoising steps, thus strongly penalizing any mismatches with the reference image. Our experiments verify that COPAINT can outperform the existing diffusion-based methods under both objective and subjective metrics. The codes are available at https://github.com/UCSB-NLP-Chang/CoPaint/.

1. Introduction

Fixed diffusion models can impose revealed-region constraints through replacement, but may leave unrevealed pixels mismatched with the context. COPAINT uses a Bayesian approach to modify both regions, with approximation errors designed to vanish by the final denoising step.

1. Introduction

  • Replacement-based methods may leave unrevealed regions unmodified to match context, producing incoherence with revealed regions.Figure 1 illustrates mismatched hair across a portrait from BLENDED and a skin-tone discontinuity from DDRM.Source: §1 · p. 1, Figure 1 · p. 1
  • Bayesian methods jointly modify revealed and unrevealed pixels, but posterior approximation or Monte Carlo errors can prevent strict satisfaction of inpainting constraints.The paper frames resolving coherence while strictly enforcing constraints as an open problem.Source: §1 · pp. 1–2
  • COPAINT approximates intermediate-image posteriors and uses one-step estimates of the final image to make inpainting-error computation tractable during denoising.The approximation errors gradually decrease and can be made zero at the final step.Source: §1 · p. 2
  • 19% average relative LPIPS reduction versus REPAINT, with 31% less computation on ImageNet, accompanies better quality and coherence across CelebA-HQ and ImageNet.The evaluations cover various revealed-region shapes and objective and subjective metrics.Source: §1 · p. 2

2. Related Work

Diffusion inpainting methods are grouped into supervised approaches that train for particular degradation types and unsupervised approaches that use pretrained models without modification. The paper positions COPAINT among the latter and contrasts its joint Bayesian optimization with replacement and resampling approaches.

Related work overview

  • Earlier inpainting methods based on autoencoders, VAEs, GANs, and autoregressive transformers are primarily supervised and require training on specific degradation types.The related-work discussion notes their resource demands and possible poor generalization to unfamiliar scenarios.Source: §2 · p. 2

Supervised diffusion inpainting

  • Supervised approaches train diffusion models for particular degradation types, which can be computationally expensive and may generalize poorly to unseen operators.Examples include PALETTE, GLIDE, LATENT DIFFUSION, CCDF, and predict-and-refine models.Source: §2 · p. 2

Unsupervised diffusion inpainting

  • Unsupervised diffusion inpainting uses pretrained models without modification; blending-based methods are efficient but can produce unreconciled recovered regions.Examples include Song and Ermon’s blending approach, BLENDEDDIFFUSION, and DDRM.Source: §2 · pp. 2–3
  • REPAINT adds time travel through a forward step after blending, while RESAMPLING uses particle filtering to resample images by revealed-region likelihood.Other unsupervised approaches include DEQ-based DDIM inversion, DDNM blending, and DPS posterior-gradient mean shifts.Source: §2 · p. 3
  • COPAINT differs by maximizing a Bayesian posterior at each denoising step to jointly modify revealed and unrevealed image regions.The paper presents this joint modification as a route to better coherence.Source: §2 · p. 3

3. Background and Notations

The background introduces DDIM’s progressively corrupted intermediate variables and its forward and reverse processes. It also explains one-step generation as a compute-efficient estimate that becomes closer to the final generation as denoising proceeds.

Diffusion framework

  • DDIM represents natural images with progressively corrupted intermediate variables and uses forward diffusion to corrupt them and reverse denoising to recover them.The forward process follows a Markov process that adds Gaussian noise; with a suitable schedule and sufficiently large T, X_T approaches a standard Gaussian.Source: §3 · p. 3
  • The reverse process uses an inference distribution q_σ whose conditional distributions for individual intermediate variables match those of the forward process.The passage describes σ_t as a free hyperparameter subject to a stated range.Source: §3 · p. 3

Denoising process

  • The denoising process estimates X_0 from X_t using a neural network trained to predict X_0 by minimizing mean squared error.The denoising process is derived by replacing X_0 with its estimate.Source: §3 · p. 3

One-step generation

  • One-step generation feeds an intermediate estimate to the inference network once, providing a compute-efficient approximation of the final generation.The gap between this estimate and the final generation typically shrinks as t decreases, enabling control over the final output through intermediate variables.Source: §3 · p. 3

4. The COPAINT Algorithm

COPAINT optimizes the diffusion trajectory under the inpainting constraint, then successively corrects approximation errors during denoising. Its optional designs target early-step errors that can affect later generation quality.

4.1. Problem Formulation

  • Inpainting seeks a natural complete image satisfying the revealed-pixel constraint, while replacement-only methods leave unrevealed pixels unadjusted for context, risking incoherence.The paper focuses on a pretrained, fixed diffusion model.Source: §4.1 · pp. 3–4

4.2. A Prototype Approach

  • For deterministic DDIM, the final-image constraint can be translated into a posterior over the initial noise, optimized across the full image with a prior and constraint penalty.The full-image optimization is intended to address incoherence between revealed and unrevealed regions.Source: §4.2 · p. 4

4.3. One-Step Approximation

  • Computing the exact posterior is impractical across tens or hundreds of denoising steps, so COPAINT substitutes a one-step prediction and accounts for its approximation error.The approximation variance must be large enough to capture that error, reducing the weight of the approximate constraint term.Source: §4.3 · pp. 4–5

4.4. Denoising Successive Correction

  • COPAINT optimizes intermediate states to enforce approximate constraints successively; approximation error decreases as t decreases and can reach zero at t = 1 under specified settings.The stated zero-error case sets σ1 = 0 and lets ξ1 approach zero.Source: §4.4 · p. 5

4.5. Additional Algorithmic Designs

  • Because early-step errors can affect later prior quality, optional multi-step approximation and time travel are introduced to reduce error and improve intermediate self-consistency.Multi-step approximation uses multiple deterministic denoising steps at selected times; time travel periodically rewinds and repeats denoising.Source: §4.5 · pp. 5–6

5. Experiments

The experiments compare COPAINT with diffusion-based inpainting baselines on CelebA-HQ and ImageNet using seven masks, objective metrics, human evaluations, and qualitative examples. COPAINT improves inpainting quality and coherence, while ablations examine its design choices and computational trade-offs.

5.1. Experiment Setup

  • Evaluation covers CelebA-HQ and ImageNet-1K, with images cropped to 256 × 256 and seven degradation masks; it reports LPIPS and human judgments assessing overall quality and coherence.The first 100 test images are used for evaluation; each of two human tests samples 50 images per mask in each dataset, producing 1,400 image pairs total.Source: §5.1 · p. 6

5.2. Experiment Results

  • COPAINT reduces average LPIPS beyond REPAINT by 5% on CelebA-HQ and 19% on ImageNet; COPAINT-TT further reduces it by 3% and 1%, respectively.COPAINT-TT achieves the best performance on eleven of fourteen tasks and comparable performance to the best baseline on the rest; human vote differences are positive in most comparisons.Source: Table 1 · p. 7, §5.2 · p. 7
  • COPAINT nearly halves time cost versus REPAINT on both datasets with almost comparable performance, while COPAINT-TT performs best but costs more than most baselines.COPAINT-FAST is four times faster than COPAINT, outperforming all baselines except REPAINT on CelebA-HQ and beating all baselines on ImageNet.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3. Coherence Study

  • On the Half mask example, BLENDED continues generating blond hair despite the revealed black hair, whereas COPAINT produces a coherent black-haired woman.The passage relates BLENDED’s result to a known CelebA-HQ association between women and blond hair.Source: §5.3 · p. 8

5.4. Ablation Study

  • Ablations show that more gradient-descent steps can overfit the revealed region, while one time-travel round is sufficient; multi-step approximation offers minor LPIPS improvement at higher cost.With H = 5, processing one image takes about six times longer than with H = 1; the authors leave efficiency improvements for future work.Source: §5.4 · p. 9, Table 2 · p. 9

6. Conclusion

COPAINT uses a Bayesian framework to modify revealed and unrevealed image regions during denoising, with approximation error designed to decrease toward zero. Experiments report better coherence and overall quality than existing diffusion-based methods, while greedy optimization and one-step approximation remain imperfect.

6. Conclusion

  • Experiments showed that COPAINT outperforms existing diffusion-based methods on coherence and overall quality under objective and subjective metrics.Source: §6 · p. 9
  • COPAINT jointly modifies revealed and unrevealed intermediate regions during denoising, with posterior approximation error designed to decrease toward zero.Source: §6 · p. 9
  • Suboptimal greedy optimization and one-step approximation error remain imperfections; future work will explore more plausible sampling and ways to reduce approximation error.Source: §6 · pp. 9–10

A.1. Human Evaluation

The appendix documents two human-evaluation interfaces and the criteria used to compare inpainted images. One test covers overall image quality, while the other focuses on coherence.

A.1. Human Evaluation

  • The overall test asks evaluators to judge naturalness and artifacts, resemblance to the source, and visual coherence of completed regions.Source: §A.1 · p. 13
  • The coherence test focuses on whether completed regions match the source style and avoid irrelevant objects.Source: §A.1 · p. 13
  • The evaluation interfaces ask users to choose the better of two candidate images according to the stated test criteria.Source: §A.1 · p. 13, Figure 7 · p. 14, Figure 8 · p. 15

A.2. Implementation Details of Baselines

The appendix records baseline implementation settings and explains the adaptive learning rate used by COPAINT. The update objective is framed as posterior optimization, with a learning-rate adjustment to avoid unstable gradients.

A.2. Implementation Details of Baselines

  • Baselines used the released Lugmayr codebase, shared pretrained diffusion models and 250 reverse steps, with method-specific settings for BLENDED, DDRM, RESAMPLING, REPAINT, DPS, and DDNM.Source: §A.2 · p. 16

A.3. Adaptive Learning Rate for Our Method

  • The adaptive-learning-rate appendix frames each update as minimizing a loss equivalent to maximizing the posterior of ˜Xt.Source: §A.3 · p. 16
  • Because large gradients can produce NaNs, the method scales a base learning rate of 0.02 by √¯αt.Source: §A.3 · p. 16

B. Qualitative Results

The appendix expands the qualitative comparisons across CelebA-HQ and ImageNet, showing COPAINT and COPAINT-TT alongside baseline methods across seven masks. It also provides enlarged versions of selected earlier figures.

B. Qualitative Results

C. Additional High-resolution Inpainting Experiments

On 512 × 512 ImageNet inpainting, COPAINT variants improve the LPIPS–computation trade-off over baselines. COPAINT-TT reduces LPIPS relative to REPAINT, while the best-performing variant depends on the time budget.

C. Additional High-resolution Inpainting Experiments

D. Additional Super-resolution Experiments

The super-resolution experiments compare reconstruction methods on CelebA-HQ and ImageNet across three downsampling scales. COPAINT is reported to outperform the baselines quantitatively and to produce more natural-looking detail than DDNM at 8×.

D. Additional Super-resolution Experiments

E. Failure Case Study

The failure study identifies limits in fine-detail inpainting and in cases with inadequate revealed context. The societal-impact discussion notes that model biases may persist and that generated images could be misused.

E. Failure Case Study

  • Large masked regions can produce unnatural images across methods when the revealed surroundings provide inadequate inpainting context.The study also notes that fine details such as hat text may fail to blend, attributing detail issues particularly to diffusion models' difficulty generating text.Source: E. Failure Case Study · p. 28, Figure 18 · p. 29

F. Potential Societal Impacts

  • Bias may persist, particularly when little information is revealed, and generated images could be used for deception or misinformation.The authors advise careful checks for potential risks.Source: F. Potential Societal Impacts · p. 30
Loading 2304.03322v1…