PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffuEraser: A Diffusion Model for Video Inpainting

Xiaowen Li, Haolan Xue, Peiran Ren, Liefeng Bo

arXiv:2501.10018v1cs.CV

TL;DR

Existing video-inpainting approaches have limited generative capability for large masked regions, while long-sequence inference faces temporal-consistency challenges. DiffuEraser combines stable-diffusion inpainting with priors and temporal-consistency strategies, and the authors report that it outperforms state-of-the-art methods in content completeness and temporal consistency.

  • Problem

    Existing Transformer-based methods have limited generative capability for large masks, and long-sequence inference has restricted temporal receptive fields and constrained information sharing across clips.

    Source: §1 · p. 2, §3.3.2 · p. 5

  • Method

    DiffuEraser uses stable-diffusion video inpainting with priors, expanded temporal receptive fields, and temporal smoothing to address generation and temporal consistency.

    Source: §1 · p. 2, §5 · p. 7

  • Results

    DiffuEraser outperforms state-of-the-art methods in content completeness and temporal consistency.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors say the approaches to incorporating priors and improving long-sequence temporal consistency may also apply to object replacement and local stylization, which they leave for future work.

    Source: §5 · p. 7

    The paper focuses on illustrating the advantages of incorporating priors and optimizing temporal consistency across clips during long-sequence inference.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Recent video inpainting algorithms integrate flow-based pixel propagation with transformer-based generation to leverage optical flow for restoring textures and objects using information from neighboring frames, while completing masked regions through visual Transformers. However, these approaches often encounter blurring and temporal inconsistencies when dealing with large masks, highlighting the need for models with enhanced generative capabilities. Recently, diffusion models have emerged as a prominent technique in image and video generation due to their impressive performance. In this paper, we introduce DiffuEraser, a video inpainting model based on stable diffusion, designed to fill masked regions with greater details and more coherent structures. We incorporate prior information to provide initialization and weak conditioning,which helps mitigate noisy artifacts and suppress hallucinations. Additionally, to improve temporal consistency during long-sequence inference, we expand the temporal receptive fields of both the prior model and DiffuEraser, and further enhance consistency by leveraging the temporal smoothing property of Video Diffusion Models. Experimental results demonstrate that our proposed method outperforms state-of-the-art techniques in both content completeness and temporal consistency while maintaining acceptable efficiency.

1. Introduction

The introduction frames video inpainting as completing masked regions plausibly and consistently, and identifies insufficient generation under large masks as a key limitation. It presents a diffusion-based approach with injected priors and expanded temporal context to address generation quality and long-sequence consistency.

1. Introduction

  • Video inpainting seeks plausible, temporally consistent completions; flow methods propagate texture and objects using optical flow, while Transformer methods complete object structure.Source: §1 · pp. 1–2
  • Mainstream methods combine flow completion, feature propagation, and content generation to propagate known pixels across frames and generate unknown pixels.Known pixels have appeared in masked frames and can be propagated; unknown pixels have never appeared in masked frames and must be generated.Source: §1 · p. 2
  • Propainter [46] combines recurrent flow completion, dual-domain propagation, and a mask-guided sparse Transformer, but large masks can produce significant artifacts because its generation is insufficient.The approach effectively propagates known pixels and shows an initial ability to generate unknown pixels.Source: §1 · p. 2
  • Diffusion models’ generative capability addresses the artifacts and insufficient structural completion that Transformer-based methods exhibit under large masks.The authors introduce a motion module for the diffusion-based image inpainting model BrushNet to generate object structures and more detailed content.Source: §1 · p. 2
  • Injected priors support easier initialization to mitigate noisy artifacts and act as weak conditioning to suppress unwanted-object generation.Source: §1 · p. 2
  • Expanding the prior and diffusion models’ temporal receptive fields, together with video diffusion’s temporal smoothing, improves consistency in long-sequence inference and continuity between clips.Source: §1 · p. 2

2. Related Works

Related work progresses from flow- and Transformer-based video inpainting to diffusion-based generation, which can improve detail and structure but still faces hallucinations and long-sequence consistency challenges.

Diffusion Models

  • Diffusion models [14] [34] have advanced image and video generation across synthesis, editing, inpainting, and controllable-generation tasks.Video diffusion models often incorporate motion modules.Source: §2 · p. 2

Video Inpainting

  • Flow propagation [7] [41] restores textures, while Transformers [40] [20] [18] [46] complete object structure; Propainter [46] struggles with large masks.Propainter combines recurrent flow completion, dual-domain propagation, and a mask-guided sparse Transformer, but its limited generative capacity causes noticeable artifacts on large masks.Source: §2 · p. 2
  • Diffusion-based video inpainting [37] [30] [9] [45] [47] uses generative capabilities to improve detail and structural integrity, though hallucinations remain a concern.BIVDiff [30] bridges image and video diffusion models; CoCoCo [47] integrates a motion module into a text-to-image model for text-guided inpainting.Source: §2 · pp. 2–3
  • Existing methods still fall short on long-sequence consistency and stability; DiffuEraser targets consistent, detailed, structurally complete inpainting without text prompts.FloED [9] focuses on object removal and background restoration, while FFF-VDI [17] propagates future-frame noise latents to improve consistency and suppress hallucinations.Source: §2 · p. 3

3. Methodology

DiffuEraser combines motion-aware latent inpainting with prior-guided initialization and long-video consistency methods that expand temporal context and smooth clip joins.

3.1. Network Overview

  • DiffuEraser combines an AnimateDiff [11] motion module with BrushNet and Stable Diffusion to propagate known pixels, generate unseen content, and maintain temporal consistency.BrushNet adds a masked-image feature branch to the denoising UNet; temporal attention follows self- and cross-attention, and blurred masks blend generated frames with the inputs.Source: §3.1 · pp. 3–4

3.3.2 Expanding the Temporal Receptive Field

  • To expand temporal receptive fields, pre-inference processes optimally sampled frames together as one clip, then guides frame-by-frame inference with information from distant frames.This addresses the limited context and constrained cross-clip sharing during long-video inference.Source: §3.1 · p. 4, §3.3.2 · p. 5

3.2. Incorporation of Priors

  • DDIM Inversion [33] adds inverted outputs from a lightweight prior model to the noisy latent, providing initialization and weak conditioning that suppress artifacts and unwanted objects.The selected prior is Propainter; its blur and mosaic artifacts are refined into richer textures and detail, while priors improve inpainting accuracy and stability.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Optimizing Temporal Consistency for Long-Sequence Inference

  • Clipwise inference handles 22 frames in this setting, limiting distant-pixel propagation and information sharing and causing content discrepancies at long-video clip boundaries.The method targets those boundary discrepancies to improve temporal consistency across the video.Source: §3.3 · p. 5, §3.3.2 · p. 5

3.3.1 Leveraging the Temporal Smoothing Property of the Video Diffusion Model (VDM)

  • Staggered denoising infers even timesteps from the clip start and odd timesteps from its midpoint, using VDM temporal smoothing to blend clip intersections more seamlessly.The resulting transitions form one gradual change across the video, but inconsistencies between its first and last frames prevent complete overall consistency.Source: §3.3.1 · p. 5, Figure 6 · p. 5

1. Enhancing Priors for Comprehensive Pixel Propagation

DiffuEraser extends Propainter’s pixel propagation across the full video by pre-propagating sampled frames and using their inpainting results to guide frame-by-frame propagation. This provides a more accurate, stable prior, though DiffuEraser’s limited temporal receptive field still causes changes at clip intersections.

1. Enhancing Priors for Comprehensive Pixel Propagation

  • Pre-propagation extends known pixels across the video’s full temporal domain, beyond the dozens of frames typically handled by a single pass, stabilizing completed content against unmasked regions.The section illustrates this temporal-receptive-field expansion using Propainter.Source: §1 · p. 5, Figure 8 · p. 6
  • Inpainting results from sampled frames guide frame-by-frame propagation, integrating pre-propagation information into every frame.Source: §1 · p. 5
  • The optimization helps Propainter use distant-frame information more effectively and provides DiffuEraser a more accurate, stable prior, but its limited temporal receptive field still causes significant changes at clip intersections.Source: §1 · p. 6

2. Expanding the Temporal Receptive Field of DiffuEraser for consistent generation of unknown pixels

DiffuEraser broadens temporal context through pre-inference over sampled video frames, then uses those results to maintain consistent generation of unknown pixels during frame-by-frame inference.

2. Expanding the Temporal Receptive Field of DiffuEraser for consistent generation of unknown pixels

  • Pre-inference processes sampled video frames in a single pass, broadening temporal context to support consistent content generation across the video.This expands DiffuEraser’s temporal receptive field beyond individual clips.Source: §2 · p. 6
  • Pre-inference results guide frame-by-frame inference, maintaining established content consistency throughout the remaining frames.Figure 9 illustrates that consistency is maintained after pre-inference.Source: §2 · p. 6, Figure 9 · p. 6
  • Extending the receptive field across the entire video lets prior optimization propagate known pixels correctly, while DiffuEraser optimization stabilizes generation of unknown pixels.Together, these enhancements address temporal consistency issues in long-sequence inference.Source: §2 · p. 6

4. Experiments

Experiments show that DiffuEraser propagates known pixels across frames and generates never-observed pixels with high consistency and stability; PCM enables two-step sampling.

Datasets

  • Training uses 3,183,727 captioned short clips from Panda-70M [2], with random-rate, direction, and shape masks simulating inpainting and object removal.Source: §4 · p. 6

Training Details and Metrics

  • The two-stage 512-resolution training first optimizes BrushNet and the denoising UNet without motion, then trains the motion module on 22-frame sequences.Stage one runs for 100,000 steps on 4 NVIDIA A100 GPUs with batch size 16; stage two runs for 80,000 steps on 8 A100 GPUs with batch size 1. Both use L2 loss and a learning rate of 1e-5.Source: §4 · p. 6

Efficiency

  • 2-step sampling with Phased Consistency Models (PCM) [35] processes a 10-second, 540p, 25 FPS video in about 200 seconds on an Nvidia GPU L20.Source: §4 · p. 6

Qualitative Comparison

  • DiffuEraser effectively propagates known pixels across frames and generates never-observed pixels with high consistency and stability, compared qualitatively with ProPainter in texture quality and temporal consistency.The comparison is illustrated in Figure 1, with further results in Figures 10–13.Source: §4 · pp. 6–7

5. Conclusion and Discussion

DiffuEraser uses stable diffusion and tailored solutions for pixel propagation, unknown-pixel generation, and temporal consistency. The paper reports improved content completeness and temporal consistency over state-of-the-art methods.

5. Conclusion and Discussion

  • DiffuEraser decomposes video inpainting into propagation of known pixels, generation of unknown pixels, and maintaining temporal consistency, with tailored solutions for each.Source: §5 · p. 7
  • Stable diffusion helps DiffuEraser avoid blurring and mosaic issues in unknown-pixel generation, while priors mitigate hallucinations for more accurate, realistic inpainting.Source: §5 · p. 7
  • The denoising UNet motion module and priors support sufficient, consistent known-pixel propagation across frames, preventing conflicts with unmasked regions and improving correctness and stability.Source: §5 · p. 7
  • For long-sequence inference, expanding the temporal receptive fields of the prior model and DiffuEraser improves consistency across frames, while VDM temporal smoothing enhances coherence at clip intersections.Source: §5 · p. 7
  • DiffuEraser outperforms state-of-the-art methods in content completeness and temporal consistency for video inpainting.Source: §5 · p. 7
  • The paper notes that priors and long-sequence consistency methods may apply to object replacement and local stylization, which it leaves for future work.Source: §5 · p. 7

Related papers

Loading 2501.10018v1…