Source-linked AI summary
DiffuEraser: A Diffusion Model for Video Inpainting
Xiaowen Li, Haolan Xue, Peiran Ren, Liefeng Bo
TL;DR
Large masks can leave Transformer-based video inpainting with artifacts, while long-sequence processing can produce temporal inconsistencies across clips. DiffuEraser uses stable diffusion, priors, and expanded temporal receptive fields to address these issues, and it outperforms state-of-the-art methods in content completeness and temporal consistency.
Problem
Large masks expose insufficient Transformer generative capability, while limited temporal receptive fields and constrained information sharing cause inconsistencies during long-sequence inference.
Method
DiffuEraser decomposes video inpainting into known-pixel propagation, unknown-pixel generation, and temporal-consistency maintenance, using stable diffusion and expanded temporal receptive fields.
Results
DiffuEraser outperforms state-of-the-art methods in both content completeness and temporal consistency for video inpainting.
Takeaways & Limitations
The prior-based approach and long-sequence temporal-consistency methods may also apply to object replacement and local stylization.
Takeaways & Limitations
The temporal smoothing approach addresses inherent inconsistencies between the first and last frames.
Abstract
from arXiv · showhide
Recent video inpainting algorithms integrate flow-based pixel propagation with transformer-based generation to leverage optical flow for restoring textures and objects using information from neighboring frames, while completing masked regions through visual Transformers. However, these approaches often encounter blurring and temporal inconsistencies when dealing with large masks, highlighting the need for models with enhanced generative capabilities. Recently, diffusion models have emerged as a prominent technique in image and video generation due to their impressive performance. In this paper, we introduce DiffuEraser, a video inpainting model based on stable diffusion, designed to fill masked regions with greater details and more coherent structures. We incorporate prior information to provide initialization and weak conditioning,which helps mitigate noisy artifacts and suppress hallucinations. Additionally, to improve temporal consistency during long-sequence inference, we expand the temporal receptive fields of both the prior model and DiffuEraser, and further enhance consistency by leveraging the temporal smoothing property of Video Diffusion Models. Experimental results demonstrate that our proposed method outperforms state-of-the-art techniques in both content completeness and temporal consistency while maintaining acceptable efficiency.
TECHNICAL REPORT
Figure 1 compares DiffuEraser with Propainter on texture quality and temporal consistency, with DiffuEraser reported as stronger on both dimensions.
- Texture Quality: DiffuEraser generates more detailed and refined textures than the transformer-based Propainter.
- Temporal Consistency: DiffuEraser demonstrates superior temporal consistency in the inpainted content compared to Propainter.
- Figure 1 presents a performance comparison between DiffuEraser and Propainter.
1. Introduction
Video inpainting combines pixel propagation and content generation, but large masks expose limitations in Transformer-based generation. DiffuEraser addresses these challenges with diffusion-based generation, priors, and expanded temporal modeling.
- Video inpainting completes masked regions with plausible content while maintaining temporal consistency.
- Flow-based methods restore textures and objects from adjacent frames, whereas Transformer-based methods complete object structure.
- Mainstream systems combine flow completion, feature propagation, and content generation to handle known and unknown pixels.
- Propainter propagates known pixels effectively, but large masks cause significant artifacts because its Transformer generation is insufficient.
- DiffuEraser introduces stable-diffusion-based generation to overcome blurring and mosaic artifacts and complete object structures with more detailed content.
- Injected priors provide easier initialization, mitigate noisy artifacts, and weakly condition generation to suppress unwanted objects.
- The model expands temporal receptive fields and uses Video Diffusion Model smoothing to improve continuity during long-sequence inference.
2. Related Works
Related work spans diffusion models, optical-flow propagation, Transformers, and emerging diffusion-based video inpainting. DiffuEraser’s architecture combines stable diffusion with BrushNet and temporal attention.
- Diffusion Models: Diffusion models have improved image and video generation across synthesis, editing, personalization, and inpainting tasks.
- Video Inpainting: Video inpainting methods use optical flow to restore textures and details, while Transformer architectures complete structural aspects of objects.
- Diffusion-Based Video Inpainting: Emerging diffusion-based video inpainting methods enhance detail and structural integrity but commonly face diffusion-model hallucinations.
- DiffuEraser Architecture: DiffuEraser uses a denoising UNet and BrushNet, integrating masked-image features into the UNet after zero convolution blocks.
- DiffuEraser Architecture: Temporal attention follows self-attention and cross-attention to improve temporal consistency.
3. Methodology
DiffuEraser combines a BrushNet-assisted diffusion inpainting architecture with priors and temporal-receptive-field expansions to generate detailed content while improving long-sequence consistency.
- Network Overview: DiffuEraser integrates an auxiliary BrushNet with a denoising UNet and temporal attention for video inpainting.BrushNet processes masked images, masks, and noisy latents, while temporal attention follows self-attention and cross-attention layers.
- Problem Decomposition: The task is decomposed into propagating known pixels, generating unknown pixels, and maintaining temporal consistency.Known pixels have appeared in masked frames, whereas unknown pixels have never appeared in any masked frames.
- Temporal Consistency: Pre-inference expands DiffuEraser’s temporal context by processing sampled frames together, then guiding frame-by-frame inference across the remaining video.The approach targets discrepancies at clip boundaries during long-sequence processing.
- Incorporation of Priors: DDIM-inverted outputs from a lightweight prior model are added to the noisy latent to initialize generation and suppress artifacts or unwanted objects.The prior provides initialization information and weak conditioning against noisy artifacts and hallucinations.
- Temporal Consistency: Temporal smoothing and receptive-field expansion make clip transitions more gradual, but complete consistency remains unattainable because the first and last frames can be inherently inconsistent.The optimizations smooth multiple intersection adjustments into a single gradual change across the video.
1. Enhancing Priors for Comprehensive Pixel Propagation
The prior optimization extends known-pixel propagation across the full video before DiffuEraser inference, improving the stability and accuracy of the supplied prior.
- Enhancing Priors for Comprehensive Pixel Propagation: Pre-propagation samples input frames and extends known pixels across the entire time domain beyond a single propagation pass.Full propagation keeps completed content consistent with unmasked regions.
- Enhancing Priors for Comprehensive Pixel Propagation: Inpainting results from sampled frames then guide frame-by-frame propagation so pre-propagation information reaches every frame.This integrates the sampled-frame results throughout the sequence.
- Enhancing Priors for Comprehensive Pixel Propagation: The optimization uses distant-frame information more effectively, producing a more accurate and stable prior for DiffuEraser.DiffuEraser can still change noticeably at clip intersections because its own temporal receptive field remains limited.
2. Expanding the Temporal Receptive Field of DiffuEraser for consistent generation of unknown pixels
DiffuEraser expands its temporal receptive field through pre-inference and guided frame-wise inference, addressing inconsistencies between clips during long-sequence generation.
- Expanding the Temporal Receptive Field of DiffuEraser: Pre-inference processes sampled frames as one inference pass, broadening temporal context for consistent generation across the video.This extends the model beyond individual clip-level context.
- Expanding the Temporal Receptive Field of DiffuEraser: The pre-inference results guide frame-by-frame inference, maintaining established content consistency across the remaining frames.The resulting consistency is preserved throughout the sequence.
- Expanding the Temporal Receptive Field of DiffuEraser: Jointly expanding the prior model’s and DiffuEraser’s receptive fields targets long-sequence temporal issues by propagating known pixels and stabilizing unknown-pixel generation.The prior optimization emphasizes correctness, while DiffuEraser optimization emphasizes consistent generation.
4. Experiments
Experiments use large-scale Panda-70M-derived clips, two-stage training, and efficient two-step sampling; qualitative comparisons report detailed and stable inpainting.
- Datasets: The dataset contains 3,183,727 short video clips with captions, filtered from Panda-70M and paired with randomly generated mask sequences.Masks vary in rates, directions, and shapes for video inpainting and object-removal tasks.
- Training Details and Metrics: Training uses two stages: content-generation training for BrushNet and the denoising UNet, followed by motion-module training for temporal consistency.Both stages use resolution 512, with separate GPU and step configurations.
- Efficiency: PCM enables two-step sample generation, and a 10-second 540p video at 25 FPS takes about 200 seconds on an Nvidia L20.This provides the reported efficiency reference.
- Qualitative Comparison: Compared with Propainter, the model propagates known pixels across frames and generates unknown pixels with high consistency and stability.The qualitative comparison covers texture quality and temporal consistency.
5. Conclusion and Discussion
DiffuEraser decomposes video inpainting into pixel propagation, unknown-pixel generation, and temporal consistency, addressing each with tailored mechanisms. The method uses stable diffusion and priors for detailed generation, consistent propagation, and improved long-sequence coherence.
- DiffuEraser addresses video inpainting through three sub-problems: known-pixel propagation, unknown-pixel generation, and temporal consistency.Each sub-problem receives a tailored solution.
- Stable diffusion improves unknown-pixel generation, while priors mitigate hallucinations and support more accurate, realistic inpainting.The generative model is intended to overcome blurring and mosaic artifacts associated with Transformer-based models.
- The denoising UNet motion module and priors propagate known pixels consistently across frames, reducing conflicts with unmasked regions.This improves the correctness and stability of completed results.
- Expanded temporal receptive fields improve consistency across long sequences, while Video Diffusion Model smoothing enhances continuity at clip intersections.Both the prior model and DiffuEraser use expanded temporal receptive fields.
- The authors report superior content completeness and temporal consistency versus state-of-the-art methods, with potential applicability to other video editing tasks.Examples include object replacement and local stylization, which the paper identifies for future exploration.