PaperlayerView exact revision on arXiv

Source-linked AI summary

ProPainter: Improving Propagation and Transformer for Video Inpainting

Shangchen Zhou, Chongyi Li, Kelvin C. K. Chan, Chen Change Loy

arXiv:2309.03897v1cs.CV

TL;DR

Video inpainting needs reliable correspondence across distant frames, but separate image or feature propagation can misalign content, while Transformer memory costs restrict temporal range. ProPainter combines dual-domain propagation with a mask-guided sparse video Transformer, and it reports a 1.46 dB PSNR advantage over prior methods while reducing memory consumption.

  • Problem

    Separate image- or feature-domain propagation can suffer from flow-related misalignment, while memory and computation constraints limit Transformer-based access to distant-frame correspondences.

  • Method

    ProPainter combines jointly trained dual-domain propagation with a mask-guided sparse video Transformer that removes unnecessary and redundant windows.

  • Results

    1.46 dB in PSNR is the reported margin over prior state-of-the-art methods, alongside significantly reduced memory consumption.

  • Takeaways & Limitations

    ProPainter provides reliable long-distance propagation and high-efficiency video inpainting through its two redesigned modules.

Abstract

from arXiv · show

Flow-based propagation and spatiotemporal Transformer are two mainstream mechanisms in video inpainting (VI). Despite the effectiveness of these components, they still suffer from some limitations that affect their performance. Previous propagation-based approaches are performed separately either in the image or feature domain. Global image propagation isolated from learning may cause spatial misalignment due to inaccurate optical flow. Moreover, memory or computational constraints limit the temporal range of feature propagation and video Transformer, preventing exploration of correspondence information from distant frames. To address these issues, we propose an improved framework, called ProPainter, which involves enhanced ProPagation and an efficient Transformer. Specifically, we introduce dual-domain propagation that combines the advantages of image and feature warping, exploiting global correspondences reliably. We also propose a mask-guided sparse video Transformer, which achieves high efficiency by discarding unnecessary and redundant tokens. With these components, ProPainter outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining appealing efficiency.

1. Introduction

ProPainter addresses video inpainting’s correspondence, alignment, and efficiency challenges through dual-domain propagation and a mask-guided sparse Transformer. The framework combines global and local information while reducing unnecessary computation and memory use.

  • Motivation: Video inpainting must fill missing regions while preserving spatial and temporal coherence, especially across distant frames.Its applications include completion, object removal, restoration, and watermark or logo removal.
  • Propagation challenges: Image propagation can create texture misalignment from inaccurate flow, while feature propagation improves integration but does not alone resolve propagation limitations.Image propagation may require a separate network to hallucinate remaining regions after filling most holes.
  • ProPainter approach: ProPainter combines image and feature propagation in a jointly trained dual-domain framework with GPU-based reliability checks and improved deformable feature alignment.The design uses global image propagation for long-range correspondence and feature propagation for learned refinement.
  • Efficiency: The flow-completion network is over 40 times faster than the state-of-the-art method, reaching approximately 192 fps while maintaining comparable performance.The footnote specifies testing on a single NVIDIA Tesla V100 GPU with 32G memory.
  • Transformer design: The mask-guided sparse Transformer discards unnecessary query windows and redundant key/value windows, reducing complexity and memory without compromising inpainting performance.It exploits the small spatial extent of masks and texture redundancy in adjacent frames.
  • Reported outcome: 1.46 dB in PSNR separates ProPainter from prior state-of-the-art methods while the model significantly reduces memory consumption.The paper presents this as its principal effectiveness-and-efficiency result.

2. Related Work

Prior video-inpainting methods use convolutional, internal-learning, and flow-based strategies to aggregate temporal information, but limited receptive fields, incomplete flow, and offline propagation constrain distant-frame correspondence. Transformer combinations improve aggregation yet remain memory-intensive, motivating more efficient long-range designs.

  • 3D convolution: 3D CNN and temporal-shift methods have limited temporal and spatial receptive fields, reducing their ability to exploit distant content.They can also suffer from misalignment between adjacent frames.
  • Internal learning: Internal-learning approaches align neighboring reference frames to improve temporal coherence, but incomplete optical flow may provide invalid propagation for missing regions.
  • Flow-based propagation: Flow-based methods first complete the flow field and then use it to guide pixel-domain propagation, simplifying RGB inpainting but keeping propagation independent of later learning.The related-work discussion identifies this as offline propagation.
  • Transformer-based methods: Combining flow-guided propagation with Transformers is effective, but Transformer memory requirements limit propagation range during training and inference.This limitation hinders the use of temporally distant content.

3. Methodology

ProPainter combines recurrent flow completion, dual-domain propagation, and a mask-guided sparse Transformer to propagate and refine video content efficiently. Completed flows guide global image and local feature propagation, while sparse attention reduces unnecessary computation and memory use.

  • 3. Methodology: ProPainter comprises recurrent flow completion, dual-domain propagation, and mask-guided sparse Transformer components.The completed flows guide subsequent propagation, and Transformer blocks refine the propagated features.
  • 3.1. Recurrent Flow Completion: A recurrent flow-completion network completes corrupted forward and backward optical flows for subsequent propagation.The recurrent design aggregates flow information from nearby frames while avoiding repeated sliding-window inference.
  • 3.2. Dual-domain Propagation: Dual-domain propagation performs global image warping and local feature propagation sequentially, using distinct alignment strategies in both temporal directions.Image propagation fills reliable regions, while feature propagation uses flow-guided deformable alignment to refine challenging regions.
  • 3.2. Dual-domain Propagation: Image propagation transfers only reliable pixels into masked regions using forward-backward flow consistency and mask-based validity checks.The consistency threshold is set to 5, and propagation updates the current mask as filled regions become unmasked.
  • 3.2. Dual-domain Propagation: Flow-guided deformable alignment uses completed flow as a base offset, learns residual offsets, and incorporates mask-aware conditions before feature fusion.This design focuses feature propagation on regions where flow is invalid or image propagation is unreliable.
  • 3.3. Mask-Guided Sparse Video Transformer: The mask-guided sparse Transformer skips query windows without masked regions and filters redundant key/value windows before spatiotemporal attention.Window expansion and global tokens enlarge spatial interrelation while preserving the efficiency gains of sparse attention.

4. Experiments

ProPainter is evaluated on standard video-inpainting datasets using reconstruction, perceptual, and temporal-consistency metrics, alongside qualitative, flow-completion, efficiency, and ablation studies. Across these evaluations, it achieves strong performance while improving flow-completion speed and benefiting from dual-domain propagation and sparse Transformer design.

  • Datasets: ProPainter is trained on 3,471 YouTube-VOS videos and evaluated on YouTube-VOS and DAVIS test sets.The test sets contain 508 and 90 sequences, respectively; 50 DAVIS clips are used for evaluation.
  • Metrics: The evaluation reports PSNR, SSIM, VFID, and flow warping error to measure reconstruction, perceptual similarity, and temporal consistency.
  • Qualitative Evaluation: Qualitative comparisons on video completion and object removal show more complete, faithful textures and stronger spatiotemporal coherence for ProPainter.
  • Quantitative Evaluation: ProPainter surpasses the state-of-the-art method by 1.14 dB in PSNR on DAVIS and outperforms competing methods across all reported quantitative metrics.Inference uses a temporal length of 20.
  • Flow Completion Comparisons: The recurrent flow-completion network is approximately 40 times faster than previous methods while maintaining comparable flow-completion accuracy.The comparison uses end-point error and running time on YouTube-VOS and DAVIS.
  • Ablation Study: Removing image propagation causes a significant performance drop and reduces the model’s ability to complete missing content with details.
  • Ablation Study: Reliability checking avoids severe texture distortion associated with incorrect optical flow, while dual-domain feature propagation further complements image-domain propagation.Replacing the image propagation module with FGVC decreases PSNR, and removing or replacing feature propagation also reduces performance.
  • Efficiency of Sparse Transformer: The sparse Transformer’s efficiency advantage grows with temporal length and video resolution, supporting longer-range attention and larger-resolution videos.FLOPs are compared using a missing-region ratio of 1/6.

5. Conclusion

ProPainter combines enhanced dual-domain propagation with a mask-guided sparse video Transformer to propagate information over long distances while maintaining efficiency.

  • ProPainter integrates enhanced dual-domain propagation and a mask-guided sparse video Transformer.These two modules provide the framework’s core propagation and efficiency improvements.

– Supplementary Materials –

The supplementary materials provide architectural and training details, additional performance analyses, quantitative evaluations, and visual comparisons.

  • The supplement details ProPainter’s architecture and loss functions.
  • It analyzes the sources of performance improvement and discusses the method’s advantages.
  • It provides additional quantitative evaluations and visual comparisons.

A.1. Architecture

The supplementary architecture section describes deformable alignment and flow-completion training, including reconstruction, smoothness, adversarial, and combined objectives.

  • Architecture: The recurrent flow completion network and feature propagation use distinct deformable alignment modules.The modules differ in their condition pools and alignment guidance.
  • Loss Functions: The RFC reconstruction loss is applied to both valid and invalid regions.It measures the L1 difference between reconstructed and target flow outputs.
  • Loss Functions: The RFC flow objective combines flow reconstruction and second-order smooth losses.The smoothness term encourages smooth and coherent completed flow fields for subsequent propagation.
  • Loss Functions: ProPainter additionally uses adversarial training with a T-PatchGAN discriminator to improve generated-video quality and coherence.The discriminator differentiates real and reconstructed videos, while the generator uses a GAN loss.
  • Loss Functions: The ProPainter learning objective weights reconstruction loss by 1 and generator GAN loss by 0.01.The combined objective is Linpaint = λ1Lrec + λ2LG.
  • Performance Analysis: ProPainter outperforms state-of-the-art networks by a large margin on all reported quantitative metrics, especially on DAVIS.

B.1. Factor Behind Improved Performance

Global image propagation fills most masked regions, reducing subsequent learning difficulty, while motion characteristics explain stronger gains on DAVIS than YouTube-VOS.

  • Global Image Propagation: Global image propagation fills the majority of masks and can entirely complete some masked regions.Later modules refine and complement these contents rather than learning the entire inpainting process.
  • Motion Distribution: ProPainter’s performance improvement is more noticeable on DAVIS than on YouTube-VOS.
  • Motion Distribution: YouTube-VOS contains many nearly stationary scenes and a greater proportion of regions with small motion.These motion characteristics limit the effectiveness of dual-domain propagation, which relies on motion flow fields.

C.1. Quantitative Evaluation on 480p Videos

The evaluation contrasts ProPainter with prior methods on high-resolution DAVIS videos and reports qualitative comparisons across flow completion and video completion datasets. Additional material documents an interactive demonstration and object-removal results.

  • Quantitative Evaluation on 480p Videos: 480p DAVIS evaluation compares accuracy and efficiency, with runtimes measured on an NVIDIA Tesla V100 (32G) GPU.Only STTN and E2FGVI are included because other methods exceed memory limits or require excessively long inference time.
  • Qualitative Comparisons on Flow Completion: ProPainter’s recurrent flow completion network produces more complete and accurate flow fields than FGVC, FGT, and ISVI in visual comparisons.The resulting flows support more reliable and precise downstream dual-domain propagation.
  • Qualitative Comparisons on Video Completion: Figures 13 and 14 compare video completion on YouTube-VOS and DAVIS, respectively, against FuseFormer, FGT, and E2FGVI.The comparisons emphasize complete and faithful textures and enhanced spatiotemporal coherence.
  • Interactive Demo: An interactive ProPainter demo showcases object removal and lets users select specific objects for removal.The demo incorporates a video instance segmentation network.
Loading 2309.03897v1…