Source-linked AI summary
Towards An End-to-End Framework for Flow-Guided Video Inpainting
Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, Ming-Ming Cheng
TL;DR
Video inpainting must preserve both spatial structure and temporal coherence, while prior flow-based pipelines rely on separate hand-crafted stages and intermediate results. E2FGVI jointly trains flow completion, feature propagation, and content hallucination, achieving state-of-the-art performance with promising efficiency.
Problem
Video inpainting requires spatial structure and temporal coherence, but prior flow-based methods use separate hand-crafted processes and depend heavily on intermediate results.
Method
E2FGVI jointly optimizes trainable flow completion, feature propagation, and content hallucination modules corresponding to stages in previous flow-based methods.
Results
E2FGVI achieves state-of-the-art quantitative and qualitative performance on two benchmark datasets, improving PSNR, SSIM, VFID, and Ewarp while processing 432 × 240 videos at 0.12 seconds per frame.
Takeaways & Limitations
The end-to-end framework addresses bottlenecks of previous flow-based methods and offers efficient inference with low computational complexity.
Takeaways & Limitations
Large motion or substantial missing object details across frames can produce implausible content and artifacts in masked regions.
Abstract
from arXiv · showhide
Optical flow, which captures motion information across frames, is exploited in recent video inpainting methods through propagating pixels along its trajectories. However, the hand-crafted flow-based processes in these methods are applied separately to form the whole inpainting pipeline. Thus, these methods are less efficient and rely heavily on the intermediate results from earlier stages. In this paper, we propose an End-to-End framework for Flow-Guided Video Inpainting (E$^2$FGVI) through elaborately designed three trainable modules, namely, flow completion, feature propagation, and content hallucination modules. The three modules correspond with the three stages of previous flow-based methods but can be jointly optimized, leading to a more efficient and effective inpainting process. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods both qualitatively and quantitatively and shows promising efficiency. The code is available at https://github.com/MCG-NKU/E2FGVI.
1. Introduction
Video inpainting must preserve both spatial structure and temporal coherence, but prior flow-based pipelines separate hand-crafted stages, accumulating errors and reducing efficiency. E2FGVI replaces these stages with jointly optimized trainable modules and achieves strong accuracy and efficiency.
- Video inpainting must fill corrupted regions with plausible, coherent content while preserving spatial structure and temporal coherence.Independent frame-wise image inpainting tends to produce temporal inconsistency and severe artifacts.
- Prior flow-based methods separately complete flow, propagate pixels, and hallucinate remaining missing regions.These stages use completed optical flow for bidirectional pixel propagation before applying a pretrained image inpainting network.
- Separate hand-crafted stages accumulate earlier errors, while non-GPU operations make inference time-consuming.Inaccurate flow can mislead propagation and content hallucination, producing unfaithful results.
- E2FGVI jointly optimizes trainable flow completion, feature propagation, and content hallucination modules in an end-to-end framework.The modules simulate prior stages while reducing dependence on intermediate results; feature propagation operates in feature space with deformable convolution.
- E2FGVI achieves significant improvements over previous state-of-the-art methods on PSNR, SSIM, VFID, and Ewarp.These metrics cover distortion, perceptual quality, and temporal consistency.
2. Related Work
Video inpainting methods include 3D convolution-based, flow-based, and attention-based approaches. Flow-based methods use optical flow to preserve temporal coherence, while attention mechanisms improve interactions and computational efficiency.
- Video inpainting methods are roughly divided into 3D convolution-based, flow-based, and attention-based classes.
- 3D convolution and attention methods can yield temporally inconsistent results because of limited temporal receptive fields.
- Flow-based methods treat video inpainting as pixel propagation and use optical flow as a strong temporal-coherence prior.
- Swin Transformer strengthens local connections through shifted windows, while Focal Transformer enhances global-local interactions with focal self-attention.
3. Method
E2FGVI uses three differentiable modules—flow completion, feature propagation, and content hallucination—in an end-to-end trainable architecture. It completes flows, propagates information in feature space, and combines local and non-local features for video inpainting.
- The architecture contains differentiable flow completion, feature propagation, and content hallucination modules trained jointly end to end.These modules simulate the corresponding stages of earlier flow-based methods while reducing dependence on independently produced intermediate results.
- Flow completion: Flow completion estimates bidirectional optical flow between adjacent frames from downsampled corrupted frames and restores it using an L1 loss.The model uses a flow estimation network on quarter-resolution frames, with ground-truth flows computed from uncorrupted videos for supervision.
- Feature propagation: Feature propagation warps neighboring backward features with completed forward flow, merges them with current features, and performs the process bidirectionally.Propagation occurs in feature space rather than image space, using convolutional operations and spatial warping.
- Feature propagation: Modulated deformable convolution learns sampling offsets and validity weights to reduce the effect of inaccurate flow during feature propagation.Candidate feature points are generated around completed flow, then weighted and sampled by a deformable convolutional layer.
- Feature propagation: A learnable 1 × 1 convolution adaptively fuses forward and backward propagation features instead of using a predefined bidirectional combination rule.This fusion replaces the fixed rule used to combine bidirectionally flow-traced pixels in prior work.
4. Experiments
Experiments on YouTube-VOS and DAVIS show that E2FGVI achieves strong quantitative, qualitative, user-preferred, and efficiency results, while ablations validate its modules and attention design.
- 4.2. Comparison: On YouTube-VOS and DAVIS, E2FGVI substantially surpasses previous state-of-the-art methods on all four quantitative metrics.The metrics assess distortion, visual plausibility, and spatial-temporal coherence.
- 4.2. Comparison: Volunteers favored E2FGVI over almost all compared methods, while it still received a majority of votes against FGVC.The study included 20 participants who evaluated randomly sampled video triplets.
- 4.3. Ablations: Removing feature propagation causes severe artifacts, whereas flow-based propagation raises PSNR by 0.42dB and produces more faithful content.Flow-based propagation cannot recover content that optical flow cannot trace.
- 4.4. Limitation: Large motion or extensive missing object details remain failure conditions, producing implausible content and severe artifacts in masked regions.These cases remain challenging for video inpainting methods.
5. Conclusion
E2FGVI is an end-to-end trainable flow-based video inpainting model whose three coordinated modules achieve state-of-the-art performance and efficient inference.
- E2FGVI jointly coordinates flow completion, feature propagation, and content hallucination modules in an end-to-end trainable framework.
- The method achieves state-of-the-art quantitative and qualitative performance on two benchmark datasets.
- E2FGVI is efficient in inference time and computational complexity and is proposed as a strong baseline for future work.
A. Architecture and Training Details
The implementation adopts established encoder-decoder and discriminator components, uses SPyNet for efficient flow completion, and trains at resized video resolution with local and non-local temporal sampling.
- Architecture: The encoder and decoder share FuseFormer's architecture with channel dimension C set to 128, while T-PatchGAN follows previous implementations.
- Architecture: SPyNet serves as the flow completion module for computational efficiency and is initialized with pretrained weights.
- Training details: All frames are resized to 432 × 240 for training, evaluation, and testing to account for GPU memory limitations.
- Training details: Training uses 5 continuous local frames and 3 randomly sampled non-local frames.
B.1. Completing flows in a offline manner.
Offline flow completion produces a slightly higher PSNR than the end-to-end setting, but the end-to-end method is substantially faster; feature propagation benefits from flow-guided deformable convolution.
- B.1. Completing flows in a offline manner.: 32.38 vs. 32.35 dB PSNR favors offline FGVC flow completion slightly over the end-to-end setting.
- B.1. Completing flows in a offline manner.: 1.21 vs. 0.16 s/frame inference speed favors the end-to-end setting over offline flow completion.
- B.2. Taking a deeper look to flow-guided feature propagation module: Deformable convolution-based warping generates smoother content than flow-based warping because it uses more sampling feature points.
- B.2. Taking a deeper look to flow-guided feature propagation module: Without flow guidance, propagated features have more distinct boundaries, indicating less faithful content propagation in the examined temporal features.
B.3. Study of the hallucination ability
When pixels traceable through flow are pre-filled, E2FGVI hallucinates the remaining regions with higher DAVIS PSNR than an image inpainting model.
- Pre-filling flow-traceable pixels isolates hallucination for regions unlikely to be visible in other video frames.
- 31.74 vs. 30.80 dB PSNR favors E2FGVI over the image inpainting model on DAVIS.
B.4. Parameter comparison
E2FGVI uses slightly more parameters than FuseFormer while maintaining a favorable performance–complexity trade-off. It also outperforms a larger FuseFormer variant with similar parameter count.
- B.4. Parameter comparison: E2FGVI consumes approximately 14% more parameters than FuseFormer while achieving a favorable trade-off between performance and computational complexity.The comparison is reported in Table 5.
- B.4. Parameter comparison: E2FGVI still performs better than a larger FuseFormer version designed to have a similar parameter count.
B.5. More Qualitative Results
Additional qualitative results on YouTube-VOS and DAVIS compare E2FGVI with CAP, FGVC, and FuseFormer. Across video completion and object removal examples, E2FGVI produces more faithful and coherent masked-region content.
- B.5. More Qualitative Results: Additional visual comparisons evaluate E2FGVI against CAP, FGVC, and FuseFormer on YouTube-VOS and DAVIS.The results include video completion and object removal examples.
- B.5. More Qualitative Results: E2FGVI generates more faithful textural and structural information than the compared methods in masked regions.
- B.5. More Qualitative Results: E2FGVI produces more coherent contents in masked regions than CAP, FGVC, and FuseFormer.
- B.5. More Qualitative Results: Figures 11 and 12 show qualitative video completion results on YouTube-VOS.
- B.5. More Qualitative Results: Figures 13 and 14 show qualitative object removal and video completion results on DAVIS.