Source-linked AI summary
Deep Flow-Guided Video Inpainting
Rui Xu, Xiaoxiao Li, Bolei Zhou, Chen Change Loy
TL;DR
Video inpainting requires spatially and temporally coherent completion despite complex motion and limitations of existing methods. The paper completes optical flow with DFC-Net and propagates pixels using that flow, achieving state-of-the-art performance and faster runtime on DAVIS and YouTube-VOS. Flow inaccuracies at boundaries can still cause propagation failures.
Problem
Video inpainting must fill missing regions while preserving spatial and temporal coherence, but complex motion and existing methods' limitations make this difficult.
Method
The approach completes optical flow with a coarse-to-fine Deep Flow Completion network, improves it with hard flow example mining, and uses the flow to guide pixel propagation.
Results
The method achieves state-of-the-art performance on DAVIS and YouTube-VOS while providing significantly faster runtime than previous methods.
Takeaways & Limitations
High-quality flow completion can facilitate video inpainting in complex scenes while avoiding assumptions about missing regions and content movements.
Takeaways & Limitations
Inaccurate completed flow at object edges can cause propagation failures that the propagation process cannot amend.
Abstract
from arXiv · showhide
Video inpainting, which aims at filling in missing regions of a video, remains challenging due to the difficulty of preserving the precise spatial and temporal coherence of video contents. In this work we propose a novel flow-guided video inpainting approach. Rather than filling in the RGB pixels of each frame directly, we consider video inpainting as a pixel propagation problem. We first synthesize a spatially and temporally coherent optical flow field across video frames using a newly designed Deep Flow Completion network. Then the synthesized flow field is used to guide the propagation of pixels to fill up the missing regions in the video. Specifically, the Deep Flow Completion network follows a coarse-to-fine refinement to complete the flow fields, while their quality is further improved by hard flow example mining. Following the guide of the completed flow, the missing video regions can be filled up precisely. Our method is evaluated on DAVIS and YouTube-VOS datasets qualitatively and quantitatively, achieving the state-of-the-art performance in terms of inpainting quality and speed.
1. Introduction
Video inpainting must fill missing regions while preserving spatial and temporal coherence, but complex motion, computational cost, and framewise deep methods remain challenging. The paper instead completes optical flow and uses it to propagate pixels, with a coarse-to-fine network and hard-example mining.
- Video inpainting targets missing regions with spatially and temporally coherent content, supporting applications including object removal and video restoration.
- Patch-based methods assume smooth, homogeneous motion and can be computationally impractical for real-world videos.One cited method requires approximately 3 hours for an 854×480 video with 90 frames and 18% missing regions.
- Applying image inpainting independently to frames produces temporal artifacts and jitters, while processing entire videos with 3D CNNs is difficult.
- The proposed approach completes a coherent optical-flow field first, then uses it to guide temporal pixel propagation and preserve video coherence.The method treats video inpainting as pixel propagation rather than direct RGB-pixel filling.
- DFC-Net stacks three subnetworks for coarse-to-fine flow refinement, uses consecutive frames to encourage temporal consistency, and applies hard flow example mining.Hard flow mining targets flow boundaries and dynamic regions.
- The approach is reported as significantly faster than previous methods and achieves state-of-the-art performance on DAVIS and YouTube-VOS.
2. Related Work
Prior inpainting methods primarily borrow appearance information from known regions through diffusion or patch matching. These approaches have limitations in handling large holes, appearance variation, and complex dynamic content.
- Diffusion-based methods propagate appearance information around target holes but cannot handle appearance variations or large holes.
- Patch-based methods fill holes by borrowing appearance information from known regions through patch matching.
- Despite acceleration from PatchMatch, Huang et al.'s method remains approximately 20 times slower than the proposed approach.
- Prior non-learning-based methods cannot capture high-level semantic information and therefore fall short on regions with complex, dynamic motion from multiple objects.
3. Methodology
The method first completes missing optical flow with a coarse-to-fine Deep Flow Completion Network, then uses valid completed flow to propagate pixels into missing regions. Hard Flow Example Mining emphasizes difficult boundary areas, while consistency checks filter unreliable flow before propagation.
- Overall pipeline: The pipeline first completes missing flow and then propagates known pixels using the completed flow as guidance.Remaining regions are handled with a conventional image inpainting network.
- Deep Flow Completion Subnetwork: The first DFC-S receives consecutive flow maps and their binary masks, producing the completed flow field of the middle frame.Using a sequence of flow maps and masks improves completion accuracy over using a single flow map.
- Coarse-to-fine flow completion: DFC-Net stacks three DFC-S subnetworks, progressively refining flow from coarse estimates to higher spatial resolution and accuracy.The three subnetworks process resized inputs at 1/2, 2/3, and 1 of the original size.
- Coarse-to-fine flow completion: The second and third subnetworks refine enlarged predictions, with the second jointly refining forward and backward flows for better alignment.The final subnetwork follows the same refinement procedure as the second.
- Hard flow example mining: Hard Flow Example Mining weights the highest-loss pixels more heavily, focusing training on difficult boundary regions to produce sharper flow boundaries.Smooth flow areas otherwise dominate the L1 loss, blurring boundaries and potentially causing propagation artifacts.
- Flow-guided pixel propagation: Before propagation, forward and backward flow estimates are checked for photometric consistency, and unreliable estimates are ignored.Valid known pixels are then propagated bidirectionally into missing regions, combining values from forward and backward pixels when both are available.
4. Experiments
Experiments evaluate the approach on DAVIS and YouTube-VOS using quantitative, user-study, qualitative, and ablation analyses. Results support the benefits of flow completion, hard-example mining, coarse-to-fine refinement, and flow-guided propagation, while exposing a failure case from inaccurate flow at object boundaries.
- Main Results: The approach achieves the best quantitative performance on DAVIS and YouTube-VOS for fixed-region inpainting and is significantly faster than conventional video inpainting methods.The comparison also finds direct per-frame image inpainting inferior and conventional methods less effective for complex motions.
- Main Results: In a 30-participant user study, the proposed method significantly outperforms Deepfill and Huang et al., while the image-inpainting baseline performs worst.Participants ranked three results under two inpainting settings.
- Main Results: Qualitative comparisons show more accurate flow completion and fewer artifacts than Huang et al., especially for complex masked regions.The comparison includes a challenging masked region containing part of a woman.
- Ablation Study: Hard flow example mining improves flow-completion and corresponding inpainting performance across settings, with the best mining proportion ranging from 30% to 50%.The experiments fix the proportion at 50%.
- Ablation Study: Coarse-to-fine refinement progressively reduces endpoint error, while stacked multi-scale processing outperforms single-stage and no-multi-scale variants.Using large-scale inputs early can destabilize training, whereas multi-scale inputs improve results.
- Ablation Study: Bidirectional propagation of known pixels through completed flow reduces the remaining missing regions and improves performance over direct image inpainting with flow warping.This intermediate propagation step eases the subsequent inpainting task.
- Ablation Study: A failure case occurs when the completed flow is inaccurate at the car’s edge, because propagation cannot amend that error.The paper proposes learning-based propagation as future mitigation.
5. Conclusion
The paper concludes that deep flow-guided video inpainting can handle arbitrary missing regions and complex motions while maintaining temporal consistency. It reports state-of-the-art performance on DAVIS and YouTube-VOS with significantly faster runtime than previous methods.
- Conclusion: The method targets arbitrary missing regions and complex motions while maintaining temporal consistency.Its Deep Flow Completion network supports high-quality flow completion for complex-scene video completion.
- Conclusion: Compared with previous methods, the approach is significantly faster and requires no assumptions about missing regions or video-content movements.The conclusion frames these properties as advantages for the reported application scope.
- Conclusion: The approach achieves state-of-the-art performance on both DAVIS and YouTube-VOS datasets.The conclusion reports this outcome without specifying a single metric.