Source-linked AI summary
Flow-edge Guided Video Completion
Chen Gao, Ayush Saraf, Jia-Bin Huang, Johannes Kopf
TL;DR
Video completion requires temporally coherent synthesis, but prior flow methods can lose sharp motion boundaries and fail to reach regions blocked by motion barriers. This method completes flow edges, guides piecewise-smooth flow completion, and adds non-local temporal connections. On DAVIS, it compares favorably with state-of-the-art methods in visual and quantitative evaluation.
Problem
Prior flow-completion methods often fail to preserve sharp motion boundaries, while adjacent-frame flow chains cannot reach some missing regions across motion barriers.
Method
The method completes flow edges, uses them to guide piecewise-smooth flow, introduces non-local temporal connections, and propagates candidates through flow trajectories.
Results
The method compares favorably with state-of-the-art algorithms in both quantitative evaluation and visual quality on the DAVIS dataset.
Takeaways & Limitations
Explicit edge completion, non-local flow, and gradient-domain processing address complementary limitations of flow-based video completion.
Takeaways & Limitations
The method runs at 0.12 fps, and the authors acknowledge its slower running time as a weakness.
Abstract
from arXiv · showhide
We present a new flow-based video completion algorithm. Previous flow completion methods are often unable to retain the sharpness of motion boundaries. Our method first extracts and completes motion edges, and then uses them to guide piecewise-smooth flow completion with sharp edges. Existing methods propagate colors among local flow connections between adjacent frames. However, not all missing regions in a video can be reached in this way because the motion boundaries form impenetrable barriers. Our method alleviates this problem by introducing non-local flow connections to temporally distant frames, enabling propagating video content over motion boundaries. We validate our approach on the DAVIS dataset. Both visual and quantitative results show that our method compares favorably against the state-of-the-art algorithms.
1 Introduction
Video completion must generate temporally coherent, imperceptible content despite camera and object motion, while existing flow methods struggle with sharp motion boundaries and disconnected temporal regions. The proposed method completes flow edges, adds non-local connections, blends in the gradient domain, and improves quantitative and visual results over prior methods.
- Motivation: Video completion fills missing space-time regions while requiring temporal coherence and seamless integration despite dynamic camera and object motion.Applications include restoration, video editing, watermark removal, and stabilization.
- Limitations of prior methods: Existing flow-based methods often over-smooth motion boundaries, limiting completion of partially visible dynamic objects and complex scenes.Their flow completion does not preserve sharp edges along object boundaries.
- Key contributions: Non-local flow connections provide shortcuts across motion barriers, enabling color propagation to regions unreachable through adjacent-frame flow chains.The approach targets cases such as periodic walking motion where local trajectories cannot reach background regions between legs.
- Key contributions: Gradient-domain processing reduces visible seams caused by temporal color changes from lighting, shadows, exposure, and white balancing.The method fuses propagated content in the gradient domain rather than directly combining color values.
- Key contributions: Explicitly completed flow edges guide piecewise-smooth flow completion with sharp motion boundaries.The algorithm computes adjacent and non-adjacent flow, completes its edges, and uses them as guidance.
- Evaluation: The method handles videos up to 4K resolution, while other methods fail from excessive GPU memory requirements.The authors also report clear improvement over prior methods in quantitative evaluation and visual quality.
2 Related Work
Video completion methods have progressed from patch-based synthesis toward flow- and learning-based approaches, but dynamic motion, resolution limits, and seamless blending remain important challenges. The cited flow-completion example illustrates the role of completed motion structure in producing piecewise-smooth results.
- Image completion: Image completion methods transfer patches or segments from known regions, sometimes augmenting searches with geometric and photometric transformations.Approaches include patch synthesis and graph-cut labeling.
- Video completion: Video completion adds temporal challenges to image completion because missing content must remain coherent across time.The field includes methods using spatio-temporal patches, spatial patches with registration, and explicit flow constraints.
- Video completion: 3D patches struggle with dynamic videos, while 3D CNN methods limit processable spatial and temporal resolution because of memory constraints.Some CNN approaches sample nearby reference frames to reduce this limitation.
- Seamless blending: Gradient-domain techniques support seamless blending in applications including image editing, panorama stitching, and moving-object insertion.In video completion, Poisson blending can hide seams along hole boundaries as post-processing.
- Flow completion: Figure 3 contrasts incomplete optical flow, completed flow edges, and piecewise-smooth flow guided by those edges.Missing regions appear as zero-valued white areas in the initial flow visualization.
3 Method
The method completes motion edges before piecewise-smooth flow, propagates colors through local and non-local temporal neighbors, and uses gradient-domain reconstruction to reduce seams. Non-local connections address flow barriers that isolate missing pixels from known content.
- Overview: The method takes a color video and binary mask video as input, then performs flow completion, temporal propagation, and iterative spatial completion.Spatial completion fills remaining missing pixels in a key frame, whose result is propagated in a subsequent iteration for temporal consistency.
- Edge-guided Flow Completion: Flow edges are extracted and completed first, then used to guide piecewise-smooth flow completion with sharp motion boundaries.The method computes adjacent and non-adjacent flow fields, removes mask-region edges, and completes the flow in an edge-guided manner.
- Edge-guided Flow Completion: Flow completion uses a smoothness objective over non-edge pixels with known-flow boundary conditions, solved as sparse linear equations by standard linear least squares.Horizontal and vertical finite differences are minimized except at flow edges, while known pixels outside the mask remain fixed.
- Temporal Propagation: For each missing pixel, forward and backward flow trajectories are followed until known pixels are reached, with cycle-consistency checks stopping tracing above τ = 5 pixels.These reachable pixels are local temporal neighbors; non-local flow adds candidates from temporally distant frames when local chains encounter barriers.
- Temporal Propagation: Non-local temporal neighbors short-cut flow barriers, reducing isolated pixels and the need for hallucination; the method uses three additional frames selected as the first, middle, and last frames.The figure illustrates extra candidates across a person’s sweeping legs, revealing background that local trajectories cannot reach.
- Fusing Temporal Neighbors: Candidate colors are weighted by flow consistency error, with T = 0.1 used to strongly down-weight neighbors having large errors, and gradients are reconstructed with Poisson reconstruction to suppress seams.The fixed non-local-frame selection was considered adequate for short videos, while longer videos may require more sophisticated neighbor schemes.
4 Experimental Results
Experiments on DAVIS evaluate two synthetic-mask completion scenarios, quantitative performance, qualitative results, component ablations, and failure cases. The method improves over prior approaches while retaining sharp flow boundaries, though large missing regions and fast motion remain challenging.
- Experimental setup: The evaluation covers stationary screen-space mask inpainting and object removal, with an additional synthetic object-mask task because ground truth is unavailable behind removed objects.Stationary masks use a uniform grid of 5 × 4 square blocks, while object removal masks cover entire moving foreground objects.
- Experimental setup: The method is evaluated on DAVIS, using 60 sequences for training the flow-edge network and 90 for testing stationary-mask inpainting.The object-removal evaluation uses 29 of the 90 test sequences with available refined masks.
- Quantitative evaluation: The method substantially improves performance over state-of-the-art algorithms on PSNR, SSIM, and LPIPS, with results reported at two resolutions.All scenes were also downscaled to 720×384 because some methods could not process the full 960 × 512 resolution.
- Qualitative evaluation: Qualitative results show temporally coherent and visually plausible completions across diverse sequences and both evaluation scenarios.Figure 7 presents stationary screen-space inpainting in the first three columns and object removal in the last three columns.
- Ablation study: Ablations show that gradient-domain propagation removes visible seams, non-local neighbors reduce artifacts in difficult cases, and edge-guided flow completion lowers flow endpoint error.Non-local neighbors are especially useful when adjacent-frame flow neighbors are incorrect or unavailable; edge-guided completion outperforms diffusion and a trained completion network on EPE.
- Limitations: Failure cases include implausible hallucinations in large missing regions and poor color completion caused by inaccurately estimated flow during fast motion.The method runs at 0.12 fps, which the authors acknowledge as a weakness relative to faster end-to-end models.