Source-linked AI summary
Flow-Guided Transformer for Video Inpainting
Kaidong Zhang, Jingjing Fu, Dong Liu
TL;DR
Video inpainting requires spatiotemporal consistency, yet image-based methods rely mainly on within-frame content and prior transformers may retrieve attention inaccurately. This paper completes optical flows and uses their motion cues to guide transformer attention. The authors report qualitative improvements over previous baselines in frame synthesis and flow completion.
Problem
Video inpainting requires spatiotemporal consistency, but direct image-inpainting methods mainly use within-frame content and prior transformers can have inaccurate attention retrieval.
Source: §1 · p. 2Method
The method completes corrupted flows using local temporal flow features, propagates content with those flows, and guides transformer attention with motion discrepancy.
Source: §1 · p. 2Results
The authors report that their method remarkably outperforms previous baselines in visualization results for frame synthesis and flow completion.
Source: §1 · p. 3Takeaways & Limitations
Completed flows support both content propagation and motion-guided attention retrieval for synthesizing remaining corrupted regions.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We propose a flow-guided transformer, which innovatively leverage the motion discrepancy exposed by optical flows to instruct the attention retrieval in transformer for high fidelity video inpainting. More specially, we design a novel flow completion network to complete the corrupted flows by exploiting the relevant flow features in a local temporal window. With the completed flows, we propagate the content across video frames, and adopt the flow-guided transformer to synthesize the rest corrupted regions. We decouple transformers along temporal and spatial dimension, so that we can easily integrate the locally relevant completed flows to instruct spatial attention only. Furthermore, we design a flow-reweight module to precisely control the impact of completed flows on each spatial transformer. For the sake of efficiency, we introduce window partition strategy to both spatial and temporal transformers. Especially in spatial transformer, we design a dual perspective spatial MHSA, which integrates the global tokens to the window-based attention. Extensive experiments demonstrate the effectiveness of the proposed method qualitatively and quantitatively. Codes are available at https://github.com/hitachinsk/FGT.
1 Introduction
Video inpainting must restore spatiotemporal consistency, while image-based and appearance-focused transformer approaches do not fully use complementary cross-frame content or motion cues. The paper proposes local flow completion and flow-guided attention, with decoupled transformers and window-based designs.
1 Introduction
- Video inpainting requires spatiotemporal consistency, which direct image-inpainting methods do not achieve by relying mainly on content within one frame.The task includes applications such as watermark and object removal.Source: §1 · p. 2
- The method uses motion discrepancy in completed optical flows to guide attention retrieval, addressing inaccurate retrieval in appearance-focused video-inpainting transformers.The approach first completes flows, propagates content, then synthesizes remaining corrupted regions with the flow-guided transformer.Source: §1 · p. 2
- Local flow features from a short temporal window support more accurate flow completion than previous counterparts.The network uses decoupled P3D blocks and an edge loss that supervises flow completion at edge regions without adding inference computation.Source: §1 · p. 2
- Spatial and temporal attention are decoupled, and completed flows guide spatial attention while a flow-reweight module adaptively controls their impact.The module accounts for imperfect completed flows and for different-appearance content that may share similar motion patterns.Source: §1 · p. 3
- Window partitioning improves transformer efficiency; dual perspective spatial MHSA combines local-window queries with global tokens to retrieve distant content.The temporal transformer uses a large window, while spatial attention uses a smaller window and incorporates global tokens into keys and values.Source: §1 · p. 3
- The authors report that their method remarkably outperforms previous baselines in visualization results for frame synthesis and flow completion.The comparison is described qualitatively in Fig. 1.Source: §1 · p. 3
2 Related Work
Prior video-inpainting work includes geometric methods and learned approaches that either complete optical flows or synthesize corrupted regions from video frames. The paper distinguishes its approach through local flow-feature use and transformer attention guided by completed flows.
2 Related Work
- Traditional methods use geometric relationships, including homographies or optical flows, to synthesize content from valid reference-frame regions.Huang et al. jointly optimize flow reconstruction and frame synthesis using energy equations.Source: §2 · p. 3
- Unlike prior flow-completion approaches, this method exploits complementary flow features in a local window; unlike prior video transformers, it uses completed flows to improve attention retrieval.The paper also describes spatial-temporal decomposition and local-plus-global tokens as transformer-efficiency strategies.Source: §2 · p. 4
- Deep-learning approaches either complete missing optical flows or synthesize corrupted regions with video-frame features.Synthesis methods include 3D CNNs, channel shifts, recurrent or attention mechanisms, internal learning, and transformers.Source: §2 · p. 4
3 Method
The method completes optical flows from local temporal references, propagates content across frames, and uses a flow-guided transformer to synthesize remaining corrupted regions. It decouples temporal and spatial attention, integrating flow guidance and local-global spatial attention in the spatial transformer.
3.1 Problem formulation
- The model seeks an inpainted video sequence that fills masked regions while maintaining spatiotemporal coherence with the ground truth.Source: §3.1 · pp. 4–5
3.2 Network overview
- Local Aggregation Flow Completion completes forward and backward optical flows from local references, then the Flow-Guided Transformer synthesizes regions left corrupted after content propagation.Source: §3.2 · p. 5
3.3 Local aggregation flow completion network
- LAFC uses short-window flow correlations and P3D blocks to aggregate local features, then decodes completed flows; an edge loss explicitly supervises flow completion at edges.P3D blocks decouple temporal and spatial feature aggregation; the edge loss adds no inference computation.Source: §3.3 · pp. 5–6
3.4 Flow-guided transformer for video inpainting
- FGT separates temporal and spatial MHSA, integrates completed-flow guidance only into spatial attention, and reweights flow tokens through their interaction with frame tokens.It uses large temporal zones and smaller spatial windows augmented with shared global tokens for dual-perspective attention.Source: §3.4 · pp. 6–8, Figure 3 · p. 7
4 Experiments
The experiments evaluate video and flow restoration on Youtube-VOS and DAVIS, and report gains over prior baselines alongside analyses of flow completion, attention design, and efficiency.
4.1 Settings
- Youtube-VOS and DAVIS are used for evaluation, with Youtube-VOS training data for training and its test set and DAVIS training set for evaluation.PSNR, SSIM, and LPIPS evaluate video inpainting; EPE evaluates flow completion.Source: §4.1 · p. 9
4.2 Implementation details
- The implementation uses RAFT flows, three input flows at interval three, gradient-propagation content transfer, and eight transformer blocks split evenly between temporal and spatial stages.The spatial transformer uses window size 8 and global-token downsampling rate 4; the temporal transformer uses 2×2 zone division.Source: §4.2 · p. 9
4.3 Quantitative evaluation
- Our method significantly outperforms previous baselines on all three metrics, with less distortion and better perceptual quality.The evaluation table covers Youtube-VOS and DAVIS; FGT also outperforms previous transformer-based baselines when filling corrupted regions without flow-guided content propagation.Source: Table 1 · p. 10, §4.3 · p. 10
4.4 Qualitative comparisons
- Across square masks, object masks, and object removal, our method achieves outstanding visual quality and more accurate optical flows than recent baselines.The authors report less distortion during content propagation than FGVC and say completed flows provide object clusters for more accurate attention retrieval.Source: §4.4 · pp. 10–11, Figure 5 · p. 11
4.5 Ablation Studies
- Local flow aggregation and edge loss substantially improve flow completion; local aggregation uses complementary nearby flow features, while edge loss produces clearer motion boundaries.Too few or too distant flow references degrade EPE, and distant flows contribute less than local ones.Source: §4.5 · p. 12, Figure 7 · p. 12, Table 3 · p. 13
- All-pair attention adds computation overhead, while flow-guided content propagation improves inpainting quality but reduces speed to 2.11s/frame.The efficiency comparison assumes 20 processed frames.Source: §4.5 · pp. 11–12, Table 2 · p. 12
- Combining global and local tokens improves spatial attention, and completed flow tokens plus flow reweighting improve performance further; removing reweighting degrades it.The authors report that the dual perspective enlarges retrieval space while maintaining local smoothness, and flow guidance tends to retrieve tokens with similar motion patterns.Source: §4.5 · p. 13, Figure 8 · p. 13
5 Conclusion
The paper concludes with a flow-guided transformer for video inpainting that uses completed optical flows to guide attention. Its experiments demonstrate the effectiveness of the proposed method.
5 Conclusion
- The transformer uses optical-flow motion discrepancy to guide attention retrieval, with attention decoupled along spatial and temporal dimensions.A flow-reweight module controls flow impact during attention retrieval.Source: §5 · pp. 13–14
- The flow-completion network exploits complementary optical-flow features in a local temporal window and uses edge loss to produce clearer motion boundaries.The completed flows support content propagation and the flow-guided transformer.Source: §5 · p. 14
- Window partition strategies in temporal and spatial transformer blocks target efficiency while maintaining competitive performance.The spatial attention design integrates global tokens with window-based attention.Source: §5 · p. 14