Source-linked AI summary
FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, Hongsheng Li
TL;DR
Patch-based Transformers for video inpainting lack fine-grained sub-patch interaction because hard splitting separates neighboring pixels, limiting detailed reconstruction. FuseFormer addresses this with overlapping Soft Split and Soft Composition operations and integrates them into a fusion feed-forward network. The resulting model achieves state-of-the-art performance in video restoration and object removal, with reported gains over prior methods.
Problem
Hard patch splitting limits sub-patch-level representation and interaction in video inpainting, where fine-grained information is needed for detailed reconstruction.
Method
FuseFormer combines overlapping Soft Split and Soft Composition with a Fusion Feed Forward Network that performs fine-grained feature fusion without extra parameters.
Results
FuseFormer achieves state-of-the-art performance in video restoration and object removal, while ViF improves PSNR and SSIM by 3.3% and 0.7%, respectively.
Takeaways & Limitations
Overlapping patch operations enable sub-token feature fusion between neighboring patches for more effective video-inpainting reconstruction.
Abstract
from arXiv · showhide
Transformer, as a strong and flexible architecture for modelling long-range relations, has been widely explored in vision tasks. However, when used in video inpainting that requires fine-grained representation, existed method still suffers from yielding blurry edges in detail due to the hard patch splitting. Here we aim to tackle this problem by proposing FuseFormer, a Transformer model designed for video inpainting via fine-grained feature fusion based on novel Soft Split and Soft Composition operations. The soft split divides feature map into many patches with given overlapping interval. On the contrary, the soft composition operates by stitching different patches into a whole feature map where pixels in overlapping regions are summed up. These two modules are first used in tokenization before Transformer layers and de-tokenization after Transformer layers, for effective mapping between tokens and features. Therefore, sub-patch level information interaction is enabled for more effective feature propagation between neighboring patches, resulting in synthesizing vivid content for hole regions in videos. Moreover, in FuseFormer, we elaborately insert the soft composition and soft split into the feed-forward network, enabling the 1D linear layers to have the capability of modelling 2D structure. And, the sub-patch level feature fusion ability is further enhanced. In both quantitative and qualitative evaluations, our proposed FuseFormer surpasses state-of-the-art methods. We also conduct detailed analysis to examine its superiority.
1. Introduction
FuseFormer addresses the loss of fine-grained information caused by hard patch splitting in Transformer-based video inpainting. It combines overlapping soft split/composition operations with fusion inside the feed-forward network to improve sub-token feature interaction and video synthesis.
- ViB-T provides a simple vanilla-Transformer baseline for video inpainting using multi-frame patch tokens with convolutional encoder and decoder modules.
- Hard non-overlapping patch splitting prevents effective sub-token interaction, which can produce inconsistent content between neighboring patches and inaccurate pixel-level reconstruction.
- Soft Split creates overlapping 2D patches, while Soft Composition stitches them back together by aggregating features in overlapping regions.
- FuseFormer replaces standard Transformer MLPs with Fusion Feed Forward Networks that reshape tokens into patches and fuse overlapping features without extra parameters.
- Extensive qualitative and quantitative experiments report FuseFormer’s superiority over state-of-the-art video-inpainting approaches.
2. Related work
Prior video-inpainting methods use convolutional, recurrent, optical-flow, copy-and-paste, and Transformer-based approaches. STTN introduced spatiotemporal Transformer propagation but struggles with local textures and arbitrary pixel flow, motivating FuseFormer’s soft operations.
- Earlier video-inpainting systems combine 2D and 3D convolutions, recurrent temporal aggregation, optical-flow warping, or learned copy-and-paste mechanisms.
- STTN applies Transformer encoders to non-overlapping spatiotemporal patches for spatial-temporal propagation in video inpainting.
- STTN has difficulty capturing local textures such as edges and lines and modeling arbitrary pixel flow.
- FuseFormer introduces two soft operations that improve video restoration and object removal while making inference faster.
3. Method
FuseFormer builds from a vanilla Transformer video-inpainting baseline and introduces overlapping Soft Split and Soft Composition operations for sub-token feature fusion. It further inserts these operations into the feed-forward network to model 2D structure without extra parameters.
- Video inpainting baseline: ViB-T combines a convolutional encoder-decoder, standard Transformer blocks, and patch-to-token/token-to-patch modules for video inpainting.It hard-splits feature maps into patches and linearly embeds them into lower-dimensional tokens for efficient processing.
- Video inpainting baseline: The baseline processes encoded frames into tokens, propagates spatial-temporal information with Transformer blocks, and decodes recomposited feature maps into inpainted frames.The pipeline uses convolutional encoding, token refinement, patch recomposition, and deconvolutional decoding.
- Video inpainting baseline: ViB-T already achieves competitive performance with STTN while providing faster inference and fewer parameters.The baseline uses equal kernel size and stride for hard patch splitting.
- Soft Split and Soft Composition: Soft Composition and Soft Split enable sub-token feature fusion, smoothing patch boundaries and enlarging the receptive field through neighboring-patch information.The resulting ViB-S baseline is reported to surpass the original baseline.
- Soft Split and Soft Composition: Soft Split creates overlapping patches with kernel size k greater than stride s, while Soft Composition reconstructs feature maps by summing values at overlapping locations.These operations replace non-overlapping patch processing with sub-patch-level interaction between neighboring patches.
- Fusion Feed Forward Network: FuseFormer replaces the standard feed-forward network with F3N, which inserts Soft Composition and Soft Split between two MLP layers without extra parameters.F3N reshapes intermediate token vectors into 2D features, composites them, then splits them back into tokens.
- Fusion Feed Forward Network: F3N changes MLP dimensions so intermediate vectors can be reshaped into 2D maps and normalizes composition to address variation from different overlap counts.The normalization accounts for spatial locations receiving different numbers of overlapping patches.
- Training objective: Training minimizes reconstruction and adversarial losses, with an auxiliary discriminator used to improve synthesis realism and temporal consistency.The reconstruction term uses L1 distance between synthesized and original videos.
4. Experiments
Experiments evaluate FuseFormer through dataset-based implementation details, ablations, qualitative and quantitative comparisons, and user studies. The results show that overlapping patch operations and sub-token fusion improve performance, while FuseFormer achieves strong restoration quality and temporal coherence.
- Implementation details: Experiments use YouTube-VOS and DAVIS for training and evaluation, with DAVIS including a 60-video test set for fair comparison.
- Ablations: Overlapping patches significantly improve ViB-S and ViF performance, and soft composition alone also provides gains without soft split.ViB-T with patch size 3 is competitive with STTN even without soft operations.
- Ablations: Replacing the standard Transformer block with F3N significantly boosts performance, while normalization further improves results.FuseFormer uses slightly fewer parameters and has negligible time cost while enabling sub-token feature fusion.
- Ablations: ViB-S and ViF produce more detailed inpainting results, supporting the effectiveness of sub-token-level feature fusion.
- Comparison with other methods: ViF improves PSNR by 3.3% and SSIM by 0.7%, while reducing VFID by 7.4% and warping error by 7.8% against state-of-the-art approaches.The comparison covers video completion on YouTube-VOS and reports improved realism and temporal coherence.
- Comparison with other methods: In a user study with 38 viewers and 30 videos, the proposed model achieves the highest first-ranking percentage for both object removal and video completion.The study compares videos produced by the proposed model, CAP, and STTN.
5. Conclusion
FuseFormer addresses fine-grained information loss in patch-based video inpainting through overlapping feature fusion and achieves state-of-the-art video restoration and object-removal performance.
- FuseFormer is a Transformer model for video inpainting that targets the lack of fine-grained information in patch-based methods.
- Soft Split creates overlapping patches, while Soft Composition stitches them into a feature map by summing pixels in overlapping regions.
- Embedding Soft Composition and Soft Split into the feed-forward network further enhances sub-patch-level feature fusion.
- FuseFormer achieves state-of-the-art performance in video restoration and object removal.