Source-linked AI summary
Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN
Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, Winston Hsu
TL;DR
Free-form video inpainting remains difficult because patch-based methods struggle with non-repetitive structures and frame-wise image inpainting produces temporal inconsistency. The paper combines 3D gated convolutions, a Temporal PatchGAN loss, and a newly constructed FVI dataset to address these challenges. Experiments on FaceForensics and FVI report state-of-the-art results.
Problem
Free-form video inpainting is an unsolved, practical problem because existing patch-based methods struggle with non-repetitive structures and image-based models produce temporally inconsistent videos.
Method
The paper develops a learning-based free-form video inpainting model using 3D gated convolutions, Temporal PatchGAN, and newly generated masks and FVI training data.
Results
Experiments on FaceForensics and FVI indicate that the model achieves state-of-the-art qualitative and quantitative results.
Takeaways & Limitations
The proposed components enhance video quality and temporal consistency for free-form video inpainting, with possible extension to video object removal, super-resolution, and interpolation.
Takeaways & Limitations
The ablation study found that simply increasing channel count could achieve performance similar to gated convolutions.
Abstract
from arXiv · showhide
Free-form video inpainting is a very challenging task that could be widely used for video editing such as text removal. Existing patch-based methods could not handle non-repetitive structures such as faces, while directly applying image-based inpainting models to videos will result in temporal inconsistency (see http://bit.ly/2Fu1n6b ). In this paper, we introduce a deep learn-ing based free-form video inpainting model, with proposed 3D gated convolutions to tackle the uncertainty of free-form masks and a novel Temporal PatchGAN loss to enhance temporal consistency. In addition, we collect videos and design a free-form mask generation algorithm to build the free-form video inpainting (FVI) dataset for training and evaluation of video inpainting models. We demonstrate the benefits of these components and experiments on both the FaceForensics and our FVI dataset suggest that our method is superior to existing ones. Related source code, full-resolution result videos and the FVI dataset could be found on Github https://github.com/amjltc295/Free-Form-Video-Inpainting .
1. Introduction
Free-form video inpainting remains challenging because arbitrary masks defeat patch retrieval and frame-wise image inpainting causes temporal inconsistency. The paper addresses these problems with a learning-based model, temporal modeling, and a new dataset and mask-generation procedure.
- Motivation: Patch-based methods often fail on complex or non-repetitive objects, especially when free-form masks cover a large fraction of the frame.Their nearest-neighbor search depends on suitable visible patches being available.
- Motivation: Applying image inpainting models independently to video frames tends to produce twisted or flickering results that are temporally inconsistent.
- Approach: The proposed model uses 3D gated convolutions to distinguish unmasked, filled, and masked video regions and attend to appropriate features.
- Results: Quantitative and qualitative evaluations on FaceForensics and FVI report state-of-the-art results for the proposed free-form video inpainting model.
- Approach: A Temporal PatchGAN discriminator is introduced to penalize high-frequency spatial-temporal features and enhance temporal consistency and video quality.
- Data: The authors design free-form mask-generation algorithms and collect the FVI dataset, containing 1,940 YouTube-VOS videos and 12,600 YouTube-BoundingBoxes videos.
2. Related Work
Prior work spans image-inpainting advances, traditional patch-based video methods, and early deep video models. These approaches improve reconstruction but remain limited by complex or large missing regions, temporal inconsistency, computation, or blurry outputs.
- Image Inpainting: Deep image-inpainting methods progressed from CNN and encoder-decoder GAN models to contextual attention, gated convolution, and spectral-normalized discriminators.Free-form image methods specifically address irregular holes, while earlier approaches target larger regions or improve blur.
- Video Inpainting: Extending image-inpainting models directly to videos can produce twisted, flickering, and temporally inconsistent frames.The inconsistency arises because independently inpainted frames differ over time.
- Video Inpainting: Patch-based video methods fill missing regions using similar unmasked patches, with later work incorporating texture features, initialization, optical flow, and color estimation.These methods include global optimization and moving-camera handling.
- Video Inpainting: Patch-based methods are computationally expensive and poorly suited to complex structures, large or long-lasting occlusions, and non-repetitive objects.Their nearest-neighbor search also becomes unreliable when free-form masks cover a high fraction of the video.
- Video Inpainting: CombCN introduced an early deep learning baseline using a 3D temporal-consistency network followed by a 2D completion network, but remained limited for free-form masks and blurry in complex scenes.The proposed model instead uses a single stage with gated convolution and perceptual and temporal adversarial losses.
3. Proposed Method
The proposed video-inpainting system combines a 3D gated-convolution generator with a Temporal PatchGAN discriminator and composite losses, while training on varied free-form video masks. Its mask-generation procedure models motion, deformation, border coverage, mask shape, and mask-to-frame ratios.
- Architecture: The model uses a generator with 3D gated convolutions and a Temporal PatchGAN discriminator to inpaint masked video regions.The generator produces output frames from masked input videos, while the discriminator evaluates spatial-temporal patches.
- 3.1. 3D Gated Convolution: 3D gated convolutions combine neighboring-frame information with learned attention over irregular masked areas.The gating map transforms learned gating values through a sigmoid and applies them to convolutional features.
- 3.3. Loss Functions: The training objective combines standard l1, masked l1, perceptual, style, and Temporal PatchGAN losses.The loss weights assign separate terms to pixel-level, masked-region, feature-level, style, and adversarial objectives.
- 3.3. Loss Functions: Temporal PatchGAN classifies spatial-temporal patches, focusing on high-frequency features while avoiding separate global and local GAN discriminators.Its receptive field covers whole videos, and the patch-scale penalty complements the low-frequency focus of l1 loss.
- 3.4. Free-form Video Masks Generation: The free-form mask generator creates masks with multiple shapes, ratios, motion patterns, deformation, and optional frame-border coverage.Mask components can move and deform independently, and the generator includes curve-like, object-like, and bounding-box masks.
- 3.4. Free-form Video Masks Generation: The dataset contains 28,000 training videos spanning mask-to-frame ratios from 0–10% to 60–70%, with 100 test videos per mask type.The generated masks support training and evaluation across different free-form mask conditions.
4. Experimental Results
Experiments on FaceForensics and the FVI dataset compare the model with patch-based, image-based, and learning-based video inpainting baselines across mask types and sizes. The results show stronger perceptual and temporal quality, favorable user preferences, and extensions to object removal and super-resolution.
- Datasets and setup: The FVI dataset contains diverse YouTube videos and supports evaluation of free-form masks and video object removal.It includes videos from YouTube-VOS and YouTube-BoundingBoxes, with object segmentation and bounding-box annotations.
- Datasets and setup: The evaluation uses FaceForensics and FVI, with FVI testing spanning seven mask-to-frame-ratio ranges from 0–10% to 60–70%.Higher mask-to-frame ratios make the task more difficult.
- Quantitative results: On FaceForensics, learning-based methods are favored because non-repetitive face features cannot be recovered reliably from nearby patches.The model outperforms the deep learning baselines on curve-like and object-like masks, while matching Edge-Connect on bounding-box masks.
- Qualitative results: Visual comparisons show wrong patch placement from TCCDS, frame inconsistency from Edge-Connect, and blur from CombCN, whereas the model produces temporally consistent frames.The qualitative comparison is reported on the FVI testing set.
- Extensions: The trained model can also be applied to video object removal and extended to spatial super-resolution with 4x up-sampling.The super-resolution comparison is reported on the VOR testing set.
- Quantitative results: On FVI, the model outperforms the baselines on LPIPS and FID, while CombCN achieves lower MSE but produces blurry results.The FVI results average performance across different mask-to-frame ratios.
- User study: In a study of 150 participants, the model outperforms baselines across object-like and curve-like masks and receives 23% average preference over ground truth.The ground-truth comparison is reported for curve-like masks.
- Ablation study: The ablation study identifies 3D convolution and T-PatchGAN as crucial components because they provide substantial temporal information.The ablation compares models with approximately matched parameter counts.
5. Discussion and Future Work
The discussion identifies distribution shift, thick masks, parameter redundancy, and uncertain gated-convolution benefits as limitations. It also proposes user-guided edge inputs for more controllable video editing.
- Limitations: The model fails when test videos differ substantially from training data and when masked areas are too thick, although it performs better than baselines in those conditions.These limitations reflect data-distribution mismatch and highly extensive missing regions.
- Limitations: 3D convolutions require more parameters than 2D convolutions, potentially increasing redundancy, despite fast single-stage feed-forward inference without optical flow.The authors suggest Temporal Shift Modules as a possible way to reduce parameters.
- Limitations: Increasing channel counts can match gated-convolution performance, suggesting possible underfitting or a smaller gated-convolution effect than in image inpainting.The authors identify improved convolutional layers as future work.
- Future work: Future user-guided edge inputs could provide object-shape information and allow users to manipulate output videos during inference.The proposed editing interface would let users draw lines that modify edge images.
6. Conclusion
The paper presents a learning-based free-form video inpainting network built with 3D gated convolution and a temporal PatchGAN loss. Experiments support state-of-the-art video quality and temporal consistency, with extensions to related video restoration tasks.
- Conclusion: The paper proposes a learning-based free-form video inpainting network using 3D gated convolution and a novel GAN loss.The conclusion identifies this as the first such network to the authors’ knowledge.
- Conclusion: Quantitative and qualitative results indicate state-of-the-art performance while demonstrating improved video quality and temporal consistency.The system is also extended to video object removal, super-resolution, and interpolation.