Source-linked AI summary
Semi-Supervised Video Inpainting with Cycle Consistency Constraints
Zhiliang Wu, Hanyu Xuan, Changchang Sun, Kang Zhang, Yan Yan
TL;DR
Existing video-inpainting methods typically need costly corrupted-region masks for every frame, while framewise image inpainting can lose motion continuity. The paper proposes a one-mask semi-supervised framework and task-specific synthetic data, and reports performance comparable to fully supervised methods. The authors assume the annotated frame is usually the video’s first frame.
Problem
Existing video-inpainting methods generally require costly corrupted-region masks for every frame, and applying image inpainting independently can lose inter-frame motion continuity and cause flicker.
Source: §1 · p. 1Method
An end-to-end framework alternates completion of the current frame and prediction of the next frame’s corrupted-region mask, with cycle consistency regularizing both networks.
Source: §1 · p. 2Results
Takeaways & Limitations
The proposed task is intended to reduce annotation costs, and the tailored dataset of 4,453 clips is intended to facilitate subsequent research.
Source: §1 · p. 2, §5 · p. 8The method usually assumes that the annotated frame is the first frame of the video.
Source: §3.2.1 · p. 4
Abstract
from arXiv · showhide
Deep learning-based video inpainting has yielded promising results and gained increasing attention from researchers. Generally, these methods usually assume that the corrupted region masks of each frame are known and easily obtained. However, the annotation of these masks are labor-intensive and expensive, which limits the practical application of current methods. Therefore, we expect to relax this assumption by defining a new semi-supervised inpainting setting, making the networks have the ability of completing the corrupted regions of the whole video using the annotated mask of only one frame. Specifically, in this work, we propose an end-to-end trainable framework consisting of completion network and mask prediction network, which are designed to generate corrupted contents of the current frame using the known mask and decide the regions to be filled of the next frame, respectively. Besides, we introduce a cycle consistency loss to regularize the training parameters of these two networks. In this way, the completion network and the mask prediction network can constrain each other, and hence the overall performance of the trained model can be maximized. Furthermore, due to the natural existence of prior knowledge (e.g., corrupted contents and clear borders), current video inpainting datasets are not suitable in the context of semi-supervised video inpainting. Thus, we create a new dataset by simulating the corrupted video of real-world scenarios. Extensive experimental results are reported to demonstrate the superiority of our model in the video inpainting task. Remarkably, although our model is trained in a semi-supervised manner, it can achieve comparable performance as fully-supervised methods.
1. Introduction
Video inpainting must preserve temporal continuity, but existing methods require costly masks for every frame. This section introduces a one-mask semi-supervised task, an end-to-end completion and mask-prediction framework, and a synthetic dataset for that setting.
1. Introduction
- Per-frame mask annotation is labor-intensive and expensive, while frame-by-frame image inpainting can lose motion continuity and produce flicker artifacts.The paper also notes that a two-stage combination of video segmentation and inpainting may be suboptimal and can make critical errors on scratch regions.Source: §1 · pp. 1–2
- The framework alternates frame completion and next-frame mask prediction, with cycle consistency loss regularizing the two networks’ parameters.The completion network uses a known mask to complete a frame; the mask prediction network uses that completed frame and the subsequent frame to predict the next mask.Source: §1 · p. 2
- The authors report that their semi-supervised method achieves inpainting results comparable to fully supervised methods.The paper presents this as an experimental result for the proposed method.Source: §1 · p. 2
- The proposed task completes a whole video using the annotated corrupted-region mask of only one frame, rather than requiring masks for every frame.The paper describes this as an end-to-end semi-supervised setting.Source: §1 · p. 2
- The task-specific synthetic dataset contains 4,453 video clips and is intended to facilitate subsequent research.The dataset design also addresses prior cues in conventional corrupted-video data by using natural images as corrupted contents and smoothing region edges.Source: §1 · p. 2
2. Related Works
Prior video-inpainting work includes patch-based and deep-learning approaches, while semi-supervised video object segmentation uses matching, propagation, or detection. These methods still leave a gap for scratched-region masks and the per-frame annotation demands of inpainting.
2. Related Works
- Patch-based methods search for and paste coherent known-region content, but their heavy optimization can impose high computational complexity.The paper notes that this can limit real-world applications.Source: §2 · p. 2
- Deep-learning video inpainting includes 3D convolutional, flow-guided, and attention-based methods, but these approaches still require masks for every frame.The cited approaches aggregate temporal information, restore flow to guide pixels, or retrieve and weight information from neighboring frames.Source: §2 · pp. 2–3
- Semi-supervised video object segmentation methods use matching, propagation, or detection, yet the paper notes limited attention to segmenting scratched regions with complex patterns.The proposed mask predictor instead uses the current completed frame and subsequent frame as inputs.Source: §2 · p. 3
- The authors reason that completion changes only the corrupted regions between the before-and-after frames, making their mask predictor suitable for damaged regions of various modalities.They connect this property to robustness of the proposed semi-supervised inpainting framework.Source: §2 · p. 3
3. Method
The method turns semi-supervised video inpainting into frame completion and next-frame mask prediction, iterating both from one annotated mask. Its two networks use aligned temporal features and cycle consistency within an end-to-end framework.
3.1. Problem Formulation
- With only one frame mask known, the method iterates frame completion and next-frame mask prediction to process the video sequence.Completion estimates what to fill; mask prediction estimates where to fill in the next frame.Source: §3.1 · p. 4
3.2. Network Design
- The end-to-end framework pairs a completion network using reference-frame information with a mask prediction network that estimates the next frame’s corrupted regions.Source: §3.2 · p. 4
3.2.1 Completion Network.
- The completion network aligns reference-frame features with deformable convolution, adaptively aggregates them, and decodes the fused features into the completed target frame.Four deformable convolution alignment blocks are cascaded; adaptive aggregation accounts for reference features that differ in usefulness under occlusion, blur, and parallax.Source: §3.2.1 · pp. 4–5
3.2.2 Mask Prediction Network.
- The mask prediction network compares features from the completed current frame and subsequent frame to estimate the subsequent frame’s corrupted-region mask.Its feature-alignment module shares parameters with the completion network’s alignment module.Source: §3.2.2 · p. 5
3.3. Cycle Consistency
- Cycle consistency nests frame completion and mask prediction in both directions to encourage reliable, consistent correspondence between their mappings.The resulting losses use L1 distance and are combined into a hybrid loss.Source: §3.3 · p. 5
3.4. Loss Function
- Training minimizes a combined loss comprising frame reconstruction, mask prediction, and cycle consistency terms.The reported implementation sets λf = 2.5, λs = 0.25, and λc = 1.Source: §3.4 · pp. 5–6
4. Experiments
Experiments use a dataset designed to reduce visual shortcuts in mask prediction and evaluate both video completion and mask prediction. The method is comparable to fully supervised inpainting baselines on two datasets, while ablations examine its components and annotation settings.
4.1. Dataset and Implementation Details
- Training data uses real-world image patches and free-form masks, with iterative Gaussian smoothing and alpha blending to reduce appearance, shape, and edge cues for mask prediction.The generated dataset contains 3,471 training, 474 validation, and 508 test video clips based on YouTube-VOS.Source: §4.1 · p. 6
4.2. Video Inpainting Evaluation
- The semi-supervised model is comparable to fully supervised video-inpainting baselines on YouTube-VOS and DAVIS, and significantly outperforms methods combining VOS and VI.The model completes the video using only the first frame’s mask; Figure 6 shows sample masks and completed results.Source: Figure 6 · p. 6, §4.2 · p. 7, Table 1 · p. 7
4.3. Mask Prediction Evaluation
- The mask prediction network obtains the best reported segmentation results against specialized video-object-segmentation baselines on YouTube-VOS and DAVIS.It uses the completed current frame and subsequent frame as input; Figure 7 shows example corrupted-region segmentations.Source: Table 2 · p. 7, Figure 7 · p. 7, §4.3 · p. 8
4.4. Ablation Studies
- Ablations report improved performance with cycle consistency loss and the proposed completion network, while changing the annotated mask’s video location has minimal effect.The study also finds that more stacked DCA blocks improve inpainting, choosing four for time-cost reasons; additional annotated frames further improve quality in some environments.Source: §4.4 · p. 8, Table 3 · p. 8, Table 4 · p. 8
5. Conclusion
The paper formulates semi-supervised video inpainting and proposes an end-to-end framework of completion and mask prediction networks. Experiments support the method’s effectiveness, and the tailored dataset is intended to facilitate subsequent research.
5. Conclusion
- The conclusion presents semi-supervised video inpainting as a new task and proposes an end-to-end completion-and-mask-prediction framework, reporting effectiveness and a tailored dataset for subsequent research.The task is described as essential for real-world applications because it can significantly reduce annotation cost.Source: §5 · p. 8