Source-linked AI summary
Video Inpainting by Jointly Learning Temporal Structure and Spatial Details
Chuan Wang, Haibin Huang, Xiaoguang Han, Jue Wang
TL;DR
Video inpainting must recover missing content that is semantically correct, spatially detailed, and temporally coherent. The paper jointly trains a low-resolution 3D temporal structure network with an original-resolution 2D detail-recovery network, and reports improved performance over existing methods. Its effectiveness is limited by severe train-test variation and difficulty capturing large motion.
Problem
Video inpainting must recover semantically correct content while maintaining spatial and temporal consistency across missing regions and moving frames.
Method
A jointly trained 3D network predicts low-resolution temporal structure, which guides a 2D network recovering spatial details at the original video resolution.
Results
The method produces plausible results for regular or random holes and significantly improves performance over existing methods.
Takeaways & Limitations
Separating temporal structure inference from spatial detail recovery preserves temporal coherence and frame-level details simultaneously.
Takeaways & Limitations
Performance may fail under severe variation from training data, and large motion is difficult to capture because of the 3D CNN receptive-field limitation.
Abstract
from arXiv · showhide
We present a new data-driven video inpainting method for recovering missing regions of video frames. A novel deep learning architecture is proposed which contains two sub-networks: a temporal structure inference network and a spatial detail recovering network. The temporal structure inference network is built upon a 3D fully convolutional architecture: it only learns to complete a low-resolution video volume given the expensive computational cost of 3D convolution. The low resolution result provides temporal guidance to the spatial detail recovering network, which performs image-based inpainting with a 2D fully convolutional network to produce recovered video frames in their original resolution. Such two-step network design ensures both the spatial quality of each frame and the temporal coherence across frames. Our method jointly trains both sub-networks in an end-to-end manner. We provide qualitative and quantitative evaluation on three datasets, demonstrating that our method outperforms previous learning-based video inpainting methods.
Introduction
Video inpainting must recover semantically correct content while remaining seamless within frames and temporally consistent across motion. The proposed end-to-end architecture separates temporal structure prediction from spatial detail recovery and jointly trains both.
- Video inpainting must satisfy semantic correctness and seamless visual filling in missing regions.These requirements apply to recovering content removed from images or videos.
- Video completion additionally requires modeling spatial context within frames and motion context across frames.The output must maintain global and local spatio-temporal consistency.
- Directly extending local 2D patch synthesis to 3D cannot guarantee global semantic correctness for video.Prior 3D patch-based approaches extend spatial synthesis across time but remain limited by local synthesis.
- The proposed architecture uses a 3D temporal structure prediction network and a 2D spatial detail recovery network.The 3D network processes a down-sampled video volume, while the 2D network recovers original-resolution frames using temporal guidance.
- Joint training lets the two sub-networks benefit from each other and further improves the overall system.The spatial network’s loss is backpropagated into temporal structure prediction.
Related Work
Prior video inpainting research extends patch synthesis or image-completion networks, but video completion must jointly handle high-level context, motion, and temporal smoothness. Related 3D CNN work also shows the computational difficulty of high-resolution volumetric completion.
- Patch-based image/video inpainting: Patch-based video inpainting extends 2D patch synthesis into spatial-temporal 3D synthesis across frames.These methods were designed largely for videos with repeated content across frames.
- Patch-based image/video inpainting: Patch-based methods cannot handle content prediction requiring high-level context understanding.Their applicability is limited when missing content cannot be recovered from repeated material.
- Image completion using 2D CNN: 2D CNN image-completion methods use encoder-decoder or multi-scale architectures to recover large regions and high-frequency details.Some methods also address arbitrary-shaped holes and global or local consistency.
- Image completion using 2D CNN: Extending image-completion networks to video requires accurate context understanding across frames and motion while preserving temporal smoothness.Framewise image completion alone does not directly provide these video-level properties.
- Shape completion using 3D CNN: 3D CNN shape-completion methods generally operate only on low-resolution grids because 3D convolutions are computationally expensive.Subsequent approaches refine or combine low-resolution global structure with local detail inference.
Algorithm
The algorithm combines a low-resolution 3D completion network for temporal structure with a 2D completion network for spatial details at the original resolution. Temporal guidance is fused into the 2D network, and the two sub-networks are jointly optimized.
- Inputs and outputs: The method takes an incomplete video and mask as input and produces a completed video, with training inputs formed by masking complete videos.The video is represented as an F × H × W volume, and the mask has the same size as the input.
- Temporal structure inference by 3DCN: 3DCN applies 3D CNNs to a down-sampled video and mask to infer global temporal structure.Its encoder-decoder architecture captures spatio-temporal structure, while low resolution limits computational cost.
- Spatial details inference by CombCN: CombCN applies 2D CNN frame by frame to the original-resolution incomplete video, using low-resolution 3DCN outputs to recover spatial details.The 3DCN output is temporally informative but lacks frame-level detail, so it is injected into CombCN through feature maps added at matching layers.
- Network design: The network uses encoder-decoder architectures with dilated convolutions and skip connections to enlarge receptive fields and mix encoder-decoder features.CombCN includes dilated layers so it can use information farther from missing regions, while U-Net-style skip connections support feature mixing.
- Temporal guidance: Temporal guidance is injected into CombCN at matching layers, and experiments report that this preserves temporal coherence when frames are inpainted separately.The guidance features are added to the first and last CombCN layers of the same size; the described combination uses the second and fifteenth layers.
- Training: Training first pre-trains 3DCN, then jointly fine-tunes it with CombCN by optimizing the weighted sum of their losses.The reported strategy outperforms versions without 3DCN pre-training or without 3DCN fine-tuning; the balancing parameter α is set to 1.0.
Experimental Results
Experiments on face and traffic-video datasets compare CombCN with 2DCN, 3DCN, random-hole settings, and training or architecture variants. CombCN combines the temporal consistency of 3DCN with the spatial detail of 2DCN, while large motion and distribution shifts remain failure conditions.
- Datasets and setup: The evaluation uses FaceForensics, 300VW, and the more challenging Caltech traffic dataset, with 32-frame samples resized to 128² and consistently positioned training holes.Caltech contains 10 hours of urban driving video, while face datasets contain near-frontal videos with limited semantic change.
- Comparisons with existing methods: Combining 3DCN and 2DCN produces stable, reasonable results across frames, avoiding 2DCN flicker and 3DCN blur while preserving temporal coherence and details.2DCN can produce unstable frame quality, missing content, blur, or distortion; 3DCN preserves smooth transitions but loses fine details.
- Random holes: For random holes, CombCN combines 3DCN's temporally consistent filling with 2DCN's clearer spatial results, producing clearer and smoother outputs.Distinct holes across frames make inter-frame consistency more difficult for 2DCN, whereas 3DCN can exploit non-hole pixels in neighboring frames.
- Performance of variants of 3DCN: The low-resolution 3DCN variant yields convergence errors of 9.51 for 3DCN and 6.39 for CombCN, versus 4.45 and 4.20 for the baseline model.Down-sampling produces blurrier frames while temporal coherence is rarely lost.
- Performance of variants of 3DCN: Temporal down-sampling raises convergence errors to 11.56 for 3DCN and 8.13 for CombCN, although CombCN's visual performance is not obviously degraded.The variant preserves less temporal coherence but produces more compact extracted features.
- Training strategy: Training both sub-networks from scratch requires over 50k iterations to converge and reaches losses of 9.28 for 3DCN and 6.31 for CombCN.Pre-training 3DCN and fine-tuning it during CombCN training provides faster convergence and lower losses than the alternative strategies.
- Limitations: The method may fail when test videos differ severely from training data, especially with large motion that exceeds the effective 3DCNN receptive field.A reported failure involves a large-motion face video captured from a distant viewpoint, producing an unreasonable face in the fourth frame.
- Limitations: Using only l1 loss may limit the combination idea's ability to produce vivid results compared with approaches that use GAN-based synthesis.The authors identify integrating GANs into 3DCN and CombCN as future work.
Conclusion
The paper presents an end-to-end joint 2D-3D CNN framework for video inpainting that fills regular or random holes and produces plausible results.
- The framework combines temporal structure inference with spatial detail recovery in a jointly trained 2D-3D CNN.