Source-linked AI summary
Deep Video Inpainting
Dahun Kim, Sanghyun Woo, Joon-Young Lee, In So Kweon
TL;DR
Video inpainting must fill missing spatio-temporal regions while handling motion and temporal consistency, challenges not addressed adequately by straightforward image-inpainting extensions. The paper presents VINet, a feed-forward 3D-2D encoder-decoder with neighbor-frame aggregation, recurrent feedback, temporal memory, and consistency losses. It reports more semantically correct and temporally smooth videos than per-frame image inpainting, competitive results versus optimization, near-real-time inference, and favorable video-retargeting results.
Problem
Extending image inpainting to video is difficult because motion and temporal consistency make frame-by-frame processing produce inconsistent, flickering results.
Method
VINet uses a sequential multi-to-single 3D-2D encoder-decoder that aggregates neighboring-frame features and combines recurrent feedback, temporal memory, flow loss, and warping loss.
Results
The method produces more accurate and visually pleasing videos than, runs near real time without test-time optical flow, and achieves the smallest FID among compared methods.
Takeaways & Limitations
A unified feed-forward network can provide temporally coherent video inpainting while avoiding computation-heavy optimization and support related tasks such as video retargeting.
Takeaways & Limitations
Large and long occlusions can cause color-saturation artifacts, inaccurate warping, and blurry regions; experiments are limited to 256 × 256 px frames.
Abstract
from arXiv · showhide
Video inpainting aims to fill spatio-temporal holes with plausible content in a video. Despite tremendous progress of deep neural networks for image inpainting, it is challenging to extend these methods to the video domain due to the additional time dimension. In this work, we propose a novel deep network architecture for fast video inpainting. Built upon an image-based encoder-decoder model, our framework is designed to collect and refine information from neighbor frames and synthesize still-unknown regions. At the same time, the output is enforced to be temporally consistent by a recurrent feedback and a temporal memory module. Compared with the state-of-the-art image inpainting algorithm, our method produces videos that are much more semantically correct and temporally smooth. In contrast to the prior video completion method which relies on time-consuming optimization, our method runs in near real-time while generating competitive video results. Finally, we applied our framework to video retargeting task, and obtain visually pleasing results.
1. Introduction
Video inpainting is difficult because temporal dynamics require both plausible hole filling and consistency across frames. The paper proposes VINet, a feed-forward architecture that aggregates neighboring-frame information, preserves temporal coherence, and operates near real time.
- Motivation: Video inpainting extends image inpainting into a harder setting because complex motion and temporal consistency must be handled.Frame-by-frame image inpainting ignores motion regularities and can produce severe flickering artifacts.
- Motivation: Prior methods address temporal consistency with motion-field completion or optimization, but optimization-based approaches are time-consuming.Greedy spatio-temporal patches, diffusion, and iterative optimization each impose practical limitations described by the authors.
- Results: VINet sequentially processes videos of arbitrary length without optical-flow computation at test time and runs at a near-real-time rate.Experiments report more accurate and visually pleasing videos than the image-inpainting method.
- Method: VINet formulates video inpainting as sequential multi-to-single-frame inpainting using a 3D-2D encoder-decoder that aggregates neighbor-frame features.The network collects and refines hints from neighboring frames to synthesize semantically coherent content in space and time.
- Method: Recurrent feedback, temporal memory, flow loss, and warping loss are used to enforce short-term and long-term temporal consistency.These components are designed to stabilize outputs across frames while retaining the feed-forward network structure.
- Applications: The unified network is also applied to video retargeting and super-resolution, with favorable results reported by the authors.The paper presents these applications as evidence of the framework’s broader utility.
2. Related Work
Earlier video-inpainting methods use object-based, patch-based, or optimization-based strategies, often requiring constrained motion, visible-region copying, or expensive computation. The paper instead introduces a deep 3D-2D encoder-decoder that exploits temporal information from multiple frames for efficient inpainting.
- Prior Approaches: Video inpainting remains less explored than image inpainting because the time dimension introduces viewpoint changes, temporal-consistency demands, and high computational complexity.Prior literature commonly addresses these challenges with object-based or patch-based methods.
- Object-Based Methods: Object-based methods require foreground-background separation and reconstruct content by copying from visible regions, limiting the content that can be synthesized.The passage identifies visible-region copying as the major limitation of this approach.
- Patch-Based Methods: Patch-based methods assume static or constrained camera motion and can propagate early patch-filling errors into globally inconsistent outputs.Their greedy filling process makes initial mistakes difficult to correct.
- Optimization-Based Methods: Global optimization methods improve temporal coherence by optimizing spatio-temporal patch assignments, but the approach remains computationally demanding.The related work describes alternating patch search and reconstruction, including optical-flow terms for temporal consistency.
- Deep Learning Approach: The proposed deep method uses a 3D-2D encoder-decoder with multiple source frames and one reference frame to learn feature flow for hole filling and coherence.The model takes six frames in total: five source frames and one reference frame.
- Deep Learning Approach: The paper positions deep CNNs as a unified alternative that learns spatial semantics and temporal dynamics for general video inpainting.The authors describe this as the first deep CNN approach addressing the general video-inpainting problem.
3.1. Problem Formulation
Video inpainting reconstructs missing regions while requiring outputs to remain accurate and consistent across space and time. VINet models each current frame from neighboring frames, prior predictions, and recurrent history.
- Video inpainting seeks outputs that are accurate to ground-truth frames and consistent in space and time.
- VINet conditions the current inpainted frame on spatio-temporal neighbors, the previous generated frame, and recurrent memory.
- With N set to 2, the model uses two lagging and two leading frames sampled with temporal stride 3.
- The formulation uses a sequential multi-to-single frame task and applies the network autoregressively.
3.2. Network Design
VINet aggregates information from multiple source frames and synthesizes the current frame with a 3D-2D encoder-decoder. Feature alignment, composition, recurrent feedback, and temporal memory support coherent outputs.
- VINet jointly performs temporal feature aggregation and single-frame inpainting using a fully convolutional encoder-decoder.
- Source and reference encoders: A six-tower encoder uses five weight-shared source streams and one reference stream to borrow non-overlapping features from neighboring and prior frames.
- Feature flow learning: Flow sub-networks align source and reference feature maps at four spatial scales using a coarse-to-fine structure.
- Learnable Feature Composition: Learnable composition concatenates aligned source features across time and uses masks to combine them with reference features.
- Decoder: At the finest scale, estimated flow warps the previous output, which is blended with the raw output to produce the final frame.
- Recurrent feedback preserves traceable features, while ConvLSTM memory connects longer-term information and helps avoid ghosting at occlusions or motion discontinuities.
3.3. Losses
VINet combines reconstruction, flow-estimation, and warping objectives. Short- and long-term temporal losses use optical-flow and occlusion information to enforce coherence.
- The full objective combines reconstruction, flow estimation, and warping losses with balancing weights λR=1, λF=10, and λW=1.The temporal losses use five recurrences.
- Reconstruction loss: The reconstruction loss contains L1 and structural-similarity terms.
- Flow loss: The flow loss measures endpoint error and warping error using pseudo-ground-truth backward flow extracted by FlowNet2.
- Warping loss: The warping loss includes short-term and long-term temporal terms based on occlusion masks and backward optical flow.
3.4. Two-Stage Training
Training proceeds in two stages: first learning temporal feature aggregation, then fine-tuning recurrent temporal-consistency components with the full loss. Training uses YouTube-VOS videos and diverse mask patterns.
- Stage 1: Stage one trains without recurrent feedback or memory using only reconstruction loss to learn temporal feature aggregation.
- Stage 2: Stage two adds recurrent feedback and ConvLSTM memory, then fine-tunes with the full loss for temporally coherent predictions.
- Training uses 4000+ YouTube-VOS videos containing 70+ common objects, with frames resized to 256 × 256 pixels.
- Video mask dataset: Four mask types simulate square, moving-square, arbitrary-shape, and video-object holes with varied motion and geometry.
3.5. Inference
Inference assumes known inpainting masks for every video frame and recursively processes frames with a sliding window.
- The method assumes that inpainting masks are provided for all video frames.
- Masks are derived by dilating ground-truth DAVIS object-segmentation masks.DAVIS provides dynamic scenes, camera motion, motion blur, and large occlusions.
- Frames are processed recursively in a sliding-window manner.
3.6. Implementation Details
The implementation uses an older PyTorch/CUDA stack and runs at 12.5 fps for 256×256 frames on a GTX 1080 Ti GPU.
- 12.5 fps is achieved on a GPU for 256×256-pixel frames.The reported hardware includes an NVIDIA GTX 1080 Ti and an Intel Xeon 2.10GHz CPU.
- Training uses Adam with β = (0.9, 0.999), a fixed learning rate of 1e-4, and training from scratch.
- The two training stages each take about one day using four NVIDIA GTX 1080 Ti GPUs.
4. Experiments
Experiments analyze feature composition, temporal-stability components, video quality, user preferences, and applications to retargeting. The method aligns neighbor features, improves temporal consistency and perceptual quality, and produces natural retargeted sequences, while remaining subject to stated artifacts and resolution limits.
- 4.1. Visualization of Learned Feature Composition: The network aligns and integrates visible neighbor features to fill large, complex holes in reference frames.Feature-flow visualization shows source features borrowed from four streams and adjusted according to spatio-temporal semantics.
- 4.2. Improvement on Temporal Consistency: Recurrent feedback and ConvLSTM reduce the full model’s flow-warping error by a factor of 10.The comparison uses Sintel videos with ground-truth optical flow and DAVIS mask sequences.
- 4.2. Improvement on Temporal Consistency: The full model outperforms the compared baselines by large margins on flow-warping error, while Yu et al.’s frame-wise method is least temporally consistent.
- 4.3. Spatio-Temporal Video Quality: The method achieves the smallest FID among compared methods, indicating better visual quality and temporal consistency.FID is averaged over 20 DAVIS videos and five trials using spatio-temporal features from a pretrained I3D network.
- 4.4. User Study on Video Object Removal: In a 30-user study, participants preferred the proposed method more often across video object-removal samples.Participants judged both image quality and temporal consistency against Huang et al.’s optimization baseline.
- 4.5. Application to Video Retargeting: Video retargeting removes salient content through inpainting, resizes the background, and overlays the salient content afterward.Examples include horizontal and vertical shrinking while preserving the moving object’s original aspect ratio.
- 4.5. Application to Video Retargeting: Retargeted sequences exhibit little warble and jittering over time and appear natural.
- 4.6. Limitation: Large, long occlusions can produce color-saturation artifacts, propagated color discrepancy, blurry unrevealed regions, and a 256×256-pixel experimental limit.
5. Conclusion
The proposed feed-forward framework aggregates and aligns neighboring-frame features, using recurrent feedback and temporal memory to produce temporally coherent video inpainting. Experiments report superior visual quality to state-of-the-art image inpainting and favorable performance against an optimization method, while avoiding computation-heavy optimization.
- The model aggregates and aligns feature maps from neighboring frames to inpaint videos.
- Recurrent feedback and temporal memory encourage temporally coherent outputs.
- The method achieves superior visual quality to the state-of-the-art image inpainting solution.
- The method performs favorably against an optimization method in qualitative and quantitative experiments.
- A feed-forward design may avoid computation-heavy optimization and broaden applicability to related vision tasks.