Source-linked AI summary
Copy-and-Paste Networks for Deep Video Inpainting
Sungho Lee, Seoung Wug Oh, DaeYeun Won, Seon Joo Kim
TL;DR
Video inpainting must fill missing regions while maintaining temporal consistency across frames. Copy-and-Paste Networks align reference frames and match their content to fill target-frame holes. The authors report results comparable to the state of the art, faster runtime, and improved lane-detection accuracy after exposure enhancement.
Problem
Video inpainting must complete holes in every frame while maintaining temporal consistency between reconstructed frames.
Source: §1 · p. 1Method
Copy-and-Paste Networks align reference frames and use context matching to select pixels for pasting into target-frame holes, updating references with prior completions.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Enhancing over- and under-exposed video frames with the framework was reported to significantly improve lane-detection accuracy.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We present a novel deep learning based algorithm for video inpainting. Video inpainting is a process of completing corrupted or missing regions in videos. Video inpainting has additional challenges compared to image inpainting due to the extra temporal information as well as the need for maintaining the temporal coherency. We propose a novel DNN-based framework called the Copy-and-Paste Networks for video inpainting that takes advantage of additional information in other frames of the video. The network is trained to copy corresponding contents in reference frames and paste them to fill the holes in the target frame. Our network also includes an alignment network that computes affine matrices between frames for the alignment, enabling the network to take information from more distant frames for robustness. Our method produces visually pleasing and temporally coherent results while running faster than the state-of-the-art optimization-based method. In addition, we extend our framework for enhancing over/under exposed frames in videos. Using this enhancement technique, we were able to significantly improve the lane detection accuracy on road videos.
1. Introduction
Video inpainting must fill missing regions while preserving temporal consistency, and this work uses information from other frames to address that challenge. Its Copy-and-Paste Networks align and match reference-frame content for completion, with reported results comparable to state-of-the-art methods and benefits for lane detection after exposure enhancement.
1. Introduction
- Video inpainting is challenging because each frame needs completion and the resulting frames must remain temporally consistent.The introduction contrasts this challenge with image inpainting’s established deep-learning progress.Source: §1 · p. 1
- Copy-and-Paste Networks use valid pixels from other video frames to fill target-frame holes, turning video’s additional temporal information into a source for completion.The network learns to copy necessary pixels from reference frames and paste them into holes in the current frame.Source: §1 · p. 1
- Progressively updating reference frames with prior inpainted results supports temporal consistency across the completed video.The input video’s reference information is updated at each step.Source: §1 · p. 2
- The method’s results are comparable to the state-of-the-art method and outperform other deep-learning approaches [13] [24]; exposure enhancement significantly increases lane-detection accuracy in saturated/under-exposed images.It produces visually pleasing completed videos and runs much faster than the state-of-the-art method.Source: §1 · p. 2
- A self-supervised alignment network estimates affine matrices between images containing large holes, while context matching selects valuable reference-frame pixels based on image similarity.The aligned reference information is used to paste and complete the target image.Source: §1 · p. 2
2. Related works
Related work spans image inpainting methods and video approaches that must balance access to information across frames with temporal consistency. The paper positions its method as a faster feed-forward alternative that can use distant-frame content and achieve quality comparable to the state of the art.
2.1. Image Inpainting
- Traditional PatchMatch can struggle with complex scenes or insufficient source information, while later deep methods report strong restoration and introduce contextual attention and mask-aware convolutions [18] [24] [15] [23].The cited image methods include partial and gated convolutions; Liu et al. [15] also used perceptual and style losses without adversarial loss to correct blur.Source: §2.1 · p. 2
2.2. Video Inpainting
- Earlier video methods used frame alignment, optical flow, or spatio-temporal patches to maintain temporal consistency, but long-term consistency remained difficult and optimization-heavy methods were slow.Newson et al.’s 3D PatchMatch had limitations for long-term consistency in complicated scenes; Huang et al. proposed optical-flow optimization in spatial patches.Source: §2.2 · p. 2
- Prior deep video methods used 3D encoder-decoder or recurrent feedback and memory, but narrow temporal referencing limited access to distant-frame pixels in some scenes.The cited work [13] was described as having difficulty with large or slowly moving objects.Source: §2.2 · p. 2
- The proposed method reconstructs general scenes using affine alignment and context matching to access valid pixels in distant frames, with quality comparable to and faster feed-forward runtime.Its framework comprises alignment, copy, and paste subnetworks.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
3. Copy-and-Paste Network Algorithm
The algorithm completes each target frame by aligning reference frames, matching useful reference content, and decoding it into missing regions. Completed frames are reused as references, and the video is processed in both temporal directions to support temporal consistency.
3. Copy-and-Paste Network Algorithm
- For each target frame, the system aligns references, selects useful content through context matching, decodes the completed frame, and adds it to memory for later references.The video is processed frame by frame in temporal order.Source: §3 · p. 3
3.1. Alignment Network
- Affine alignment expands temporal search beyond optical flow; a self-supervised L1 loss over mutually visible pixels trains alignment jointly with the other networks end to end.More distant references can provide useful information when their hole regions overlap less.Source: Figure 3 · p. 3, §3.1 · p. 4
3.2. Copy-and-Paste Network
- Context matching scores aligned reference features against target features, then uses similarity-derived pixel weights and visibility masks to aggregate reference information.The global feature similarity is computed as cosine similarity while excluding hole pixels.Source: §3.2 · pp. 4–5
- The decoder combines target features, aggregated reference features, and Cmask; dilated convolutions broaden its receptive field to synthesize pixels never visible in reference frames.Cmask marks pixels that are never visible throughout the reference frames.Source: §3.2 · p. 5
3.3. Temporal Consistency
- Completed frames replace their references during sequential processing, and a second pass in reverse order is used to further ensure temporal consistency.The paper reports that iterative reference updates improve restored-image quality and temporal consistency.Source: §3.3 · p. 5
4. Training
Training combines losses for distinct regions of completed frames with a synthesized video dataset and end-to-end network training.
4.1. Loss functions
- The completed target frame is compared with ground truth using separate losses for hole and non-hole regions, with hole losses split by copyability.All networks are trained jointly end-to-end.Source: §4.1 · p. 5
- Perceptual, style, and total-variation losses supplement the regional losses; total variation is used to smooth checkerboard effects.The loss weights were empirically determined.Source: §4.1 · p. 5
4.2. Datasets
- Because no public video-inpainting dataset existed, the authors synthesized training sequences by compositing background image sequences with moving object masks.Backgrounds came from transformed Places images or sampled YouTube scenes, and masks came from MIT Saliency Benchmark and Pascal VOC 2012.Source: §4.2 · pp. 5–6, Figure 6 · p. 6
4.3. Training Details
- Training used five 256 × 256 frames per sample, batch size 40, Adam at 10^-4, and took about seven days on three TITAN XP GPUs.The learning rate was reduced by a factor of 10 every million iterations.Source: §4.3 · p. 6
5. Experiments
The experiments evaluate video restoration and object removal on DAVIS sequences, including quantitative comparisons, qualitative review, and a user study, as well as exposure restoration on road footage. Results were comparable to optimization-based, while lane detection improved after exposure restoration.
5.1. Quantitative Results
- On 25 DAVIS sequences, PSNR and SSIM for video restoration were similar to.VINet [13] was excluded because its official code had not been published.Source: Table 1 · p. 6, §5.1 · p. 6
5.2. User Study and Qualitative Analysis
- 27.14 seconds per video versus ’s 952 seconds; among 40 participants, our method was highly competitive, though was slightly more favored.The user study asked participants to rank video-completion results and allowed ties.Source: Table 2 · p. 6, §5.2 · p. 6
- For dynamic object removal, qualitative comparisons used 30 DAVIS videos and showed results comparable to and better than [13].The comparison scenes shown are elephant and tennis.Source: §5.2 · pp. 6–7, Figure 7 · p. 7
5.3. Applications
- Lane-detection results significantly improved after exposure restoration on 469 video frames with rapid exposure changes.The restoration also improved texture and color.Source: Table 3 · p. 7, §5.3 · p. 7
6. Ablation Study
The ablations examine masked softmax and iterative reference updating, reporting sharper outputs and improved temporal consistency, respectively.
Masked softmax
- Masked softmax produced sharper results than normal softmax in the ablation study.The methods were trained under the same conditions.Source: §6 · p. 7, Figure 10 · p. 8
Reference update
- Updating past reference frames with inpainted results enhanced temporal consistency in the temporal-profile comparison.The figure compares temporal profiles along the red line in the input.Source: §6 · p. 8
7. Conclusion
The framework completes missing video content by copying and pasting from reference frames, which are dynamically updated using previous completions. Experiments support its comparison with optimization-based methods and its use for exposure restoration to improve lane detection.
7. Conclusion
- The framework copies and pastes content from reference frames, which are dynamically updated with previous completions to ensure temporal consistency.Source: §7 · p. 8
- Experiments found the framework comparable to optimization-based methods, better than other deep-learning approaches, and able to significantly increase lane-detection accuracy after exposure restoration.Source: §7 · p. 8