Source-linked AI summary
Onion-Peel Networks for Deep Video Completion
Seoung Wug Oh, Sungho Lee, Joon-Young Lee, Seon Joo Kim
TL;DR
Video completion needs to preserve temporal coherence by using content visible in other frames, and reference-guided image completion also remains difficult for prior methods. The paper introduces a recurrent onion-peel network with non-local attention; it reports competitive video-completion quality at a fraction of the time and applicability to reference-guided image completion without modification.
Problem
Video completion must maintain temporal coherence by drawing on content visible in other frames, while prior methods have difficulty with reference-guided image completion.
Source: §1 · pp. 1–2Method
Results
The method achieves video-completion quality competitive with state-of-the-art optimization methods while running in a fraction of the time.
Source: §5 · p. 8Takeaways & Limitations
The framework also applies without modification to image completion guided by reference frames.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We propose the onion-peel networks for video completion. Given a set of reference images and a target image with holes, our network fills the hole by referring the contents in the reference images. Our onion-peel network progressively fills the hole from the hole boundary enabling it to exploit richer contextual information for the missing regions every step. Given a sufficient number of recurrences, even a large hole can be inpainted successfully. To attend to the missing information visible in the reference images, we propose an asymmetric attention block that computes similarities between the hole boundary pixels in the target and the non-hole pixels in the references in a non-local manner. With our attention block, our network can have an unlimited spatial-temporal window size and fill the holes with globally coherent contents. In addition, our framework is applicable to the image completion guided by the reference images without any modification, which is difficult to do with the previous methods. We validate that our method produces visually pleasing image and video inpainting results in realistic test cases.
1. Introduction
Video completion must preserve temporal coherence by using content visible in other frames, yet deep methods have been limited and flow-based approaches struggle with occlusions or large holes. The paper proposes a recurrent onion-peel network with non-local attention and reports competitive video-completion quality with faster computation, while also supporting reference-guided image completion.
1. Introduction
- Video completion must preserve temporal coherence, since content hidden in one frame may be visible in potentially distant frames and ignoring it can break consistency.The introduction notes that directly extending image-inpainting networks with 3D convolutions or recurrent networks is difficult because their temporal receptive fields are limited or directional.Source: §1 · p. 1
- The Onion-Peel Network fills a hole one boundary layer at a time, gaining richer context at each recurrence and enabling large-hole completion with sufficient recurrences.The network uses sampled video frames as references, taking visible content from them or synthesizing coherent content when no reference contains the missing region.Source: §1 · p. 2
- Non-local asymmetric attention matches target peel pixels to non-hole reference pixels, providing an unlimited spatial-temporal window without optical flow.The paper says this supports globally coherent completion and handles challenging occlusions or large holes that are difficult to model with flow.Source: §1 · p. 2
- Reference-guided image completion uses multiple photographs of a scene to remove unwanted objects while preserving original content, a setting previous video methods struggle with when frames are distant.The authors describe the task as useful for image editing with photos taken from different angles and at different times.Source: §1 · p. 2
- The authors report video-completion quality comparable to state-of-the-art optimization methods while running in a fraction of the time.The contribution summary also characterizes the results as comparable to state-of-the-art methods with fast computational time.Source: §1 · p. 2
2. Related Work
Related work spans example-based and learning-based image completion, as well as optimization-based and data-driven video completion. The paper positions its method as reference-based multi-image completion with feed-forward speed and non-local matching for a global temporal window.
2.1. Image Completion.
- Example-based image completion reconstructs missing regions from information outside them, while learning-based methods synthesize missing content and may use partial or gated convolutions.The paper uses gated convolution in its network.Source: §2.1 · p. 2
- The paper distinguishes its multi-image completion task by drawing example information from reference images rather than from the target image itself.Figure 2 presents the network overview and identifies gated convolution as its basic building block.Source: §2.1 · p. 2, Figure 2 · p. 3
2.2. Video Completion
- Video completion work includes global spatio-temporal patch optimization with dense optical-flow estimation, but execution time is extremely long because of computational complexity.Deep approaches combine 3D and 2D CNNs, or use an image-based encoder-decoder with optical flow to aggregate information from nearby frames [30] [17].Source: §2.2 · pp. 2–3
- The authors report their feed-forward method runs an order of magnitude faster than global optimization and is more flexible with unseen backgrounds.They also argue non-local pixel matching provides a global temporal window, unlike methods limited by their spatial-temporal windows.Source: §2.2 · pp. 2–3
3. Onion-peel Network
The onion-peel network recursively fills only the target hole’s boundary using valid reference-image regions, then repeats until the hole is filled. Its attention, training, and temporal-stabilization components support reference-guided completion of images and videos.
3. Onion-peel Network
- Each recurrence fills the target’s peel from valid reference regions, using non-local key matching to retrieve reference values and update the target features before decoding.The peel is the hole boundary; the target and references are embedded as key and value feature maps through a shared encoder.Source: §3 · pp. 3–4
3.1. Inference
- Video inference samples reference frames and processes target frames sequentially, repeating peel filling until each hole is completely filled.The peel is defined by a distance threshold of p = 8 from the nearest non-hole pixel; reference frames are encoded once because they remain unchanged.Source: §3.1 · p. 4
3.2. Network Design
- The asymmetric attention block matches target peel keys to valid reference keys, retrieves reference values by softmax-weighted similarity, and adds them at the target peel positions.Indexing the peel and valid regions reduces the matching matrices; the decoder reconstructs the peel while retaining the target’s non-hole region.Source: §3.2 · pp. 4–5
3.3. Training
- Training uses one target and four same-scene references, synthetic image and object-mask data, and pixel and perceptual losses at each recurrence.Training images come from Places2 and collected YouTube clips; losses are computed on decoder outputs and on outputs merged with the target’s non-hole region.Source: §3.3 · pp. 5–6
3.4. Video Temporal Consistency
- A separate temporal-consistency network post-processes frame-by-frame video completions to stabilize inpainted content while balancing temporal stability and current-frame perceptual similarity.The stabilizer uses an encoder-decoder with a convolutional GRU.Source: §3.4 · p. 6
4. Experiments
Experiments evaluate reference-guided image completion and video completion, then analyze onion peeling and temporal post-processing. The reported results include best quantitative accuracy on the video measures, user-study ranks close to, and improved handling of large image holes through peeling.
4.1. Reference-guided Image Completion
- With no standard deep-learning test data for reference-guided image completion, the authors built photo groups from Youtube-VOS, sampling five images per scene: one target and four references.The photos show the same scene from different camera angles and times and are annotated with undesired objects.Source: §4.1 · p. 7
- On large reference-guided holes, prior methods show artifacts; Yu et al. hallucinates inconsistent content and Content aware fill sometimes pastes wrong patches, while onion peeling handles large holes.The image-completion comparison uses a single-image method and an example-based method adapted to use the reference frames.Source: §4.1 · p. 7
4.2. Video Completion
- Across 24 DAVIS videos, 100 raters per video ranked OPN close to (with a slight edge to) and above VINet [17]; OPN runs more than 50 times faster than.Figure 7 presents per-video ranks, and Figure 8 shows sample comparisons on DAVIS.Source: §4.2 · p. 7, Figure 7 · p. 7, Figure 8 · p. 8
- Our method achieves the best accuracy on PSNR, SSIM, and VFID among the compared methods.Table 1 also reports execution time per video.Source: §4.2 · pp. 7–8, Table 1 · p. 8
4.3. Analysis
- The one-shot variant runs about 3× faster than onion peeling with p=8, but has lower output quality and often fails on large holes.Figure 9 illustrates the effect of onion peeling.Source: §4.3 · p. 8, Figure 9 · p. 8
- Temporal-consistency post-processing smooths temporal profiles and makes results more temporally coherent, but tends to blur video frames.Figure 10 shows results before and after post-processing.Source: §4.3 · p. 8, Figure 10 · p. 8
5. Conclusion
The method provides competitive video-completion quality relative to an optimization-based method while running in a fraction of its time. It also applies without modification to reference-guided image completion.
5. Conclusion
- For video completion, the method shows competitive quality to the state-of-the-art optimization-based method.The conclusion describes its runtime as a fraction of that method’s time.Source: §5 · p. 8
- The method runs in a fraction of the time required by the optimization-based method.The conclusion pairs this runtime advantage with competitive video-completion quality.Source: §5 · p. 8
- Without modification, the network applies to image completion guided by reference frames, a setting the conclusion says existing methods hardly achieve.Source: §5 · p. 8