Source-linked AI summary
Blind Video Temporal Consistency via Deep Video Prior
Chenyang Lei, Yazhou Xing, Qifeng Chen
TL;DR
Applying image-processing algorithms independently to video frames can produce temporal inconsistency. The paper trains a CNN on a single video using the Deep Video Prior and an iteratively reweighted strategy, and reports the best average scores on temporal-consistency and data-fidelity metrics.
Problem
Applying image-processing algorithms independently to video frames often produces undesirable temporal inconsistency, such as flickering.
Source: §1 · p. 1Method
The method trains a CNN on a single video using the Deep Video Prior and iteratively reweighted training to address multimodal inconsistency.
Source: §1 · p. 2Results
The method obtains the best average scores on both temporal-consistency and data-fidelity metrics, with qualitative results reporting better multimodal consistency than baselines.
Source: §4.3 · pp. 6–7Takeaways & Limitations
The framework offers a way to apply existing image-processing and enhancement algorithms to videos with strong temporal consistency, using a CNN prior in place of complicated handcrafted regularization.
Source: Broader Impact · p. 9
Abstract
from arXiv · showhide
Applying image processing algorithms independently to each video frame often leads to temporal inconsistency in the resulting video. To address this issue, we present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. Unlike most previous methods that enforce temporal consistency with optical flow, we show that temporal consistency can be achieved by training a convolutional network on a video with the Deep Video Prior. Moreover, a carefully designed iteratively reweighted training strategy is proposed to address the challenging multimodal inconsistency problem. We demonstrate the effectiveness of our approach on 7 computer vision tasks on videos. Extensive quantitative and perceptual experiments show that our approach obtains superior performance than state-of-the-art methods on blind video temporal consistency. Our source codes are publicly available at github.com/ChenyangLEI/deep-video-prior.
1 Introduction
Framewise image processing can cause temporal inconsistency, while task-specific video solutions do not readily transfer. This paper proposes a general framework trained on a single video, with iteratively reweighted training to address multimodal inconsistency.
1 Introduction
- Task-specific video algorithms can improve temporal coherence, but similar strategies are unclear or challenging to apply to other tasks.The paper motivates a generic framework that converts image-processing algorithms into temporally consistent video counterparts.Source: §1 · p. 1
- Prior work [7] [19] studies general frameworks for temporal consistency; most approaches [19] use dense correspondence, while assume gradient-domain similarity that may not hold in practice.Long-term temporal consistency often degrades in these correspondence-based approaches [19].Source: §1 · p. 1
- The CNN is trained on video so outputs for corresponding patches become consistent, without handcrafted temporal regularization.The Deep Video Prior allows most video information to be recovered before flickering artifacts are eventually overfit.Source: §1 · p. 2
- Training uses only the single test video, so no large training dataset is required.The authors contrast this internal-learning setup with methods trained on large-scale data.Source: §1 · p. 2
- Across diverse computer-vision tasks, the method reports better temporal consistency and less performance degradation than current state-of-the-art methods.The authors describe iteratively reweighted training as addressing multimodal inconsistency while preserving perceptual quality.Source: §1 · p. 2
2 Background
The background defines processed frames as outputs of an image operator applied independently to input frames. It distinguishes within-mode variation from switching between multiple possible output modes, then contrasts correspondence-based regularization with direct training on a test video.
2 Background
- A processed frame is obtained by applying an image-processing operator f to input frame I_t, and blind video temporal consistency seeks consistent outputs O_t.The operator can be colorization, dehazing, or another image-processing algorithm.Source: §2 · p. 2
- Unimodal inconsistency is variation within one output mode, whereas multimodal inconsistency occurs when similar inputs map to different possible modes.For colorization, the same car may plausibly be colored red or blue.Source: §2 · p. 2
- Prior methods [19] use correspondences such as optical flow or PatchMatch to regularize similar features across output frames.Their objectives commonly combine correspondence regularization with a data term; the described flow-based regularizer is unnecessary in this model.Source: §2 · pp. 2–3
- The proposed pipeline trains a network directly on the test video, using one frame per iteration to mimic the image operator.The network is not restricted to a specific type, allowing the pipeline to adapt to different tasks.Source: §2 · p. 3
3 Method
Deep Video Prior (DVP) uses single-video CNN training to obtain temporal consistency, while iteratively reweighted training (IRT) addresses multimodal inconsistency by selecting a main mode and excluding outliers.
3.1 Deep Video Prior (DVP)
- DVP implicitly encourages corresponding patches across video frames to have consistent CNN outputs, recovering video information before flickering artifacts are overfit.The prior relies on similar patches receiving similar CNN outputs early in training.Source: §3.1 · p. 3
- The framework trains a randomly initialized fully convolutional network on one video, using one frame per iteration and a data term matching its output to the processed frame.The implementation uses U-Net and perceptual loss, and stops before artifacts such as flickering are overfit.Source: §3.1 · p. 3
- For multimodal inconsistency, DVP outputs initially tend to agree but later separate and overfit the processed frames.The basic architecture can yield outputs that are consistent at 100 iterations but not close to either ground-truth mode.Source: §3.1 · p. 4
3.2 Iteratively Reweighted Training (IRT)
- With IRT, outputs at the 100th and 200th iterations are consistent and close to one ground-truth mode.The toy example contrasts this with outputs that are consistent but not close to either ground-truth mode without IRT.Source: §3.1 · p. 4
- IRT uses a pixel-wise confidence map to select a main mode and ignore outliers among multiple modes.Confidence is higher when the processed pixel is closer to the main output than to the other modes; the confidence map updates the training loss.Source: §3.2 · pp. 4–5
4 Experiments
The experiments evaluate the framework across seven video-processing tasks against prior methods, using temporal-consistency and data-fidelity metrics alongside perceptual comparisons. Results favor the method overall, while analyses examine IRT, network architectures, training choices, and cost.
4.1 Experimental setup
- The evaluation spans seven tasks, using the Bonneel et al. test set for dehazing and white balancing and DAVIS's 30 videos for the other applications; baselines are Bonneel et al. and Lai et al.Tasks include colorization, dehazing, enhancement, style transfer, image-to-image translation, intrinsic decomposition, and white balancing; training uses no separate dataset or optical-flow estimation.Source: §4 · p. 5
4.2 Evaluation metrics
- Ewarp averages warping error against adjacent and first frames; Fdata measures similarity to processed frames, not perceptual quality, and excludes frame one for fair baseline comparison.The warping error uses an occlusion map and backward optical-flow warping.Source: §4.2 · p. 6
4.3 Results
- The method achieves the best average scores on both temporal-consistency and data-fidelity metrics; Lai et al. have comparable fidelity but weaker temporal consistency, while Bonneel et al. suffer degradation on some tasks.The numerical metrics do not completely reflect visual quality.Source: §4.3 · pp. 6–7, Table 2 · p. 6
- Qualitatively, the method maintains short- and long-term consistency in a single mode; Lai et al. retain first-to-last drift, while Bonneel et al. reduce flicker but alter colors undesirably.The examples cover colorization and white balancing, and the mean-intensity plots show the same short- versus long-term distinction.Source: §4.3 · p. 7, Figure 5 · p. 7, Figure 8 · p. 8
- In a study of 107 videos rated by 20 subjects, the method is preferred over baselines and processed videos in most tasks, with a larger advantage for multimodal inconsistency.For simple unimodal enhancement, the difference from Lai et al. is minor.Source: §4.3 · p. 7, Table 3 · p. 7
4.4 Analysis
- With IRT, Fdata is 28.20 versus 29.61 for the basic architecture on DAVIS colorization, while the IRT results are perceptually more consistent.Figure 6 compares IRT with the basic training architecture in the multimodal case.Source: §4.4 · p. 8, Figure 6 · p. 8
- Deep video prior also works with FCN and ResUnet: all three architectures are temporally more consistent than processed frames, with only minor differences among them.The comparison is reported in Figure 7.Source: §4.4 · p. 8, Figure 7 · p. 8
- Training trades off temporal consistency against data fidelity; the authors use 25 or 50 epochs across videos despite duration and motion differences, reporting about 100 seconds for a 50-frame video.For that runtime example, training uses 25 epochs at 80 ms per iteration on an RTX 2080 Ti; a lightweight CNN may speed processing.Source: §4.4 · pp. 8–9
5 Discussion
The paper uses a CNN trained on a single video to improve temporal consistency, with iteratively reweighted training addressing multimodal inconsistency. It also notes longer test-time processing and proposes future extensions to multi-view images and 3D data.
5 Discussion
- The approach is simple and general, and produces better temporal consistency for videos processed by image operators.The authors also report that iteratively reweighted training addresses multimodal inconsistency.Source: §5 · p. 9
- Training a CNN from scratch on a single video uses the Deep Video Prior: corresponding patches in video frames should have consistent outputs.The iteratively reweighted training strategy addresses multimodal inconsistency.Source: §5 · p. 9
- The method has relatively long test-time processing because it trains an individual model for each video, taking more time than direct inference compared with Lai et al. [19].The authors contrast this with methods that explicitly use optical flow, demonstrating temporal consistency implicitly through neural-network training.Source: §5 · p. 9
- The authors propose improving efficiency and extending the Deep Video Prior to multi-view images and 3D volumetric data.They note that the prior does not rely on frame ordering and suggest that 3D CNNs may exhibit a similar property.Source: §5 · p. 9
Broader Impact
The authors present the Deep Video Prior as a way to apply existing image-processing and enhancement algorithms to videos with strong temporal consistency. They also describe it as an implicit CNN-training characteristic that can replace complicated handcrafted regularization.
Broader Impact
- The framework lets users apply existing image-processing and enhancement algorithms to videos with strong temporal consistency.The authors connect this capability to using recent image-processing algorithms directly on video.Source: Broader Impact · p. 9
- The authors argue that avoiding extra effort to solve temporal inconsistency lets researchers spend more time using video information to improve performance.Source: Broader Impact · p. 9
- The Deep Video Prior is described as an implicit characteristic of CNN training on video that can replace complicated handcrafted regularization.The authors suggest this observation may inspire better use of CNNs for video-processing tasks.Source: Broader Impact · p. 9