Source-linked AI summary
Learning Blind Video Temporal Consistency
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, Ming-Hsuan Yang
TL;DR
Per-frame video processing can flicker, while task-specific temporal extensions do not generalize broadly. The paper learns a deep recurrent network from original and processed videos, combining temporal and perceptual losses without optical flow at test time. The resulting single model supports multiple trained and unseen applications and performs favorably against existing blind temporal-consistency methods.
Problem
Independent frame processing often produces temporal inconsistency, while video extensions require task-specific knowledge and may not generalize to other applications.
Method
A deep recurrent network takes original and per-frame processed videos as inputs and minimizes short-term, long-term, and VGG perceptual losses.
Results
The single model handles multiple trained and unseen applications and performs favorably against existing methods across diverse videos.
Takeaways & Limitations
The approach provides task-agnostic temporal stabilization while preserving perceptual similarity to processed videos and operating without optical flow at test time.
Takeaways & Limitations
The method cannot handle applications that generate entirely different image content on each frame, such as image completion or synthesis.
Abstract
from arXiv · showhide
Applying image processing algorithms independently to each frame of a video often leads to undesired inconsistent results over time. Developing temporally consistent video-based extensions, however, requires domain knowledge for individual tasks and is unable to generalize to other applications. In this paper, we present an efficient end-to-end approach based on deep recurrent network for enforcing temporal consistency in a video. Our method takes the original unprocessed and per-frame processed videos as inputs to produce a temporally consistent video. Consequently, our approach is agnostic to specific image processing algorithms applied on the original video. We train the proposed network by minimizing both short-term and long-term temporal losses as well as the perceptual loss to strike a balance between temporal stability and perceptual similarity with the processed frames. At test time, our model does not require computing optical flow and thus achieves real-time speed even for high-resolution videos. We show that our single model can handle multiple and unseen tasks, including but not limited to artistic style transfer, enhancement, colorization, image-to-image translation and intrinsic image decomposition. Extensive objective evaluation and subject study demonstrate that the proposed approach performs favorably against the state-of-the-art methods on various types of videos.
1 Introduction
The paper frames video temporal consistency as a task-independent learning problem and proposes a recurrent network that stabilizes processed videos while preserving perceptual similarity. A single model handles multiple trained and unseen applications without optical flow at test time.
- Independent frame processing often causes temporal flickering, while existing video extensions face computational, memory, dataset, or task-specific limitations.The introduction also notes that flow-based temporal consistency can be difficult to extend across applications.
- The proposed method learns a deep recurrent network from original and per-frame processed videos to generate temporally stable output videos.The formulation treats temporal consistency as a learning task rather than embedding task-specific knowledge into each image-processing algorithm.
- The network combines short-term and long-term temporal losses with a VGG perceptual loss to improve stability while maintaining similarity to processed frames.A ConvLSTM layer captures spatial-temporal correlations in natural videos.
- The model processes arbitrary-length videos sequentially and avoids optical-flow computation at test time, reaching 400+ FPS on 1280 × 720 videos.Sequential processing supports online operation while the recurrent state carries information across frames.
- A high-quality dataset contains 80 training videos and 20 evaluation videos, covering trained applications and unseen tasks with one shared model.Unseen applications include intrinsic image decomposition and image-to-image translation.
- Objective and subjective evaluations report favorable performance against existing approaches across varied videos and applications.The evaluation considers temporal warping error, a learned perceptual metric, and user preferences.
2 Related Work
Prior work uses optical flow, temporal filtering, task-specific training, or assumptions about frame transformations to improve video consistency. The proposed recurrent method instead uses a sequential image transformation network trained with temporal and perceptual losses.
- Temporal consistency research spans automatic white balancing, harmonization, dehazing, enhancement, style transfer, colorization, image-to-image translation, and intrinsic decomposition.
- Task-specific approaches: Task-specific methods commonly use optical flow to propagate information between frames, but flow estimation is computationally expensive for high-resolution and long sequences.
- Task-independent approaches: Bonneel et al. and Yao et al. require dense correspondences, while Yao et al. also selects key frames from the entire video and cannot operate online.
- Task-specific approaches: Temporal filtering efficiently extends image algorithms to video but assumes a specific filter formulation and cannot generalize to other applications.
- CNN-based style-transfer methods jointly optimize content, style, and temporal warping losses, yet remain limited to styles represented during training.
- Task-independent approaches: Task-independent approaches assume gradient similarity, local affine transformation, segmentation consistency, or related formulations, but cannot handle more complex cases.
- The proposed recurrent network takes prior and current frames plus the processed frame, then feeds its output into the next time step.
3 Learning Temporal Consistency
The method learns temporal consistency with a recurrent network that combines perceptual, short-term, and long-term temporal objectives while processing videos sequentially. Its architecture uses residual prediction, ConvLSTM recurrence, and split input streams to preserve processed-frame appearance without transferring unwanted low-level information.
- 3.1 Recurrent network: The recurrent network takes original and per-frame processed videos as inputs and generates temporally consistent outputs sequentially for videos of arbitrary length.The first output frame is set to the first processed frame.
- 3.2 Loss functions: Training combines perceptual content loss with short-term and long-term temporal losses to balance temporal stability and similarity to processed frames.The overall loss weights the content perceptual, short-term, and long-term terms separately.
- 3.2 Loss functions: The perceptual loss compares output and processed frames using features from the fourth layer, relu4-3, of a pretrained VGG-19 network.The feature representation is used to measure perceptual similarity rather than relying only on pixel values.
- 3.2 Loss functions: Short-term loss measures warping error between neighboring output frames using optical flow and a visibility mask computed from the input frames.FlowNet2 computes flow during training, and bilinear sampling performs the warping.
- 3.2 Loss functions: Long-term loss compares the first output frame with all output frames, avoiding the cost and convergence issues of applying losses to every pair.Training enforces this coherence over a maximum of 10 frames.
- 3.3 Image transformation network: The transformation network predicts residuals over processed frames and uses ConvLSTM recurrence; split streams restrict skip connections to processed-frame inputs when appearances differ strongly.This design avoids transferring low-level information from unprocessed inputs that can create artifacts.
4 Experimental Results
The experiments evaluate the approach across diverse image-processing applications using high-quality video data and task-independent processing. The method is presented as applicable across tasks, including unseen applications, while addressing the flickering produced by frame-wise algorithms.
- 4.3 Evaluation: The experiments assess temporal stability and perceptual similarity using temporal warping error and a learned perceptual metric.The study also includes quantitative and subjective comparisons with existing approaches.
- 4.1 Dataset: The evaluation uses 80 additional high-quality videos for training and 20 for testing, complementing the shorter DAVIS-2017 videos.DAVIS-2017 contains 60 training and 30 validation videos, with 4,209 training frames in total.
- 4.2 Applications: The method is designed without assumptions about underlying image-processing algorithms, supporting a wide variety of applications.The paper addresses white balancing, harmonization, dehazing, enhancement, style transfer, colorization, translation, and intrinsic decomposition.
- 4.2 Applications: Frame-wise processing produces low-frequency flickering for colorization, high-frequency flickering for enhancement, and serious temporal artifacts for intrinsic decomposition.Style transfer and image-to-image translation are also described as applications where temporal consistency is challenging.
- 4.2 Applications: Training covers style transfer, enhancement, colorization, and shading-layer decomposition, while other applications are reserved for testing generalization.The training tasks use WCT style transfer, an enhancement model, Zhang et al.'s colorization method, and Bell et al.'s shading layer.
4.3 Evaluation metrics
The paper evaluates temporal stability using flow-based warping error and perceptual similarity using calibrated deep features, excluding the fixed first frame from perceptual-distance computation.
- Temporal stability: Temporal stability is measured by flow warping error between consecutive frames.The metric uses a warped next frame and evaluates non-occluded regions.
- Temporal stability: The video-level warping error is computed as the average warping error over the entire sequence.
- Perceptual similarity: Perceptual distance between processed and output videos is measured using calibrated SqueezeNet features.The calibrated feature model is denoted G.
- Perceptual similarity: The first frame is excluded from perceptual-distance computation because it is fixed as a reference.
4.4 Analysis and discussions
The analysis examines the trade-off between temporal stability and perceptual similarity, finding that the loss-weight ratio is important and that r = 10 provides a good balance.
- Balancing temporal stability and perceptual similarity is necessary because extreme blurring and unprocessed results optimize only one property.
- The ratio r = λt/λp controls the balance between temporal flickering and perceptual similarity.When r < 10, perceptual loss dominates and temporal flickering remains.
- r = 10 achieves a good balance between reducing temporal warping error and perceptual distance when λt is sufficiently large.
- On style-transfer results, the proposed method maintains the stylized effect while reducing temporal flickering compared with Bonneel et al. [6].
4.5 Comparison with state-of-the-art methods
Across multiple applications, the proposed method achieves temporal stability comparable to Bonneel et al. while producing lower perceptual distance and preserving processed-video content more effectively.
- Across 16 applications, the method is evaluated using temporal warping error and perceptual distance on two video test sets.The applications include style transfer, enhancement, intrinsic decomposition, image-to-image translation, and colorization.
- The method has lower perceptual distance than Bonneel et al..
- Bonneel et al. can overly smooth stylized content and fail under large-region occlusion, whereas the proposed method reduces flickering while preserving processed-video similarity.
- The method achieves similarly reduced temporal warping error to Bonneel et al. [6], substantially below the original processed video Vp.
4.6 Subjective evaluation
A pairwise user study on held-out applications found that participants preferred the proposed results across all five evaluated applications, mainly because they preserved processed-video effects.
- The user study uses pairwise comparisons between Bonneel et al. and the proposed method, with original and processed videos as references.
- The study evaluates 50 test videos from 10 applications held out during training, using responses from 60 subjects.
- The proposed method is preferred on all five applications shown in the user-study results.
- Users select Bonneel et al. for temporal stability but prefer the proposed results because they preserve the processed-video effect better.
4.7 Execution time
The proposed method achieves real-time execution at high resolution without computing optical flow at test time, substantially exceeding Bonneel et al.’s reported speed.
- 418 FPS on GPU for 1280 × 720 videos, compared with 0.25 FPS for Bonneel et al. on CPU.Both measurements were obtained on the reported CPU/GPU evaluation machine.
- The method’s execution speed is enabled by avoiding optical-flow computation at test time.
- The reported speed supports real-time processing of high-resolution videos.
4.8 Limitations and discussion
The approach has clear scope boundaries and an inherent trade-off between temporal coherence and perceptual similarity, while receiving favorable subjective evaluation and handling occlusion better than a cited baseline.
- The approach cannot handle applications that generate entirely different image content on each frame, such as image completion or synthesis.Extending such applications would likely require strong video priors or temporal constraints within the task-specific algorithms.
- The task formulation always trades temporal coherence against perceptual similarity to the processed video.Depending on the applied effect, flicker may be preferable to blur, or blur may be preferable to flicker.
- Bonneel et al. [6] cannot preserve the colorized effect when occlusion occurs.
- 62% of users preferred the proposed method on average in the subjective evaluation.The error bars represent the 95% confidence interval.
5 Conclusions
The paper proposes a deep recurrent network for reducing temporal flickering while preserving perceptual similarity. It is agnostic to the underlying image-processing algorithm, generalizes to unseen applications, and performs favorably against existing blind temporal-consistency methods.
- The proposed deep recurrent neural network reduces temporal flickering in per-frame processed videos.
- The method jointly optimizes short-term and long-term temporal losses with a perceptual loss.This is intended to reduce temporal instability while preserving similarity to processed videos.
- The approach is agnostic to the underlying image-based algorithms and generalizes to a wide range of unseen applications.
- The proposed algorithm performs favorably against existing blind temporal-consistency methods across diverse applications and video types.