Source-linked AI summary
Coherent Online Video Style Transfer
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, Gang Hua
TL;DR
Naively stylizing video frames independently can cause flickering and false discontinuities because small content perturbations produce unstable stylized results. The paper presents an end-to-end feed-forward network using short-term temporal coherence and propagation for online, near-real-time video style transfer, achieving thousands-fold speedups over optimization-based methods.
Problem
Naively processing video frames independently often produces flickering and false discontinuities because video style-transfer solutions are unstable.
Method
The paper introduces an end-to-end feed-forward network that models two-frame coherence and propagates short-term coherence along motion trajectories for long-term consistency.
Results
The network produces stable stylized videos online at 15 fps and is thousands of times faster than optimization-based video style transfer.
Takeaways & Limitations
The method can be used with existing stylization networks, including new styles without retraining, while supporting camera motion, object motion, and occlusions.
Takeaways & Limitations
The propagation approximation may suffer from shifting errors and inconsistency before and after occlusion, although obvious ghosting or flickering was not observed in practice.
Abstract
from arXiv · showhide
Training a feed-forward network for fast neural style transfer of images is proven to be successful. However, the naive extension to process video frame by frame is prone to producing flickering results. We propose the first end-to-end network for online video style transfer, which generates temporally coherent stylized video sequences in near real-time. Two key ideas include an efficient network by incorporating short-term coherence, and propagating short-term coherence to long-term, which ensures the consistency over larger period of time. Our network can incorporate different image stylization networks. We show that the proposed method clearly outperforms the per-frame baseline both qualitatively and quantitatively. Moreover, it can achieve visually comparable coherence to optimization-based video style transfer, but is three orders of magnitudes faster in runtime.
1. Introduction
Naively stylizing video frames independently causes flickering because small input changes can produce large output variations. The paper introduces an end-to-end feed-forward network that uses short-term temporal coherence and propagates it over time for efficient online video style transfer.
- Independent frame processing produces flickering and false discontinuities because subtle lighting, noise, and motion changes can cause strong stylization variations.
- The proposed network is the first end-to-end feed-forward system to leverage temporal information for video style transfer and generate stable results.
- The architecture estimates feature correspondences between consecutive frames and uses motion-discontinuity or occlusion masks to blend previous and current feature maps.
- Short-term coherence between two frames is propagated along motion trajectories to approximate long-term consistency while retaining efficiency.
- Thousands of times faster than optimization-based video style transfer, the feed-forward network reaches 15 fps on modern GPUs.
- The method is online, inexpensive in learning and inference, and compatible with both per-style-per-network and multiple-style-per-network stylization networks.
2. Related Work
Prior video stylization methods preserve temporal coherence through optical flow, temporal filtering, 3D extensions, or post-processing, while feed-forward stylization networks had not incorporated flow constraints. Deep optical-flow networks offer a route toward end-to-end integration, but stylization requires task-specific network design.
- Neural video style transfer methods use optical flow to maintain color and texture coherence along motion trajectories, with masks reducing artifacts near boundaries and occlusions.
- Traditional video filtering methods preserve coherence by temporally smoothing filter parameters or extending filters into the temporal domain.
- Post-processing approaches provide filter-agnostic temporal coherence, but can fracture texture patterns or introduce ghosting artifacts.
- Before this work, adding flow constraints to feed-forward stylization networks had not been investigated.
- Deep optical-flow methods such as FlowNet estimate motion directly with CNNs, but stylization requires specialized loss functions and feature composition.
3. Method
The method combines warped previous-frame features with current-frame features using learned flow and masking, preserving short-term coherence while propagating it over longer periods. It integrates these components into a pretrained image stylization network and trains them with temporal consistency objectives.
- Motivation: Independent frame stylization can amplify small appearance changes into strong flickering, motivating explicit temporal coherence.Such changes may be useful for spatially rich still-image patterns but destabilize video stylization.
- Two-frame temporal consistency: The method warps previous-frame features to the current frame, then adaptively fuses them with current-frame features to preserve traceable regions and avoid occlusion ghosting.Traceable regions reuse propagated features, while untraceable regions use newly computed current-frame features.
- System overview: The system processes video sequentially, using the first frame’s encoded features and stylized output to initialize processing from the second frame onward.At inference, composite features from the previous frame are reused instead of recomputing its encoded features.
- Network Architecture: The network contains a pretrained style encoder-decoder, a flow sub-network for intermediate feature flow, and a mask sub-network for composition.The default style network is based on Johnson et al., while FlowNet is used to estimate feature correspondences between consecutive frames.
- Network Architecture: The mask predicts where warped or current features should dominate: traceable regions favor warped features, whereas occlusions and false-flow regions favor current features.The mask is computed from the absolute difference between feature maps and is shared across feature channels during composition.
- The Loss Function: Training jointly uses coherence, occlusion, and flow terms to constrain stylized outputs and feature-flow estimation.The coherence term penalizes inconsistencies between consecutive stylized frames, while occlusion supervision keeps untraceable regions close to independently stylized results.
4. Experiments
The experiments evaluate the proposed online video style-transfer network on motion-rich videos, comparing stability, runtime, visual quality, transferability, and feature-composition choices. Across these tests, the method improves temporal coherence over per-frame baselines while remaining near real-time, with trade-offs involving sharpness, style transferability, and flow accuracy.
- Dataset Set-up: The training set contains around 28,000 frames from ten short videos, while MPI Sintel supplies 23 test videos with varied motions and ground-truth flow and occlusion masks.Bidirectional DeepFlow2 estimates provide approximate training flow, and occlusion or motion-boundary pixels are masked during composition.
- Quantitative and Qualitative Comparison: Our method achieves much lower stability error than per-frame baselines for all four styles, while running only 2.5–2.8× slower.The evaluation measures stability error on traceable regions and uses average error over each video.
- Quantitative and Qualitative Comparison: 15 fps on a Titan X keeps the proposed method near real-time despite extra flow and mask sub-network computation.The per-frame baseline is faster, but the proposed network adds temporal processing to improve coherence.
- Quantitative and Qualitative Comparison: Ours has slightly larger temporal coherence error than optimization-based video transfer, 0.0089 versus 0.0067, but runs thousands of times faster than 0.0089 fps.The comparison uses the Candy style; the optimization method is optimized for one video, whereas ours is trained for all videos.
- Quantitative and Qualitative Comparison: Compared with per-frame stylization, our method reduces flicker and false discontinuities in nearly static, camera-motion, and object-motion scenes, except in occluded regions.A separate post-processing comparison reports that the alternative produces less-clear results with ghosting artifacts.
- Ablation Study: Feature composition improves stability toward output layers but decreases sharpness, so the authors recommend integrating the sub-networks at the middle r1/4(E) layer.The middle layer balances stability and perceptual sharpness while supporting robust flow estimation and feature composition.
- Ablation Study: Fine-tuning the flow sub-network yields more temporally coherent results than using fixed FlowNet.The experiment compares fixed and fine-tuned flow sub-networks within the same video style-transfer framework.
- Ablation Study: Flow and mask sub-networks trained for one style are difficult to transfer while preserving stability, whereas multiple-style training supports two new styles without retraining.This suggests style independence for sub-networks trained with the multiple-style-per-network model.
5. Conclusion and Discussion
The paper concludes that its end-to-end temporal-coherence network enables online video style transfer with near-real-time speed and stable results. It achieves long-term coherence by propagating short-term coherence and remains limited by optical-flow errors, especially under large motion and long propagation.
- Conclusion: The system is presented as the first end-to-end video style-transfer network incorporating temporal coherence, with near-real-time speed and thousands-fold acceleration over optimization-based methods.It is designed for online processing on modern GPUs.
- Conclusion: Long-term temporal coherence is approximated by propagating short-term coherence, enabling online processing and compatibility with existing stylization networks.The method can also be directly used for new styles without retraining.
- Conclusion: The method produces stable stylized videos with camera motion, object motion, and occlusions.The conclusion states this outcome within the demonstrated scope of the system.
- Discussion: Large motions can cause incoherence when optical flow cannot track them, and small flow errors may accumulate into blurriness after long propagation.These limitations are attributed to the accuracy of the DeepFlow2-based ground-truth optical flow.