Source-linked AI summary
Artistic style transfer for videos
Manuel Ruder, Alexey Dosovitskiy, Thomas Brox
TL;DR
The paper addresses flickering and discontinuities when transferring an artistic style to video frames independently. It introduces temporal constraints, long-term motion handling, multi-pass processing, and suitable initialization, producing stable and coherent stylized videos that outperform simpler baselines qualitatively and quantitatively.
Problem
Independent framewise style transfer produces flickering and false discontinuities because the optimization is unstable across frames.
Method
The method combines video-specific initialization with short- and long-term temporal losses based on optical flow and a multi-pass algorithm using forward and backward flow.
Results
The approach successfully eliminates most temporal artifacts and creates smooth, coherent stylized videos, outperforming simpler baselines qualitatively and quantitatively.
Takeaways & Limitations
The techniques produce stable and visually appealing stylized videos even with fast motion and strong occlusion.
Takeaways & Limitations
Performance strongly depends on optical-flow quality, and the short-term benchmark cannot evaluate long-term consistency or visual quality.
Abstract
from arXiv · showhide
In the past, manually re-drawing an image in a certain artistic style required a professional artist and a long time. Doing this for a video sequence single-handed was beyond imagination. Nowadays computers provide new possibilities. We present an approach that transfers the style from one image (for example, a painting) to a whole video sequence. We make use of recent advances in style transfer in still images and propose new initializations and loss functions applicable to videos. This allows us to generate consistent and stable stylized video sequences, even in cases with large motion and strong occlusion. We show that the proposed method clearly outperforms simpler baselines both qualitatively and quantitatively.
1 Introduction
The paper extends neural style transfer from still images to videos by adding temporal constraints and video-specific initialization strategies. These techniques reduce temporal artifacts and produce smoother, more coherent stylized sequences despite motion and occlusion.
- Motivation: Neural style transfer separates image content from artistic style using deep-network features and Gram-matrix correlations.The approach builds on Gatys et al., whose optimization matches content activations and style feature correlations.
- Video extension: Independent frame processing causes flickering and false discontinuities because style-transfer solutions are unstable.The paper addresses this instability by regularizing transitions between frames.
- Video extension: A temporal constraint penalizes deviations along optical-flow trajectories while excluding disoccluded regions and motion boundaries.This preserves existing appearance while allowing newly visible regions to be rebuilt.
- Extensions: Long-term motion estimates enforce consistency between synthesized frames before and after an occlusion.Without this extension, rebuilt regions may acquire a different appearance after being temporarily occluded.
- Extensions: A multi-pass algorithm alternates forward- and backward-flow processing to reduce boundary artifacts amplified by strong camera motion.The authors report that this produces a more coherent video.
- Evaluation: The method eliminated most temporal artifacts and created smooth, coherent stylized videos in quantitative Sintel evaluations and qualitative movie-shot results.The evaluation used different optical-flow algorithms and included qualitative comparisons on several movie shots.
2 Related work
Prior work developed neural style transfer for still images and optical-flow-based temporal consistency for painted animations. The paper positions its contribution as applying deep-network style transfer to video sequences.
- Style transfer using deep networks: Gatys et al. introduced deep-network style transfer by matching high-level content features and style representations in still images.Later work modified how style is represented, including local activation patches and alternative neural networks.
- Style transfer using deep networks: Later neural style-transfer studies pursued goals such as preserving local style patterns, illumination transfer, and season transfer.These approaches varied the network or style representation rather than extending the method to videos.
- Research gap: The authors state that they are unaware of prior work applying this kind of neural style transfer to videos.This distinguishes their video extension from earlier still-image and brush-stroke animation approaches.
- Painted animations: Painted-animation methods create brush strokes and use optical flow to move them through subsequent frames for temporal consistency.Later work refined brush-stroke parameters and placement methods.
3 Style transfer in still images
The still-image foundation formulates stylization as minimizing content and style losses computed from convolutional-network representations. The non-convex optimization makes initialization important, particularly for video frames.
- Objective: The still-image objective generates a stylized image by matching the content of one image with the style of another.Convolutional features encode content, while correlations among features encode style.
- Representations: Feature maps from the content, style, and stylized images are extracted by the convolutional network up to selected layers.Each feature map has dimensions N_l × M_l, representing channels and spatial locations.
- Losses: The content loss is mean squared error between content-image and stylized-image feature maps.Content can be represented using one or multiple selected network layers.
- Losses: The style loss is mean squared error between Gram-matrix correlations of style-image and stylized-image filter responses.The Gram matrices are represented as A_l and G_l with dimensions N_l × N_l.
- Optimization: Weighting factors α and β control the relative importance of the content and style components in the overall loss.The stylized image is obtained by minimizing this combined energy with respect to x.
- Optimization: Gradient-based optimization typically starts from random Gaussian noise, but its non-convex loss can lead to local minima.Consequently, initialization is especially important when processing video frames.
4 Style transfer in videos
The video style-transfer method combines flow-based initialization with temporal penalties to stabilize stylization across motion, occlusion, and camera movement. Long-term constraints and alternating multi-pass processing address consistency over occlusions and boundary artifacts.
- Short-term consistency by initialization: Independent random initialization causes consecutive frames to converge to different local minima and produce strong flickering.Warping the previous stylized frame using optical flow provides a motion-aware initialization for the next frame.
- Evaluation: The approach was quantitatively evaluated with DeepFlow and EpicFlow on Sintel and qualitatively on movie shots, successfully eliminating most temporal artifacts.Independent frame processing performed roughly an order of magnitude worse, while erroneous flow impaired temporal constraints in high-motion ambush scenes.
- Temporal consistency loss: The temporal consistency loss penalizes deviations from the previous stylized frame warped by optical flow, excluding disocclusions and motion boundaries.Per-pixel weights are zero in disoccluded regions and at motion boundaries, and one elsewhere.
- Long-term consistency: Long-term motion constraints preserve appearance when regions are occluded and later disoccluded by connecting each pixel to the closest consistent earlier frame.The method searches backward for a frame with consistent correspondences, avoiding unnecessarily long and more erroneous flow paths.
- Multi-pass algorithm: A multi-pass algorithm alternates forward and backward processing to prevent boundary artifacts from propagating inward during strong camera motion.Each pass blends warped non-disoccluded regions with current frames before further optimization, and temporal loss can be enabled after stabilization.
5 Experiments
The experiments evaluate implementation choices, runtime, short-term temporal consistency, and qualitative behavior of long-term and multi-pass extensions. The proposed methods reduce temporal artifacts, while long-term and visual-quality evaluations remain qualitative.
- Runtime: Roughly 3 times faster than naive per-frame processing, the modified algorithm provides temporally consistent output videos.Warped-frame initialization with temporal loss converges in about three minutes on average, compared with eight to ten minutes using random initialization.
- Experimental setup: The evaluation combines implementation details, runtime measurements, a Sintel short-term consistency benchmark, and qualitative movie-shot results.The benchmark uses five scenes, 20–50 frames per scene, six paintings, and ground-truth optical flow and occlusion areas.
- Short-term consistency: DeepFlow performs slightly better than EpicFlow on average in these experiments, despite EpicFlow outperforming DeepFlow on Sintel optical flow.The comparison includes temporal losses with DeepFlow and EpicFlow, plus three non-temporal initialization strategies.
- Short-term consistency: An order of magnitude worse than sophisticated methods, independent per-frame processing performs substantially worse on the short-term consistency benchmark.The temporal penalty significantly improves results in most cases, but erroneous optical flow impairs it in high-motion ambush scenes.
- Long-term consistency and multi-pass algorithm: The short-term benchmark cannot evaluate long-term consistency or visual quality, so the long-term penalty and multi-pass approach are compared qualitatively.Long-term ground-truth flow is unavailable, while visual quality requires human judgment.
- Long-term consistency and multi-pass algorithm: The long-term model keeps the background unchanged after a person passes through the scene, while the multi-pass algorithm avoids artifacts during fast camera motion.These qualitative examples are shown on scenes from Miss Marple.
6 Conclusion
The paper combines initialization, short-term and long-term temporal losses, and multi-pass processing for video style transfer. Together, these techniques produce stable, visually appealing stylized videos under fast motion and strong occlusion.
- 6 Conclusion: The proposed video style-transfer system combines suitable initialization, short-term and long-term consistency losses, and a multi-pass approach.These techniques extend still-image style transfer to temporally coherent video sequences.
- 6 Conclusion: Stable and visually appealing stylized videos can be produced even with fast motion and strong occlusion.The conclusion attributes this outcome to the combined set of proposed techniques.
- 6 Conclusion: Long-term consistency uses J = {1, 10, 20, 40}, while the multi-pass example addresses brightness, contrast, and image-quality changes over time.The accompanying figures compare short-term-only versus long-term consistency and the default method versus multi-pass processing.
7 Additional details of experimental setup
The Sintel experiments use six named artistic style images, and Table 2 reports loss-component weights selected for different input resolutions.
- 7.1 Style images: Six paintings, including works by Kandinsky, Picasso, Matisse, Munch, and Turner, provide the styles for Sintel experiments.
- 7.2 Weighting of the loss components: Table 2 lists the weights assigned to loss-function components for different input resolutions.
8 Additional experiments
Additional experiments examine supplementary video results, alternative temporal losses, and behavior under optical-flow errors. The results show that squared error is generally retained, while flow quality remains important for reconstruction quality.
- Supplementary results: Supplementary videos provide additional Sintel sequences, style comparisons, and evaluations of the basic, multi-pass, and long-term algorithms.
- 8.3 Effect of errors in optical flow estimation: Absolute error improves some cases but causes large fluctuations in others, so the experiments retain mean squared error for temporal consistency.The absolute-error weight was doubled in this comparison.
- 8.3 Effect of errors in optical flow estimation: Optical-flow errors degrade stylization quality, although optimization can partially repair artifacts introduced by warping and disocclusion.Correct flow enables repair in disoccluded regions, whereas erroneous flow cannot be fully recovered.