Source-linked AI summary
Artistic style transfer for videos and spherical images
Manuel Ruder, Alexey Dosovitskiy, Thomas Brox
TL;DR
Stylizing videos frame by frame causes temporal artifacts, while existing optimization can be slow and recursive network stylization can degenerate over time. The paper introduces optimization- and network-based methods with temporal consistency mechanisms, then extends them to spherical images and videos. The methods produce stable, coherent stylized videos, outperform simpler baselines, and support spherical media, although some components remain computationally limited.
Problem
Frame-independent stylization causes temporal artifacts, while recursive network stylization can propagate errors and degrade image quality over time.
Method
The paper combines flow-aware initialization, temporal losses, long-term consistency, multi-pass optimization, and a recursively trained network, extending both to spherical images and videos.
Results
The proposed methods eliminate most temporal artifacts and produce smooth, coherent stylized videos, with the network approach often outperforming the optimization approach and simpler baselines.
Takeaways & Limitations
Video style transfer can produce stable artistic sequences and can be combined with cube-map processing for spherical videos and virtual-reality applications.
Takeaways & Limitations
Recursive network stylization can otherwise propagate errors, causing blurring and loss of content accuracy, while some spherical processing remains too slow for real-time use.
Abstract
from arXiv · showhide
Manually re-drawing an image in a certain artistic style takes a professional artist a long time. Doing this for a video sequence single-handedly is beyond imagination. We present two computational approaches that transfer the style from one image (for example, a painting) to a whole video sequence. In our first approach, we adapt to videos the original image style transfer technique by Gatys et al. based on energy minimization. We introduce new ways of initialization and new loss functions to generate consistent and stable stylized video sequences even in cases with large motion and strong occlusion. Our second approach formulates video stylization as a learning problem. We propose a deep network architecture and training procedures that allow us to stylize arbitrary-length videos in a consistent and stable way, and nearly in real time. We show that the proposed methods clearly outperform simpler baselines both qualitatively and quantitatively. Finally, we propose a way to adapt these approaches also to 360 degree images and videos as they emerge with recent virtual reality hardware.
1 Introduction
The paper extends image style transfer to videos with optimization-based and network-based approaches that enforce temporal coherence. It also adapts style transfer to spherical images and videos for virtual-reality applications.
- Video style transfer: Independent frame processing causes flickering and false discontinuities, motivating temporal regularization for stylized videos.The temporal constraint penalizes deviations between successive frames while accounting for optical flow.
- Video style transfer: Long-term motion estimates and multi-pass processing address appearance changes after occlusion and artifacts near image boundaries.The multi-pass algorithm alternates forward and backward temporal directions.
- Video style transfer: A network-based approach recursively stylizes frames using the previously stylized, flow-warped frame as additional input and trains with a temporal constraint.This approach targets substantially faster processing than optimization-based transfer.
- Results: The methods eliminate most temporal artifacts and produce smooth, coherent stylized videos in quantitative and qualitative evaluations.Experiments use the Sintel MPI dataset, movie shots, and an ablation study.
- Spherical media: Spherical style transfer enforces consistency across adjacent cube-map edges and can be combined with video stylization for spherical videos.The approach remains somewhat too slow for real-time processing but demonstrates potential for artistic virtual-reality applications.
2 Related work
Related work spans neural image style transfer, fast feed-forward networks, temporally consistent painted animations, and video stylization. The paper extends earlier optimization-based video transfer toward faster network-based and spherical-media methods.
- Image style transfer: Gatys et al. represent style with feature correlations and content with high-level neural feature maps.Their work established a foundation for deep-learning-based style transfer.
- Fast image style transfer: Fast image style transfer replaces per-image optimization with trained networks, but traditional models generally learn one specific style template.Later methods introduced multi-style networks.
- Painted animations: Painted-animation methods maintain consistency by moving generated brush strokes along optical flow and varying stroke parameters to control artistic style.These approaches preceded neural video style transfer.
- Video style transfer: The authors’ earlier optimization-based video work added loss functions for consistent video stylization, while the present paper adds network-based and spherical-image extensions.The contribution is positioned as an extension of that earlier conference work.
- Video style transfer: Other video methods reuse warped intermediate features or penalize temporal inconsistencies without using optical flow at test time.The cited approaches differ in their network inputs and use of flow.
3 Style transfer in still images
Still-image style transfer either optimizes an image to match content and style representations or learns a feed-forward transformation network. Both approaches rely on neural features and perceptual losses.
- Optimization-based transfer: Gatys-style transfer generates an image showing one image’s content in another image’s style by minimizing content and style losses.Content is represented by convolutional features, while style is represented by feature correlations.
- Optimization-based transfer: The content loss compares feature maps of the content and stylized images using mean squared error.Content may be represented using one or multiple selected network layers.
- Optimization-based transfer: The style loss compares Gram-matrix correlations of filter responses from the style and stylized images.The style representation is computed over selected style layers.
- Optimization-based transfer: The overall objective weights content and style terms with α and β and minimizes the result with respect to the stylized image.The nonconvex optimization is sensitive to initialization, especially for video frames.
- Network-based transfer: A feed-forward convolutional network learns a style-transfer function by minimizing an expected perceptual loss for arbitrary input images.The loss uses neural features rather than ground-truth stylized images.
4 Optimization-based coherent video style transfer
The optimization-based video method combines flow-based initialization, temporal penalties, long-term correspondences, and alternating multi-pass processing. These mechanisms target consistency across motion, occlusion, and camera movement.
- Short-term consistency: Each frame can be initialized from the preceding stylized frame warped by optical flow, while only the first frame is initialized randomly.Warping aligns the prior stylized appearance with moving scene content.
- Temporal consistency loss: The temporal consistency loss penalizes deviations from the warped prior image while excluding disocclusions and motion boundaries.Forward-backward flow consistency identifies unreliable regions that the optimizer should rebuild.
- Long-term consistency: Long-term consistency connects each frame to selected earlier frames so regions revealed after occlusion can recover a prior appearance.The method prefers the closest reliable past correspondence because longer-range flow is more erroneous.
- Multi-pass algorithm: A multi-pass algorithm alternates forward and backward processing to reduce boundary-related quality loss during camera motion.Temporal consistency can be disabled in early passes and enabled after images stabilize.
- Evaluation: The temporal constraint visibly reduces frame-to-frame differences compared with warped initialization without the constraint or random initialization.The comparison uses contrast-enhanced error images based on ground-truth optical flow.
5 Fast coherent video style transfer
The section develops a fast network-based video stylization method that uses warped prior frames and training strategies to maintain temporal consistency while limiting recursive degradation.
- Consistency by architecture: The feed-forward network loses shift invariance for shifts smaller than its total downsampling factor of four pixels.Dilated convolutions improve temporal consistency but substantially increase memory and computation, and do not handle more complex transformations.
- Training with the prior image: The network receives the current content frame, a warped prior stylized frame, and a validity mask, then recursively produces subsequent stylized frames.The first frame is generated by an image network, while later frames use the previous output warped by optical flow.
- Training with the prior image: Two-frame recursive training is used because jointly optimizing an entire video is computationally infeasible.The basic procedure trains the network to generate a successor frame from the first frame and its associated prior information.
- Training with the prior image: Recursive application can propagate errors, causing blurring and loss of content accuracy over time.The authors address this by mixing samples without prior images and by training across longer recursive sequences so the network learns to correct degeneration.
- Mixed training: Mixed training disables the temporal loss for zero-prior samples, while the consistency-loss weight is increased proportionally to their frequency.A fraction of 0.5 zero-prior samples produced good results in the reported experiments.
- Training data: The evaluation uses challenging Hollywood2 shots and synthetic COCO sequences with motion and zoom transformations.The Hollywood2 data selects the five most differing frame tuples per shot, while COCO supplies 80,000 training images for synthetic sequences.
6 Spherical images and videos
The paper adapts network-based style transfer to spherical media by stylizing cube-map faces with consistent neighboring boundaries. The same temporal and border mechanisms are combined to process spherical videos.
- Spherical images: Cube mapping represents a sphere with six non-distorted square faces, avoiding the severe polar distortion of equirectangular projection.The faces must be stylized without introducing discontinuities along their cut edges.
- Spherical videos: The spherical-media approach uses the fast network-based method because virtual-reality applications require real-time processing.The combined spherical-video system demonstrates the potential for artistic virtual-reality applications but remains somewhat too slow for real-time processing.
- Border consistency: Overlapping extensions near cube-face boundaries provide prior regions for enforcing consistency between adjacent faces.Adjacent-face regions are transformed to match the current face projection before being used as prior information.
- Training dataset: Regular video-trained style-transfer networks perform poorly when the prior image contains a perspective-transformed cube-face border.The authors therefore create dedicated training samples for this setting.
- Spherical videos: Spherical video stylization must satisfy both temporal consistency along motion trajectories and boundary consistency between neighboring cube faces.The method uses per-face optical flow for temporal warping and blends warped prior frames with transformed borders from already stylized neighbors.
7 Experiments on video style transfer
The experiments compare optimization-based and network-based video stylization using temporal-consistency strategies, baselines, ablations, and perceptual evaluation. The proposed methods produce smoother, more coherent videos, while the network approach is substantially faster and often preferred.
- Experimental setup: The study evaluates optimization-based and network-based stylization against independent-frame and initialization baselines using Sintel scenes, styles, and temporal-consistency measures.The evaluation also includes qualitative movie shots, a user study, and ablations of model components and optical-flow choices.
- Comparison to baselines: Temporal consistency makes optimization-based stylization roughly an order of magnitude better than independent frame processing.Previous-frame initialization improves over random initialization but remains 2.3 to 16 times worse than the proposed approach, depending on the scene.
- Comparison to baselines: The network-based method consistently outperforms its independent-frame baseline by a factor of two or more and can outperform optimization while being much faster.It is more accurate on high-motion ambush scenes, where the authors conclude it can partially correct optical-flow errors.
- Perceptual quality: User-study results show a clear preference for network-generated videos over the more costly optimization-based approach.The paper attributes this preference to greater robustness to flow errors and more consistent disocclusion filling, which reduce artifacts.
- Optimization-based analysis: Motion-compensated initialization and an explicit temporal penalty progressively improve optimization-based results, with DeepFlow slightly better than EpicFlow on average.Long-term consistency keeps backgrounds unchanged after occlusions, while the multi-pass algorithm avoids artifacts in newly exposed image areas.
- Network-based analysis: Dilated convolution improves temporal consistency while keeping content and style losses constant, whereas mixed and multi-frame training improve losses over two-frame training with only marginal consistency reduction.The two-frame procedure has very good temporal consistency but substantially increases content and style loss; multi-frame and mixed training produce the best results.
8 Experiments on spherical images and videos
The paper evaluates stylization of spherical images and videos through cube-map edge consistency, fine-tuning, and disocclusion filling. Fine-tuning and random fill-in reduce visible artifacts in difficult regions, while spherical-video errors decrease over time.
- Spherical-image evaluation: Spherical stylization evaluates cube-face stitching by measuring gradient magnitudes at cut edges and inner edges of neighboring prior-image regions.A relative gradient magnitude of 1 indicates no unusual gradient; values above 1 indicate excess gradients compared with the image average.
- Spherical images: Edge consistency reduces false gradients at cut edges but can introduce gradients along prior-image inner edges, especially in low-structure regions.Fine-tuning on spherical-image data improves overall gradient error and reduces the disturbing visual effect in homogeneous regions, although measured gradients can remain elevated when style features mask edges.
- Spherical images: Fine-tuning reduces unnatural inner-edge artifacts in regions with little structure, while both fine-tuned and unfine-tuned networks adapt well in more structured regions.The fine-tuned model sometimes uses stylistic features to mask the transition.
- Spherical videos: Unusual gradients in spherical videos decrease over time, including when the scene is mostly still and only a few objects move.This demonstrates improvement from impaired prior images during processing.
- Disocclusion filling: Random-noise filling improves the Egradient measure from 1.21 to 1.18 and makes false prior-image boundary gradients less visible in low-structure regions.The method replaces zero masking, equivalent to the ImageNet mean pixel, to avoid sharp edges along missing regions.
- Disocclusion filling: Training and inference with random noise, or mixing random-noise training with zero-filled inference, changes how the network fills missing or disoccluded regions.The paper examines these variants using an empty prior image representing a wholly missing region.
9 Runtimes
The runtime experiments show a large speed difference between optimization-based and feed-forward video stylization, while spherical-image processing remains slower and may require additional reprojection time.
- Video stylization: Optimization-based stylization takes around eight to ten minutes for an initially random frame at 1024 × 436 resolution.Warped previous-frame initialization combined with temporal loss reduces convergence time to three minutes on average, a 2 to 3 times speedup.
- Video stylization: The feed-forward approach stylizes a new frame in 400 ms and achieves a 450-fold speedup over optimization-based processing.The timing includes GPU preprocessing of the previous frame but excludes input/output operations.
- Spherical images: Panoramic-image stylization takes 3.9 seconds per full frame, or about 650 ms per 768 × 768 cube face including overlap.Additional reprojection time is required when the video is not distributed in cube-face projection.
10 Conclusion
The paper presents techniques for stable optimization-based video style transfer and a faster network-based approach that also extends to spherical images and videos. These methods produce coherent stylized results under challenging motion and occlusion conditions.
- Optimization-based approach: Optimization-based transfer combines suitable initialization, short- and long-term temporal losses, and multi-pass processing for stable stylization.The techniques target temporal consistency across video frames.
- Optimization-based approach: The optimization-based method remains stable and visually appealing despite fast motion and strong occlusion.
- Network-based approach: A recursively applied network uses a prior image to create coherent style transfer with much lower runtime per frame.Training on longer frame sequences helps the network counteract propagated degradation.
- Spherical extension: The network-based model also adapts to spherical images by using a perspectively transformed neighboring cube-face edge as its prior image when sufficient image structure is available.
A.1 Supplementary videos
The supplementary material documents evaluation styles, video comparisons, failure cases, training behavior, spherical-image processing, and implementation details. Together, these materials show how optical-flow quality and temporal-loss choices affect stylization quality and convergence.
- Supplementary videos: Supplementary videos compare optimization-based, feed-forward, panoramic, and spherical-video results, including basic versus advanced consistency methods.The listed comparisons include long-term consistency, error-propagation reduction, and optimization-versus-fast transfer.
- Evaluation styles: The supplementary figures use paintings by Kandinsky, Picasso, Matisse, Munch, Turner, Wescoat, and a glass painting to evaluate style transfer.The style images support experiments on Sintel, the network approach, and spherical images and videos.
- Optical-flow effects: Correct optical flow allows repair of warping artifacts in disoccluded regions, whereas erroneous flow degrades performance and cannot be fully compensated.The method’s output quality therefore depends strongly on optical-flow estimation.
- Temporal-loss comparison: Absolute temporal error improves some examples but causes large fluctuations in others, so the experiments retain mean squared error.
- Network convergence: The network converges without overfitting, with validation loss computed over five recursively processed consecutive frames; training may stop before 120,000 iterations.
- Spherical-image processing: Spherical-image processing projects a stylized cube-face border onto a neighboring face using a 90° arrangement and a pinhole-camera perspective transformation.The gradient metric compares weighted horizontal and vertical gradients in a selected region with those in the overall image.