Source-linked AI summary
Arbitrary Video Style Transfer via Multi-Channel Correlation
Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, Changsheng Xu
TL;DR
Video style transfer must deliver arbitrary high-quality stylization while preserving temporal coherence across frames. MCCNet rearranges and fuses style and content features, aligns outputs to input features, and adds illumination loss; the paper reports stable, vivid stylization for arbitrary video and image transfer.
Problem
Video style transfer must support arbitrary specified styles while maintaining temporal coherence, since direct application of image methods can produce flickering between frames.
Method
MCCNet rearranges style features according to multi-channel content-style correlations, fuses them with content features, aligns outputs to input features, and uses illumination loss for lighting variation.
Results
The method produces stylized results with vivid style patterns, detailed content structures, and stable video outputs, including under complex light conditions.
Takeaways & Limitations
MCCNet provides a frame-based route to arbitrary image and video style transfer that preserves input-video coherence without optical-flow calculation.
Abstract
from arXiv · showhide
Video style transfer is getting more attention in AI community for its numerous applications such as augmented reality and animation productions. Compared with traditional image style transfer, performing this task on video presents new challenges: how to effectively generate satisfactory stylized results for any specified style, and maintain temporal coherence across frames at the same time. Towards this end, we propose Multi-Channel Correction network (MCCNet), which can be trained to fuse the exemplar style features and input content features for efficient style transfer while naturally maintaining the coherence of input videos. Specifically, MCCNet works directly on the feature space of style and content domain where it learns to rearrange and fuse style features based on their similarity with content features. The outputs generated by MCC are features containing the desired style patterns which can further be decoded into images with vivid style textures. Moreover, MCCNet is also designed to explicitly align the features to input which ensures the output maintains the content structures as well as the temporal continuity. To further improve the performance of MCCNet under complex light conditions, we also introduce the illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in both arbitrary video and image style transfer tasks.
1 Introduction
Video style transfer must combine high-quality arbitrary stylization with temporal coherence, because image methods can flicker across frames. MCCNet addresses this by rearranging and aligning style-content features without optical-flow calculation, with illumination loss improving stability under complex lighting.
- Deep CNNs have become widely studied for artistic image generation and translation.
- Figure 1 reports stable stylization across different Sintel frames, with the same object receiving consistent rendered effects.
- Video style transfer must produce satisfactory per-frame stylization while preserving continuity between adjacent frames.Existing image methods can cause flickering when directly applied to videos.
- MCCNet adaptively rearranges style representations using multi-channel correlations with content representations, then merges them into decodable stylized features.This design targets clear content structures and vivid style patterns.
- MCCNet aligns generated features to content features and avoids optical-flow calculation, helping transfer input-video coherence and reduce flickering.
- Illumination loss improves stability by training the model with simulated illumination variations, supporting videos with complex light conditions.
2 Related Work
Image style transfer has progressed from low-level and optimization-based techniques to arbitrary, attention-based methods, while video approaches add temporal information but often depend on optical flow. MCCNet is positioned as an arbitrary style-transfer approach that avoids optical-flow estimation while targeting stable video results with vivid patterns.
- Image style transfer: Traditional image style transfer uses stroke rendering, image analogy, or filtering, while later methods employ deep CNNs for artistic generation and translation.
- Image style transfer: Optimization-based NST is time-consuming, motivating real-time feed-forward methods for specific styles and later arbitrary-style approaches.
- Image style transfer: DFP-based holistic transformations can produce unsatisfactory results, while SANet provides abundant style patterns but fails to maintain distinct content structures.
- Video style transfer: MCCNet is proposed as an arbitrary style-transfer approach applicable to video, avoiding optical-flow estimation while targeting stable transfer with vivid style patterns.
- Video style transfer: Most video style transfer methods build on image-transfer methods, adding temporal constraints or temporal information to reduce flickering and enable faster processing.
3 Methodology
MCCNet uses an encoder-decoder with multi-channel correlation to rearrange and fuse style and content features, aligning outputs to content for coherent video stylization. Its training combines perceptual, identity, and illumination losses to preserve structure, style richness, and robustness to lighting changes.
- Network Structure: MCCNet encodes content and style images into feature maps, applies multi-channel correlation, and decodes fused features into stylized images.The encoder produces fc and fs, while the MCC operation generates fcs for decoding.
- Multi-Channel Correlation: Multi-channel correlation rearranges style features according to content representations while modeling relationships among style channels.The method computes correlations between content channels and every style channel, then combines weighted style channels with content features.
- Feature Alignment: MCCNet aligns generated features with content features, allowing input coherence to transfer to stylized videos without additional temporal constraints.The paper states that slight adjacent-frame changes therefore do not produce violent changes in stylized videos.
- Loss Functions: The training objective combines content and style perceptual losses with identity and illumination losses.The reported weights for content, style, identity, and illumination losses are 4, 15, 70, and 3,000, respectively.
- Loss Functions: Illumination loss adds Gaussian noise to content images and penalizes differences between outputs from perturbed and unperturbed inputs.This is intended to improve robustness under complex light conditions.
4 Experiments
Experiments evaluate MCCNet for image and video style transfer through qualitative comparisons, classification accuracy, user studies, timing, and temporal-consistency analysis. MCCNet produces satisfactory stylization while preserving content structures and video coherence, including under complex lighting.
- Implementation Details and Statistics: MCCNet offers real-time transfer speed comparable to AdaIN, Linear, and SANet, and is faster than several optimization- or transformation-based methods.Inference time was measured on a 16G TitanX GPU across three image sizes.
- Image Style Transfer Results: MCCNet rearranges and fuses style features according to content–style multi-channel correlations, producing clear content structures and controllable style patterns.The rearranged style features fit the original content features before fusion.
- Image Style Transfer Results: MCCNet balances content preservation and style migration in classification-based evaluation, while outperforming AAMS and AdaIN in visual effects.Style classification measures style information, whereas content classification measures retention of original content information.
- Image Style Transfer Results: MCCNet achieved the best image stylization effect in a user study collecting 2,500 votes from 50 participants.Participants judged content-structure integrity and style-pattern visibility.
- Video Style Transfer Results: MCCNet generated video results whose adjacent-frame differences were closest to those of input frames, whereas most comparison methods failed to retain input-video coherence.The evaluation synthesized 14 stylized video clips and visualized adjacent-frame differences with heat maps.
- Video Style Transfer Results: The video-consistency metric uses the mean and variance of adjacent-frame differences, and Table 2 reports MCCNet as yielding the best video consistency.For adjacent frames F_t and F_t−1, Diff_F(t) = ||F_t − F_t−1||.
- Ablation Study: Illumination loss enables MCCNet to be applied flexibly to videos with complex light conditions.The loss is introduced to improve stability under changing illumination.
- Ablation Study: A shallower decoder reduces vivid style patterns in image transfer but has little impact on stylized-video coherence.The result suggests coherence in frame features transfers to generated video despite decoder convolution.
5 Conclusion
MCCNet transfers input-video coherence to stylized videos while producing vivid style patterns and detailed content structures. Illumination loss improves generated-video stability under complex light conditions.
- MCCNet migrates the coherence of input videos to stylized videos, supporting stable rendered results.
- MCCNet generates vivid style patterns and detailed content structures by analyzing multi-channel correlations between content and style features.
- Illumination loss improves the stability of generated videos under complex light conditions.